For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo trait
MLAKernels
A GPU vendor's MLA decode and prefill dispatch.
The generic flare_mla_* entry points take one as kernels, so a caller
that names a vendor reaches only that vendor's kernels. Pre-elaboration
passes process every arm of a comptime if, so routing on the target
inside a generic body would compile every vendor's call tree.
Implemented traits
Required methods
decode_dispatch
static def decode_dispatch[k_t: MHAOperand, mask_t: MHAMask, dtype: DType, //, kv_num_heads: Int, config: MHAConfig[dtype], ragged: Bool = False, _is_cache_length_accurate: Bool = False, _use_valid_length: Bool = True, decoding_warp_split_k: Bool = False, per_token_scale_rope_aware: Bool = False, sparse: Bool = False, fold_shared_index: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], sparse_row: SparseRowLayout = SparseRowLayout(), LseOutPtrType: OptionalPointer = NullPointer[.float32], *, skip_masked_tiles: Bool = False](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], max_prompt_len: Int, max_cache_valid_length: Int, scale: Float32, ctx: DeviceContext, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[k_t] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False)
Dispatches an MLA decoding request to the vendor's kernels.
Routes to the SM100 (mla_decode_sm100_dispatch), AMD, or legacy NVIDIA
decode path based on the target GPU, selects the block-M tile geometry
(including the AMD MTP token-fold and BM=32/64 heuristic), and launches the
mla_decoding kernel with optional split-K reduction via
mla_splitk_reduce.
Parameters:
- k_t (
MHAOperand): KV cache operand type backingk(inferred). EitherKVCacheMHAOperand(paged cache) orTileTensorMHAOperand(TileTensor input). - mask_t (
MHAMask): Mask functor type applied to attention scores (inferred). - dtype (
DType): Element type ofq(inferred). Must be a half-float orfloat8_e4m3fn. - kv_num_heads (
Int): Number of KV attention heads. Must be 1 for MLA. - config (
MHAConfig[dtype]): MLA attention config carryingnum_headsanddepth(576 for DeepSeek V2/3). - ragged (
Bool): Whetherqis a ragged rank-3 tensor with batch offsets invalid_lengthinstead of a padded rank-4 tensor (defaults toFalse). - _is_cache_length_accurate (
Bool): Workaround unifying KVCache and TileTensor inputs:Truewhenmax_cache_valid_lengthis the accurate latest length (TileTensor case);Falsewhen it precedes the latest tokens, for example zero for continuous execution (KV cache case) (defaults toFalse). - _use_valid_length (
Bool): Whethervalid_lengthis needed for masking;Falseskips it for TileTensor inputs to avoid benchmark overhead (defaults toTrue). - decoding_warp_split_k (
Bool): Whether to use warp-level split-K reduction within the decode kernel (defaults toFalse). - per_token_scale_rope_aware (
Bool): Whetherqand the KV cache use the interleaved FP8+BF16 rope-aware layout (640 bytes/row, 576 logical dims) (defaults toFalse). - sparse (
Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults toFalse). - fold_shared_index (
Bool): Whether to use the read-once shared-index fold that packs folded output/LSE slots into one CTA (defaults toFalse). - has_extra_k (
Bool): Whetherextra_kis supplied (defaults toFalse). Must matchextra_k is not None; comptime so callers without an extra cache do not build the extra-KV sparse kernels. - physical_indices (
Bool): Whetherd_indicesholds pagedphysical_block * page_size + offsetrows rather than token positions, which the kernels decode through the cache's lookup table (defaults toTrue). - AttnSinkPtrType (
OptionalPointer):OptionalPointertype ofattn_sink_ptr(inferred, defaults toNullPointer). - TopkLengthsPtrType (
OptionalPointer):OptionalPointertype shared bytopk_lengthsandextra_topk_lengths(inferred, defaults toNullPointer). - sparse_row (
SparseRowLayout): How the sparse index producer packs each row; SM100 masks the-1padding past each row's closed-form valid length (defaults to plain top-k, which adds no bound). - LseOutPtrType (
OptionalPointer):OptionalPointertype oflse_out(inferred, defaults toNullPointer). - skip_masked_tiles (
Bool): Whether each query token stops at the slots its causal key count covers instead of walking the sequence-wide slot count; SM100 only, and requires one query token per CTA. Under ittopk_lengthsholds one count per query token rather than per batch (defaults toFalse).
Args:
- output (
TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor with shape[batch, num_heads, depth_v](or ragged equivalent). Dtype isbfloat16whenqisfloat8_e4m3fn, else matchesq. - q (
TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. Padded rank-4 shape[batch, q_seq_len, num_heads, depth](or rank-3 ragged). Forper_token_scale_rope_aware, the last dim is 640 FP8 elements representing 576 logical dims. - k (
k_t): KV cache operand. V is derived asK[:, :, :depth_v], so V is not loaded separately. - mask_functor (
mask_t): Mask functor instance applied to attention scores. - valid_length (
TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-batchuint32tensor of valid (pre-padding) sequence lengths. For ragged inputs, carries cumulative row offsets. - max_prompt_len (
Int): Maximum query sequence length (query tokens per batch); the MTP token-fold ceiling. - max_cache_valid_length (
Int): Total number of cached KV entries (the cache context length). - scale (
Float32): Softmax scale factor applied to QK^T. - ctx (
DeviceContext): Device context used to enqueue the kernel. - scalar_args_buf (
NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Optional GPU buffer of pre-computed scalar dispatch args for capturable graph launches; null for the legacy host-computed path. - kv_input_row_offsets (
OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batch row offsets into a ragged KV layout;Nonefor non-ragged inputs. - num_partitions (
Optional[Int]): Optional explicit split-K partition count;Noneselects via heuristic. - q_scale_ptr (
OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token Q scale array (float32); folded into the softmax assigma_Q[token_idx].Nonemeans scale 1.0. - d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices:d_indices[batch * indices_stride + token]gives the physical KV row index. Valid entries must form a prefix of each row, followed by-1: the scan covers onlymin(topk_length, cache_len + q_len)entries.Nonefor non-sparse. - indices_stride (
Int): Allocation stride (max topk across all batches) ford_indices(defaults to 0). - topk_lengths (
TopkLengthsPtrType): Optional valid sparse-slot counts, per batch by default or per packed query token withskip_masked_tiles. Null usesindices_stride; present arrays must cover that domain. - attn_sink_ptr (
AttnSinkPtrType): Optional attention-sink scale pointer (float32); null to disable. - extra_k (
OptionalReg[k_t]): Optional separate always-attend KV cache operand, appended after the topk tokens in the attention loop;Noneto disable. - extra_d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices forextra_k, same layout asd_indices;Nonefor non-sparse extra KV. - extra_indices_stride (
Int): Allocation stride forextra_d_indices(defaults to 0). - extra_topk_lengths (
TopkLengthsPtrType): Optional per-batch valid index counts forextra_k; null means every batch usesextra_indices_stride. - extra_scales_ptr (
OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token scale pointer forextra_k;Noneto disable. - num_partitions_in (
Optional[Int]): Optional capturable-graph scalar from the Python resolver; when set, the SM100 dispatcher uses it instead of recomputingnum_partitionsat grid time. - logical_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Logical sparse indices for position-based causal masking;Nonekeeps the prior slot-count behavior. - selection_lag (
Int): Positions between the token each sparse row was computed for and the token that reads it (defaults to 0). - lse_out (
LseOutPtrType): Optional[total_q_tokens, num_heads]float32 output for each (token, head)'s attention LSE in base 2; SM100 sparse withskip_masked_tilesonly. A token with no keys gets-infand an O of zero. - split_len_bound (
Int): Withlse_out, an upper bound on the per-token lengths that sizes split-K in place ofindices_stride(0 keeps the stride). - row_invariant_verify (
Bool): Whether this launch is the row-invariant pin's own verify, which the SM100 dispatcher sizes at the pin's count instead of from the live cache; the other targets ignore it.
prefill_dispatch
static def prefill_dispatch[k_t: MHAOperand, v_t: MHAOperand, k_rope_t: MHAOperand, mask_t: MHAMask, dtype: DType, output_type: DType, //, kv_num_heads: Int, config: MHAConfig[dtype], q_depth: Int = Int(192), cache_depth: Int = Int(576), _ndbuffer_mha_operand: Bool = False](output: TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, k_rope: k_rope_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], max_prompt_len: Int, scale: Float32, ctx: DeviceContext, cache_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Dispatches an MLA prefill request to the vendor's kernels.
Routes to mla_sm100_prefill on SM100 GPUs or enqueues the generic
mla_prefill kernel on other NVIDIA and AMD targets, computing the grid
and shared-memory layout from the MHA config.
Parameters:
- k_t (
MHAOperand): Key operand type backingk(inferred). EitherRaggedMHAOperandorTileTensorMHAOperand. - v_t (
MHAOperand): Value operand type backingv(inferred). EitherRaggedMHAOperandorTileTensorMHAOperand. - k_rope_t (
MHAOperand): KV cache operand type backingk_rope(inferred). EitherKVCacheMHAOperandorTileTensorMHAOperand. - mask_t (
MHAMask): Mask functor type applied to attention scores (inferred). - dtype (
DType): Element type ofq(inferred). Must bebfloat16orfloat8_e4m3fn. - output_type (
DType): Element type ofoutput(inferred).bfloat16whenqisfloat8_e4m3fn, else matchesq. - kv_num_heads (
Int): Number of KV attention heads. Must be 1 for MLA. - config (
MHAConfig[dtype]): MLA attention config carryingnum_heads,depth, and tile geometry (block_m,block_n,block_k). - q_depth (
Int): Q head dimension in elements (defaults to 192). For DeepSeek V2/3, 192 = 128 nope + 64 rope. - cache_depth (
Int): KV cache head dimension in elements (defaults to 576). The absorbed-latent width of the paged MLA cache. - _ndbuffer_mha_operand (
Bool): Whether the K/V/k_ropeoperands use ND-buffer layout instead of the default TileTensor layout (defaults toFalse).
Args:
- output (
TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor with ragged rank-3 shape[total_q_tokens, num_heads, v_depth]. Dtype isbfloat16whenqisfloat8_e4m3fn, else matchesq. - q (
TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor with ragged rank-3 shape[total_q_tokens, num_heads, q_depth]. - k (
k_t): Key operand (nope part). Carries the ragged K tensor and row offsets. - v (
v_t): Value operand. Carries the ragged V tensor and row offsets. - k_rope (
k_rope_t): KV cache operand providing the rope part of K, with shape[cache_len, 1, q_depth - depth]. - mask_functor (
mask_t): Mask functor instance applied to attention scores. - valid_length (
TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-batchuint32tensor of cumulative row offsets withbatch_size + 1entries;offsets[b]tooffsets[b+1]gives batchb's token range. - max_prompt_len (
Int): Maximum query sequence length (tokens per batch across all batches); drives the grid's M dimension. - scale (
Float32): Softmax scale factor applied to QK^T. - ctx (
DeviceContext): Device context used to enqueue the kernel. - cache_offsets (
OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batchuint32tensor of starting offsets into the paged KV cache;Nonewhen the cache is contiguous.