IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo trait

MLAKernels

A GPU vendor's MLA decode and prefill dispatch.

The generic flare_mla_* entry points take one as kernels, so a caller that names a vendor reaches only that vendor's kernels. Pre-elaboration passes process every arm of a comptime if, so routing on the target inside a generic body would compile every vendor's call tree.

Implemented traits​

AnyType

Required methods​

decode_dispatch​

static def decode_dispatch[k_t: MHAOperand, mask_t: MHAMask, dtype: DType, //, kv_num_heads: Int, config: MHAConfig[dtype], ragged: Bool = False, _is_cache_length_accurate: Bool = False, _use_valid_length: Bool = True, decoding_warp_split_k: Bool = False, per_token_scale_rope_aware: Bool = False, sparse: Bool = False, fold_shared_index: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], sparse_row: SparseRowLayout = SparseRowLayout(), LseOutPtrType: OptionalPointer = NullPointer[.float32], *, skip_masked_tiles: Bool = False](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], max_prompt_len: Int, max_cache_valid_length: Int, scale: Float32, ctx: DeviceContext, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[k_t] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False)

Dispatches an MLA decoding request to the vendor's kernels.

Routes to the SM100 (mla_decode_sm100_dispatch), AMD, or legacy NVIDIA decode path based on the target GPU, selects the block-M tile geometry (including the AMD MTP token-fold and BM=32/64 heuristic), and launches the mla_decoding kernel with optional split-K reduction via mla_splitk_reduce.

Parameters:

  • ​k_t (MHAOperand): KV cache operand type backing k (inferred). Either KVCacheMHAOperand (paged cache) or TileTensorMHAOperand (TileTensor input).
  • ​mask_t (MHAMask): Mask functor type applied to attention scores (inferred).
  • ​dtype (DType): Element type of q (inferred). Must be a half-float or float8_e4m3fn.
  • ​kv_num_heads (Int): Number of KV attention heads. Must be 1 for MLA.
  • ​config (MHAConfig[dtype]): MLA attention config carrying num_heads and depth (576 for DeepSeek V2/3).
  • ​ragged (Bool): Whether q is a ragged rank-3 tensor with batch offsets in valid_length instead of a padded rank-4 tensor (defaults to False).
  • ​_is_cache_length_accurate (Bool): Workaround unifying KVCache and TileTensor inputs: True when max_cache_valid_length is the accurate latest length (TileTensor case); False when it precedes the latest tokens, for example zero for continuous execution (KV cache case) (defaults to False).
  • ​_use_valid_length (Bool): Whether valid_length is needed for masking; False skips it for TileTensor inputs to avoid benchmark overhead (defaults to True).
  • ​decoding_warp_split_k (Bool): Whether to use warp-level split-K reduction within the decode kernel (defaults to False).
  • ​per_token_scale_rope_aware (Bool): Whether q and the KV cache use the interleaved FP8+BF16 rope-aware layout (640 bytes/row, 576 logical dims) (defaults to False).
  • ​sparse (Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults to False).
  • ​fold_shared_index (Bool): Whether to use the read-once shared-index fold that packs folded output/LSE slots into one CTA (defaults to False).
  • ​has_extra_k (Bool): Whether extra_k is supplied (defaults to False). Must match extra_k is not None; comptime so callers without an extra cache do not build the extra-KV sparse kernels.
  • ​physical_indices (Bool): Whether d_indices holds paged physical_block * page_size + offset rows rather than token positions, which the kernels decode through the cache's lookup table (defaults to True).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type shared by topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row; SM100 masks the -1 padding past each row's closed-form valid length (defaults to plain top-k, which adds no bound).
  • ​LseOutPtrType (OptionalPointer): OptionalPointer type of lse_out (inferred, defaults to NullPointer).
  • ​skip_masked_tiles (Bool): Whether each query token stops at the slots its causal key count covers instead of walking the sequence-wide slot count; SM100 only, and requires one query token per CTA. Under it topk_lengths holds one count per query token rather than per batch (defaults to False).

Args:

  • ​output (TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor with shape [batch, num_heads, depth_v] (or ragged equivalent). Dtype is bfloat16 when q is float8_e4m3fn, else matches q.
  • ​q (TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. Padded rank-4 shape [batch, q_seq_len, num_heads, depth] (or rank-3 ragged). For per_token_scale_rope_aware, the last dim is 640 FP8 elements representing 576 logical dims.
  • ​k (k_t): KV cache operand. V is derived as K[:, :, :depth_v], so V is not loaded separately.
  • ​mask_functor (mask_t): Mask functor instance applied to attention scores.
  • ​valid_length (TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-batch uint32 tensor of valid (pre-padding) sequence lengths. For ragged inputs, carries cumulative row offsets.
  • ​max_prompt_len (Int): Maximum query sequence length (query tokens per batch); the MTP token-fold ceiling.
  • ​max_cache_valid_length (Int): Total number of cached KV entries (the cache context length).
  • ​scale (Float32): Softmax scale factor applied to QK^T.
  • ​ctx (DeviceContext): Device context used to enqueue the kernel.
  • ​scalar_args_buf (NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Optional GPU buffer of pre-computed scalar dispatch args for capturable graph launches; null for the legacy host-computed path.
  • ​kv_input_row_offsets (OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batch row offsets into a ragged KV layout; None for non-ragged inputs.
  • ​num_partitions (Optional[Int]): Optional explicit split-K partition count; None selects via heuristic.
  • ​q_scale_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token Q scale array (float32); folded into the softmax as sigma_Q[token_idx]. None means scale 1.0.
  • ​d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices: d_indices[batch * indices_stride + token] gives the physical KV row index. Valid entries must form a prefix of each row, followed by -1: the scan covers only min(topk_length, cache_len + q_len) entries. None for non-sparse.
  • ​indices_stride (Int): Allocation stride (max topk across all batches) for d_indices (defaults to 0).
  • ​topk_lengths (TopkLengthsPtrType): Optional valid sparse-slot counts, per batch by default or per packed query token with skip_masked_tiles. Null uses indices_stride; present arrays must cover that domain.
  • ​attn_sink_ptr (AttnSinkPtrType): Optional attention-sink scale pointer (float32); null to disable.
  • ​extra_k (OptionalReg[k_t]): Optional separate always-attend KV cache operand, appended after the topk tokens in the attention loop; None to disable.
  • ​extra_d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices for extra_k, same layout as d_indices; None for non-sparse extra KV.
  • ​extra_indices_stride (Int): Allocation stride for extra_d_indices (defaults to 0).
  • ​extra_topk_lengths (TopkLengthsPtrType): Optional per-batch valid index counts for extra_k; null means every batch uses extra_indices_stride.
  • ​extra_scales_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token scale pointer for extra_k; None to disable.
  • ​num_partitions_in (Optional[Int]): Optional capturable-graph scalar from the Python resolver; when set, the SM100 dispatcher uses it instead of recomputing num_partitions at grid time.
  • ​logical_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Logical sparse indices for position-based causal masking; None keeps the prior slot-count behavior.
  • ​selection_lag (Int): Positions between the token each sparse row was computed for and the token that reads it (defaults to 0).
  • ​lse_out (LseOutPtrType): Optional [total_q_tokens, num_heads] float32 output for each (token, head)'s attention LSE in base 2; SM100 sparse with skip_masked_tiles only. A token with no keys gets -inf and an O of zero.
  • ​split_len_bound (Int): With lse_out, an upper bound on the per-token lengths that sizes split-K in place of indices_stride (0 keeps the stride).
  • ​row_invariant_verify (Bool): Whether this launch is the row-invariant pin's own verify, which the SM100 dispatcher sizes at the pin's count instead of from the live cache; the other targets ignore it.

prefill_dispatch​

static def prefill_dispatch[k_t: MHAOperand, v_t: MHAOperand, k_rope_t: MHAOperand, mask_t: MHAMask, dtype: DType, output_type: DType, //, kv_num_heads: Int, config: MHAConfig[dtype], q_depth: Int = Int(192), cache_depth: Int = Int(576), _ndbuffer_mha_operand: Bool = False](output: TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, k_rope: k_rope_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], max_prompt_len: Int, scale: Float32, ctx: DeviceContext, cache_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Dispatches an MLA prefill request to the vendor's kernels.

Routes to mla_sm100_prefill on SM100 GPUs or enqueues the generic mla_prefill kernel on other NVIDIA and AMD targets, computing the grid and shared-memory layout from the MHA config.

Parameters:

  • ​k_t (MHAOperand): Key operand type backing k (inferred). Either RaggedMHAOperand or TileTensorMHAOperand.
  • ​v_t (MHAOperand): Value operand type backing v (inferred). Either RaggedMHAOperand or TileTensorMHAOperand.
  • ​k_rope_t (MHAOperand): KV cache operand type backing k_rope (inferred). Either KVCacheMHAOperand or TileTensorMHAOperand.
  • ​mask_t (MHAMask): Mask functor type applied to attention scores (inferred).
  • ​dtype (DType): Element type of q (inferred). Must be bfloat16 or float8_e4m3fn.
  • ​output_type (DType): Element type of output (inferred). bfloat16 when q is float8_e4m3fn, else matches q.
  • ​kv_num_heads (Int): Number of KV attention heads. Must be 1 for MLA.
  • ​config (MHAConfig[dtype]): MLA attention config carrying num_heads, depth, and tile geometry (block_m, block_n, block_k).
  • ​q_depth (Int): Q head dimension in elements (defaults to 192). For DeepSeek V2/3, 192 = 128 nope + 64 rope.
  • ​cache_depth (Int): KV cache head dimension in elements (defaults to 576). The absorbed-latent width of the paged MLA cache.
  • ​_ndbuffer_mha_operand (Bool): Whether the K/V/k_rope operands use ND-buffer layout instead of the default TileTensor layout (defaults to False).

Args:

Was this page helpful?