IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

flare_mla_decoding

def flare_mla_decoding[rank: Int, cache_t: KVCacheT, mask_t: MHAMask, dtype: DType, //, config: MHAConfig[dtype], ragged: Bool = False, decoding_warp_split_k: Bool = False, per_token_scale_rope_aware: Bool = False, sparse: Bool = False, fold_shared_index: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], sparse_row: SparseRowLayout = SparseRowLayout(), LseOutPtrType: OptionalPointer = NullPointer[.float32], *, skip_masked_tiles: Bool = False, kernels: MLAKernels](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, ctx: DeviceContext, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], q_max_seq_len: OptionalReg[Int] = None, kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[cache_t] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False)

MLA decoding kernel that would only be called in the optimized compute graph.

The Q input has a shape of [seq_len, num_heads, depth]. The K input has a shape of [seq_len, 1, depth]. The V tensor is derived by reusing K, where V = K[:, :, :depth_v].

Specifically, for DeepSeek V2/3, depth = 576 and depth_v = 512.

When per_token_scale_rope_aware is True, Q and KV cache have an interleaved FP8+BF16 layout: FP8 content (512 bytes) + BF16 rope (128 bytes) = 640 bytes/row. Q's last dimension is 640 (FP8 elements) but represents 576 logical dimensions (512 nope + 64 rope).

This kernel computes attention without needing to load V twice. This kernel only handles decoding requests. In this case q_max_seq_len = 1.

This kernel handles batches with different valid lengths (i.e., before the padding). Such lengths are passed in valid_length argument.

Parameters:

  • ​rank (Int): Tensor rank of q and output (inferred). Must be 4 for padded inputs or 3 for ragged inputs.
  • ​cache_t (KVCacheT): Paged KV cache type backing k (inferred). Carries the KV layout, dtype, and head geometry.
  • ​mask_t (MHAMask): Mask functor type applied to attention scores (inferred).
  • ​dtype (DType): Element type of q (inferred). Must be a half-float or float8_e4m3fn.
  • ​config (MHAConfig[dtype]): MLA attention config carrying num_heads and depth (576 for DeepSeek V2/3).
  • ​ragged (Bool): Whether q is a ragged rank-3 tensor with batch offsets in valid_length instead of a padded rank-4 tensor (defaults to False).
  • ​decoding_warp_split_k (Bool): Whether to use warp-level split-K reduction within the decode kernel (defaults to False).
  • ​per_token_scale_rope_aware (Bool): Whether q and the KV cache use the interleaved FP8+BF16 rope-aware layout (640 bytes/row, 576 logical dims) (defaults to False).
  • ​sparse (Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults to False).
  • ​fold_shared_index (Bool): Whether to use the read-once shared-index fold that packs folded output/LSE slots into one CTA (defaults to False).
  • ​has_extra_k (Bool): Whether extra_k is supplied (defaults to False). Must match extra_k is not None; comptime so callers without an extra cache do not build the extra-KV sparse kernels.
  • ​physical_indices (Bool): Whether d_indices holds paged physical_block * page_size + offset rows rather than token positions, which the kernels decode through the cache's lookup table (defaults to True).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type shared by topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row; SM100 masks the -1 padding past each row's closed-form valid length (defaults to plain top-k, which adds no bound).
  • ​LseOutPtrType (OptionalPointer): OptionalPointer type of lse_out (inferred, defaults to NullPointer).
  • ​skip_masked_tiles (Bool): Whether each query token stops at the slots its causal key count covers instead of walking the sequence-wide slot count; SM100 only, and requires one query token per CTA. Under it topk_lengths holds one count per query token rather than per batch (defaults to False).
  • ​kernels (MLAKernels): The vendor MLA kernels to dispatch to. Callers pass their vendor's, so they never reach other vendors' kernels.

Args:

  • ​output (TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor with shape [batch, num_heads, depth_v] (or ragged equivalent). Dtype is bfloat16 when q is float8_e4m3fn, else matches q.
  • ​q (TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. Padded rank-4 shape [batch, q_seq_len, num_heads, depth] (or rank-3 ragged). For per_token_scale_rope_aware, the last dim is 640 FP8 elements representing 576 logical dims.
  • ​k (cache_t): Paged KV cache operand. V is derived as K[:, :, :depth_v], so V is not loaded separately.
  • ​mask_functor (mask_t): Mask functor instance applied to attention scores.
  • ​valid_length (TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-batch uint32 tensor of valid (pre-padding) sequence lengths. For ragged inputs, carries cumulative row offsets.
  • ​scale (Float32): Softmax scale factor applied to QK^T.
  • ​ctx (DeviceContext): Device context used to enqueue the kernel.
  • ​scalar_args_buf (NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Optional GPU buffer of pre-computed scalar dispatch args for capturable graph launches; null for the legacy host-computed path.
  • ​q_max_seq_len (OptionalReg[Int]): Maximum query sequence length (query tokens per batch). Defaults to the cache's max_prompt_length when None.
  • ​kv_input_row_offsets (OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batch row offsets into a ragged KV layout; None for non-ragged inputs.
  • ​num_partitions (Optional[Int]): Optional explicit split-K partition count; None selects via heuristic.
  • ​q_scale_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token Q scale array (float32); folded into the softmax as sigma_Q[token_idx]. None means scale 1.0.
  • ​d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices: d_indices[batch * indices_stride + token] gives the physical KV row index. Valid entries must form a prefix of each row, followed by -1: the scan covers only min(topk_length, cache_len + q_len) entries. None for non-sparse.
  • ​indices_stride (Int): Allocation stride (max topk across all batches) for d_indices (defaults to 0).
  • ​topk_lengths (TopkLengthsPtrType): Optional valid sparse-slot counts, per batch by default or per packed query token with skip_masked_tiles. Null uses indices_stride; present arrays must cover that domain.
  • ​attn_sink_ptr (AttnSinkPtrType): Optional attention-sink scale pointer (float32); null to disable.
  • ​extra_k (OptionalReg[cache_t]): Optional separate always-attend KV cache operand, appended after the topk tokens in the attention loop; None to disable.
  • ​extra_d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices for extra_k, same layout as d_indices; None for non-sparse extra KV.
  • ​extra_indices_stride (Int): Allocation stride for extra_d_indices (defaults to 0).
  • ​extra_topk_lengths (TopkLengthsPtrType): Optional per-batch valid index counts for extra_k; null means every batch uses extra_indices_stride.
  • ​extra_scales_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token scale pointer for extra_k; None to disable.
  • ​num_partitions_in (Optional[Int]): Optional capturable-graph scalar from the Python resolver; when set, the SM100 dispatcher uses it instead of recomputing num_partitions at grid time.
  • ​logical_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Logical sparse indices for position-based causal masking; None keeps the prior slot-count behavior.
  • ​selection_lag (Int): Positions between the token each sparse row was computed for and the token that reads it (defaults to 0).
  • ​lse_out (LseOutPtrType): Optional [total_q_tokens, num_heads] float32 output for each (token, head)'s attention LSE in base 2; SM100 sparse with skip_masked_tiles only. A token with no keys gets -inf and an O of zero.
  • ​split_len_bound (Int): With lse_out, an upper bound on the per-token lengths that sizes split-K in place of indices_stride (0 keeps the stride).
  • ​row_invariant_verify (Bool): Whether this launch is the row-invariant pin's own verify, which the SM100 dispatcher sizes at the pin's count instead of from the live cache; other targets ignore it.

def flare_mla_decoding[mask_t: MHAMask, dtype: DType, //, config: MHAConfig[dtype], decoding_warp_split_k: Bool = False, *, kernels: MLAKernels](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type], mask_functor: mask_t, scale: Float32, ctx: DeviceContext, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], num_partitions: Optional[Int] = None)

Was this page helpful?