IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

flare_mla_decoding_dispatch

def flare_mla_decoding_dispatch[k_t: MHAOperand, mask_t: MHAMask, dtype: DType, //, kv_num_heads: Int, config: MHAConfig[dtype], ragged: Bool = False, _is_cache_length_accurate: Bool = False, _use_valid_length: Bool = True, decoding_warp_split_k: Bool = False, per_token_scale_rope_aware: Bool = False, sparse: Bool = False, rope_aware_kv_sparse: Bool = False, fold_shared_index: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], sparse_row: SparseRowLayout = SparseRowLayout()](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], max_prompt_len: Int, max_cache_valid_length: Int, scale: Float32, ctx: DeviceContextBase, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[k_t] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0))

Dispatches an MLA decoding request to the platform-specific kernel.

Routes to the SM100 (mla_decode_sm100_dispatch), AMD, or legacy NVIDIA decode path based on the target GPU, selects the block-M tile geometry (including the AMD MTP token-fold and BM=32/64 heuristic), and launches the mla_decoding kernel with optional split-K reduction via mla_splitk_reduce.

Parameters:

  • ​k_t (MHAOperand): KV cache operand type backing k (inferred). Either KVCacheMHAOperand (paged cache) or TileTensorMHAOperand (TileTensor input).
  • ​mask_t (MHAMask): Mask functor type applied to attention scores (inferred).
  • ​dtype (DType): Element type of q (inferred). Must be a half-float or float8_e4m3fn.
  • ​kv_num_heads (Int): Number of KV attention heads. Must be 1 for MLA.
  • ​config (MHAConfig[dtype]): MLA attention config carrying num_heads and depth (576 for DeepSeek V2/3).
  • ​ragged (Bool): Whether q is a ragged rank-3 tensor with batch offsets in valid_length instead of a padded rank-4 tensor (defaults to False).
  • ​_is_cache_length_accurate (Bool): Workaround unifying KVCache and TileTensor inputs: True when max_cache_valid_length is the accurate latest length (TileTensor case); False when it precedes the latest tokens, for example zero for continuous execution (KV cache case) (defaults to False).
  • ​_use_valid_length (Bool): Whether valid_length is needed for masking; False skips it for TileTensor inputs to avoid benchmark overhead (defaults to True).
  • ​decoding_warp_split_k (Bool): Whether to use warp-level split-K reduction within the decode kernel (defaults to False).
  • ​per_token_scale_rope_aware (Bool): Whether q and the KV cache use the interleaved FP8+BF16 rope-aware layout (640 bytes/row, 576 logical dims) (defaults to False).
  • ​sparse (Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults to False).
  • ​rope_aware_kv_sparse (Bool): Sparse-only routing flag: True selects the BF16-rope sparse kernel (split FP8 nope + BF16 rope); False selects the all-FP8 sparse kernel (defaults to False).
  • ​fold_shared_index (Bool): Whether to use the read-once shared-index fold that packs folded output/LSE slots into one CTA (defaults to False).
  • ​has_extra_k (Bool): Whether extra_k is supplied (defaults to False). Must match extra_k is not None; comptime so callers without an extra cache do not build the extra-KV sparse kernels.
  • ​physical_indices (Bool): Whether d_indices holds paged physical_block * page_size + offset rows rather than token positions, which the kernels decode through the cache's lookup table (defaults to True).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type shared by topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row; SM100 masks the -1 padding past each row's closed-form valid length (defaults to plain top-k, which adds no bound).

Args:

  • ​output (TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor with shape [batch, num_heads, depth_v] (or ragged equivalent). Dtype is bfloat16 when q is float8_e4m3fn, else matches q.
  • ​q (TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. Padded rank-4 shape [batch, q_seq_len, num_heads, depth] (or rank-3 ragged). For per_token_scale_rope_aware, the last dim is 640 FP8 elements representing 576 logical dims.
  • ​k (k_t): KV cache operand. V is derived as K[:, :, :depth_v], so V is not loaded separately.
  • ​mask_functor (mask_t): Mask functor instance applied to attention scores.
  • ​valid_length (TileTensor[.uint32, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-batch uint32 tensor of valid (pre-padding) sequence lengths. For ragged inputs, carries cumulative row offsets.
  • ​max_prompt_len (Int): Maximum query sequence length (query tokens per batch); the MTP token-fold ceiling.
  • ​max_cache_valid_length (Int): Total number of cached KV entries (the cache context length).
  • ​scale (Float32): Softmax scale factor applied to QK^T.
  • ​ctx (DeviceContextBase): Device context used to enqueue the kernel.
  • ​scalar_args_buf (NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Optional GPU buffer of pre-computed scalar dispatch args for capturable graph launches; null for the legacy host-computed path.
  • ​kv_input_row_offsets (OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batch row offsets into a ragged KV layout; None for non-ragged inputs.
  • ​num_partitions (Optional[Int]): Optional explicit split-K partition count; None selects via heuristic.
  • ​q_scale_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token Q scale array (float32); folded into the softmax as sigma_Q[token_idx]. None means scale 1.0.
  • ​d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices: d_indices[batch * indices_stride + token] gives the physical KV row index. Valid entries must form a prefix of each row, followed by -1: the scan covers only min(topk_length, cache_len + q_len) entries. None for non-sparse.
  • ​indices_stride (Int): Allocation stride (max topk across all batches) for d_indices (defaults to 0).
  • ​topk_lengths (TopkLengthsPtrType): Optional per-batch array of actual valid sparse index counts; null means every batch uses indices_stride.
  • ​attn_sink_ptr (AttnSinkPtrType): Optional attention-sink scale pointer (float32); null to disable.
  • ​extra_k (OptionalReg[k_t]): Optional separate always-attend KV cache operand, appended after the topk tokens in the attention loop; None to disable.
  • ​extra_d_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional sparse indices for extra_k, same layout as d_indices; None for non-sparse extra KV.
  • ​extra_indices_stride (Int): Allocation stride for extra_d_indices (defaults to 0).
  • ​extra_topk_lengths (TopkLengthsPtrType): Optional per-batch valid index counts for extra_k; null means every batch uses extra_indices_stride.
  • ​extra_scales_ptr (OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional per-token scale pointer for extra_k; None to disable.
  • ​num_partitions_in (Optional[Int]): Optional capturable-graph scalar from the Python resolver; when set, the SM100 dispatcher uses it instead of recomputing num_partitions at grid time.
  • ​logical_indices (OptionalReg[Pointer[Int32, MutAnyOrigin]]): Logical sparse indices for position-based causal masking; None keeps the prior slot-count behavior.
  • ​selection_lag (Int): Positions between the token each sparse row was computed for and the token that reads it (defaults to 0).

Was this page helpful?