For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_flare_mla_decode_kv_cache_ragged
def generic_flare_mla_decode_kv_cache_ragged[collection_t: KVCollectionT, q_dtype: DType, //, mask_str: StaticString, local_window_size: Int = Int(-1), per_token_scale_rope_aware: Bool = False, sparse_mla: Bool = False, fold_shared_index: Bool = False, sparse_row: SparseRowLayout = SparseRowLayout(), skip_masked_tiles: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], LseOutPtrType: OptionalPointer = NullPointer[.float32], *, kernels: MLAKernels](q: TileTensor[q_dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], context: DeviceContext, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[collection_t.CacheType] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False)
Dispatches MLA decode attention over a ragged batch against a paged KV cache.
Parameters:
- collection_t (
KVCollectionT): The KV cache collection type storing the K cache for this layer (inferred). - q_dtype (
DType): Data type of the query tensor (inferred). - mask_str (
StaticString): Attention mask name selecting the masking strategy, such as "causal", "null", or "sliding_window_causal". - local_window_size (
Int): Sliding-window size in tokens for windowed masks; -1 for masks that ignore it (defaults to -1). - per_token_scale_rope_aware (
Bool): Whetherqand the KV cache use the interleaved FP8+BF16 rope-aware layout (defaults toFalse). - sparse_mla (
Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults toFalse). - fold_shared_index (
Bool): Whether to use the read-once shared-index MTP fold threaded toflare_mla_decoding(defaults toFalse). - sparse_row (
SparseRowLayout): How the sparse index producer packs each row, which bounds each row by its valid length (defaults to plain top-k). - skip_masked_tiles (
Bool): Whether each query token stops at the slots its causal key count covers, readingtopk_lengthsper token (defaults toFalse); seeflare_mla_decoding. - has_extra_k (
Bool): Whetherextra_kis supplied (defaults toFalse). - physical_indices (
Bool): Whetherd_indicesholds paged physical rows rather than token positions (defaults toTrue); seeflare_mla_decoding. - AttnSinkPtrType (
OptionalPointer):OptionalPointertype ofattn_sink_ptr(inferred). - TopkLengthsPtrType (
OptionalPointer):OptionalPointertype oftopk_lengthsandextra_topk_lengths(inferred). - LseOutPtrType (
OptionalPointer):OptionalPointertype oflse_out(inferred). - kernels (
MLAKernels): The vendor's MLA kernels; seeMLAKernels.
Args:
- q (
TileTensor[q_dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tile tensor with shape (batch_size, num_heads, q_head_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tile tensor with shape (batch_size + 1,) denoting the start of each Q entry in the batch. - kv_collection (
collection_t): The collection storing the KVCache entries for this layer, retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed, used to retrieve the KVCache objects from kv_collection. - scale (
Float32): The scaling factor in scaled dot-product attention, usually rsqrt(head_size). - output (
TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): The pre-allocated output tile tensor to write results to. - scalar_args_buf (
TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Packed MLA dispatch metadata buffer. - context (
DeviceContext): The call context pointer, passed by the graph compiler. - q_scale_ptr (
OptionalReg[Pointer[Float32, MutAnyOrigin]]): Per-token Q scale pointer (float32 array, one per Q token); defaults to null (sigma_Q = 1.0). - d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional device pointer to packed int32 physical KV row indices for sparse decode. - indices_stride (
Int): Stride between batch rows in d_indices (e.g. max top-k). - topk_lengths (
TopkLengthsPtrType): Optional valid top-k counts, per batch by default or per packed query token withskip_masked_tiles. - attn_sink_ptr (
AttnSinkPtrType): Optional per-batch attention sink weights. - extra_k (
OptionalReg[collection_t.CacheType]): Optional second KV cache operand for the extra stream. - extra_d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional extra KV stream sparse indices. - extra_indices_stride (
Int): Stride for extra_d_indices. - extra_topk_lengths (
TopkLengthsPtrType): Optional per-batch lengths for the extra stream. - extra_scales_ptr (
OptionalReg[Pointer[Float32, MutAnyOrigin]]): Optional extra stream scales. - num_partitions_in (
Optional[Int]): Capturable-graph num_partitions override forwarded from the MoGG op so SM100 grid sizing matches the kernel. - logical_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Logical sparse indices for position-based causal masking;Nonekeeps the prior slot-count behavior. - selection_lag (
Int): Positions between the token each sparse row was computed for and the token that reads it. - lse_out (
LseOutPtrType): Optional[total_q_tokens, num_heads]float32 LSE output, base 2; seeflare_mla_decoding. - split_len_bound (
Int): Withlse_out, the split-K sizing bound; seeflare_mla_decoding. - row_invariant_verify (
Bool): Whether this launch is the row-invariant pin's own verify, which the SM100 dispatcher sizes at the pin's count instead of from the live cache; see_mla_decode_split.