IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

generic_flare_mla_decode_kv_cache_ragged

def generic_flare_mla_decode_kv_cache_ragged[collection_t: KVCollectionT, q_dtype: DType, //, mask_str: StaticString, local_window_size: Int = Int(-1), per_token_scale_rope_aware: Bool = False, sparse_mla: Bool = False, fold_shared_index: Bool = False, sparse_row: SparseRowLayout = SparseRowLayout(), skip_masked_tiles: Bool = False, has_extra_k: Bool = False, physical_indices: Bool = True, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], LseOutPtrType: OptionalPointer = NullPointer[.float32], *, kernels: MLAKernels](q: TileTensor[q_dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], context: DeviceContext, q_scale_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[collection_t.CacheType] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, logical_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False)

Dispatches MLA decode attention over a ragged batch against a paged KV cache.

Parameters:

  • ​collection_t (KVCollectionT): The KV cache collection type storing the K cache for this layer (inferred).
  • ​q_dtype (DType): Data type of the query tensor (inferred).
  • ​mask_str (StaticString): Attention mask name selecting the masking strategy, such as "causal", "null", or "sliding_window_causal".
  • ​local_window_size (Int): Sliding-window size in tokens for windowed masks; -1 for masks that ignore it (defaults to -1).
  • ​per_token_scale_rope_aware (Bool): Whether q and the KV cache use the interleaved FP8+BF16 rope-aware layout (defaults to False).
  • ​sparse_mla (Bool): Whether to use sparse attention with pre-computed physical KV row indices via gather4 TMA (defaults to False).
  • ​fold_shared_index (Bool): Whether to use the read-once shared-index MTP fold threaded to flare_mla_decoding (defaults to False).
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each row by its valid length (defaults to plain top-k).
  • ​skip_masked_tiles (Bool): Whether each query token stops at the slots its causal key count covers, reading topk_lengths per token (defaults to False); see flare_mla_decoding.
  • ​has_extra_k (Bool): Whether extra_k is supplied (defaults to False).
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions (defaults to True); see flare_mla_decoding.
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred).
  • ​LseOutPtrType (OptionalPointer): OptionalPointer type of lse_out (inferred).
  • ​kernels (MLAKernels): The vendor's MLA kernels; see MLAKernels.

Args:

Was this page helpful?