For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_prefill_decode_graph_bf16
def mla_prefill_decode_graph_bf16[collection_t: KVCollectionT & Deinitable, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, sparse_mla: Bool = False, sparse_indices_stride: Int = Int(0), physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0), OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels, allow_split: Bool = True](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, max_seq_len: Int, w_k: TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, prefill_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), mixed_split: Optional[MixedSplit] = None, decode_num_partitions: Optional[Int] = None, row_invariant_verify: Bool = False, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))
BF16 MLA prefill/decode graph.
Dispatches to prefill or decode based on max sequence length in the batch.
Parameters:
- collection_t (
KVCollectionT&Deinitable): Type of the KV collection (inferred). - KVInputFnType (
def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred). - mask_str (
StaticString): Mask variant. - sparse_mla (
Bool): Whether to use sparse MLA (defaults to False). - sparse_indices_stride (
Int): Row stride of the sparse decode index buffer (defaults to 0). - physical_indices (
Bool): Whetherd_indicesholds paged physical rows rather than token positions for the decode branch (defaults to True); seeflare_mla_decodingandprefill_d_indices. - sparse_row (
SparseRowLayout): How the sparse index producer packs each row, which bounds each decode row by its valid length (defaults to plain top-k). - AttnSinkPtrType (
OptionalPointer):OptionalPointertype ofattn_sink_ptr(inferred, defaults toNullPointer). - TopkLengthsPtrType (
OptionalPointer):OptionalPointertype oftopk_lengthsandextra_topk_lengths(inferred, defaults toNullPointer). - nvfp4_convert_ratio (
Int): Most context rows per query token at which sparse prefill over an NVFP4 cache converts the context to FP8 once (defaults to 0, never); seenvfp4_prefill_converts_once. - nvfp4_convert_max_rows (
Int): Most rows that conversion may allocate (defaults to 0). - OutputGatePtrType (
OptionalPointer):OptionalPointertype ofoutput_gate(inferred, defaults toNullPointer); a null type applies no gate. - kernels (
MLAGraphKernels): The vendor's MLA graph kernels; seeMLAGraphKernels. - allow_split (
Bool): Whether a passedmixed_splitmay run as two self-calls (defaults to True); the per-view calls pass False.
Args:
- output (
TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor of shape [tot_seq_len, num_heads, v_head_dim]. - q (
TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Combined query tensor containing both nope and rope parts. Shape: [tot_seq_len, num_heads, qk_nope_head_dim + qk_rope_head_dim]. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Indicates where each request starts and ends inq. Shape: [num_batches + 1]. - freqs_cis (
TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type]): Precomputed RoPE frequency values for rotary position embeddings. Shape: [max_seq_len, qk_rope_head_dim]. - kv_norm_gamma (
TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type]): RMSNorm gamma weights for normalizing the KV cache. Shape: [kv_lora_rank]. - kv_collection (
collection_t): Paged KV Cache object. - layer_idx (
UInt32): Layer index. - scale (
Float32): Scale for the attention calculation. - epsilon (
Float32): Small constant for numerical stability in RMSNorm. - buffer_row_offsets (
TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type]): Indicates where each request's KV latent values should be stored in the contiguous K buffer. This is a 1D tensor of shape [num_batches + 1]. - cache_offsets (
TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type]): Indicates the starting token position in the KV cache from which to copy KV latent values for each request. This is a 1D tensor of shape [num_batches + 1]. - buffer_length (
Int): The total number of tokens in the KV cache. Scalar. - max_seq_len (
Int): Maximum sequence length in the batch, used to select prefill versus decode. - w_k (
TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type]): Weight matrix for up-projecting the latent cache to full K. Shape: [num_heads * qk_nope_head_dim, kv_latent_dim]. - w_uk (
TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type]): Weight matrix for projecting the non-rope part of each query head to KV latent space. Shape: [num_heads, kv_latent_dim, qk_nope_head_dim]. - w_uv (
TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type]): Weight matrix for projecting the output of the attention back to each head's original space. Shape: [num_heads, v_head_dim, kv_latent_dim]. - scalar_args_buf (
TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Packed MLA dispatch metadata buffer. - kv_input_fn (
KVInputFnType): Closure that loads the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank- qk_rope_head_dim. It runs on the device, so it must copy what it captures.
- ctx (
DeviceContext): Device context. - d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Optional device pointer to packed int32 physical KV row indices for sparse decode (defaults to None). - indices_stride (
Int): Stride between batch rows ind_indices(defaults to 0). - topk_lengths (
TopkLengthsPtrType): Optional per-batch valid top-k counts (defaults to None). - attn_sink_ptr (
AttnSinkPtrType): Optional per-batch attention sink weights (defaults to None). - num_partitions_in (
Optional[Int]): Capturable-graph num_partitions override (defaults to None). - prefill_d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Paged physical rows the sparse prefill branch gathers in place ofd_indices(defaults to None, meaningd_indices). Required withoutphysical_indices, since only the decode kernels decode token positions. - selection_lag (
Int): Positions between the token each sparse row was computed for and the token that reads it; read by the decode branch only. - mixed_split (
Optional[MixedSplit]): The validated split of the batch into a decode and a prefill view (defaults to None, one whole-batch route); the caller checks the record against the batch before passing one. - decode_num_partitions (
Optional[Int]): The decode view's split-K count, overriding the heuristic for that view only (defaults to None). - row_invariant_verify (
Bool): Whether this launch is the row-invariant pin's own verify; forwarded to the decode branch, see_mla_decode_split. - output_gate (
OutputGatePtrType): Optional bfloat16[seq_len, num_heads * v_head_dim]per-token output gate, row-strided byoutput_gate_row_stride. The decode branch applies it inside thew_uvprojection's store, so a caller that would otherwise runoutput * sigmoid(gate)saves that launch. The prefill branch has no such store, so it falls back to one gated pass, which keeps the result identical at every row count. Null applies no gate. - output_gate_row_stride (
Int): Row stride ofoutput_gate, in elements.