For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_decode_branch_bf16
def mla_decode_branch_bf16[collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, sparse_mla: Bool = False, physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), skip_masked_tiles: Bool = False, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], dcp_size: Int = Int(1), dcp_interleave: Int = Int(1), LseOutPtrType: OptionalPointer = NullPointer[.float32], q_absorbed: Bool = False, latent_output: Bool = False, OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, selection_lag: Int = Int(0), dcp_rank: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))
BF16 MLA decode path.
Applies RoPE and RMSNorm, projects q_nope to latent space, concatenates with q_rope, and runs decode.
Parameters:
- collection_t (
KVCollectionT): Type of the KV collection (inferred). - KVInputFnType (
def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred). - mask_str (
StaticString): Mask variant. - sparse_mla (
Bool): Whether to use sparse MLA (defaults to False). - physical_indices (
Bool): Whetherd_indicesholds paged physical rows rather than token positions (defaults to True); seeflare_mla_decoding. - sparse_row (
SparseRowLayout): How the sparse index producer packs each row, which bounds each row by its valid length (defaults to plain top-k). - skip_masked_tiles (
Bool): Whether each query token stops at the slots its causal key count covers, readingtopk_lengthsper token (defaults toFalse); seeflare_mla_decoding. - AttnSinkPtrType (
OptionalPointer):OptionalPointertype ofattn_sink_ptr(inferred, defaults toNullPointer). - TopkLengthsPtrType (
OptionalPointer):OptionalPointertype oftopk_lengthsandextra_topk_lengths(inferred, defaults toNullPointer). - dcp_size (
Int): Ranks of the decode context-parallel group sharing each request's KV positions (defaults to 1, which stores them all). - dcp_interleave (
Int): Consecutive positions a rank owns before the next rank's turn (defaults to 1). - LseOutPtrType (
OptionalPointer):OptionalPointertype oflse_out(inferred, defaults toNullPointer). - q_absorbed (
Bool): Whetherqalready holds each head's latent query,[q_nope @ w_uk, q_rope]withq_ropenot yet rotated, sow_ukis not read (defaults toFalse). - latent_output (
Bool): Whetheroutputreceives the attention's latent output, beforew_uv, sow_uvis not read (defaults toFalse). - OutputGatePtrType (
OptionalPointer):OptionalPointertype ofoutput_gate(inferred, defaults toNullPointer); a null type applies no gate. - kernels (
MLAGraphKernels): The vendor's MLA graph kernels; seeMLAGraphKernels.
Args:
- output (
TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor of shape [tot_seq_len, num_heads, v_head_dim], or [tot_seq_len, num_heads, kv_latent_dim] withlatent_output. - q (
TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Combined query tensor containing both nope and rope parts. Shape: [tot_seq_len, num_heads, qk_nope_head_dim + qk_rope_head_dim], or [tot_seq_len, num_heads, kv_latent_dim + qk_rope_head_dim] withq_absorbed. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Indicates where each request starts and ends inq. Shape: [num_batches + 1]. - freqs_cis (
TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type]): Precomputed RoPE frequency values for rotary position embeddings. Shape: [max_seq_len, qk_rope_head_dim]. - kv_norm_gamma (
TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type]): RMSNorm gamma weights for normalizing the KV cache. Shape: [kv_lora_rank]. - kv_collection (
collection_t): Paged KV Cache object. - layer_idx (
UInt32): Layer index. - scale (
Float32): Scale for the attention calculation. - epsilon (
Float32): Small constant for numerical stability in RMSNorm. - w_uk (
TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type]): Weight matrix for projecting the non-rope part of each query head to KV latent space. Shape: [num_heads, kv_latent_dim, qk_nope_head_dim]. - w_uv (
TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type]): Weight matrix for projecting the output of the attention back to each head's original space. Shape: [num_heads, v_head_dim, kv_latent_dim]. - scalar_args_buf (
TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Packed MLA dispatch metadata buffer. - kv_input_fn (
KVInputFnType): Closure that loads the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank- qk_rope_head_dim. It runs on the device, so it must copy what it captures.
- ctx (
DeviceContext): Device context. - d_indices (
OptionalReg[Pointer[Int32, MutAnyOrigin]]): Sparse decode packed indices (null when dense). - indices_stride (
Int): Row stride ind_indices(defaults to 0). - topk_lengths (
TopkLengthsPtrType): Per-batch valid top-k counts. - attn_sink_ptr (
AttnSinkPtrType): Optional per-batch attention sink weights. - num_partitions_in (
Optional[Int]): Capturable-graph num_partitions override. - selection_lag (
Int): Positions between the token each sparse row was computed for and the token that reads it. - dcp_rank (
Int): This rank's index in the decode context-parallel group; the fused latent store writes only the positions it owns. - lse_out (
LseOutPtrType): Optional[tot_seq_len, num_heads]float32 output for the attention LSE, base 2 (SM100 sparse withskip_masked_tilesonly); seeflare_mla_decoding. - split_len_bound (
Int): Withlse_out, an upper bound on the per-token lengths that sizes split-K (0 keepsindices_stride). - row_invariant_verify (
Bool): Whether this launch is the row-invariant pin's own verify, which the SM100 dispatcher sizes at the pin's count instead of from the live cache; see_mla_decode_split. - output_gate (
OutputGatePtrType): Optional bfloat16[seq_len, num_heads * v_head_dim]per-token output gate, row-strided byoutput_gate_row_stride. When present, thew_uvprojection's store multiplies each value bysigmoid(gate), so the caller needs no separate elementwise pass. Null applies no gate. - output_gate_row_stride (
Int): Row stride ofoutput_gate, in elements.