IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_decode_branch_bf16

def mla_decode_branch_bf16[collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, sparse_mla: Bool = False, physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), skip_masked_tiles: Bool = False, AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], dcp_size: Int = Int(1), dcp_interleave: Int = Int(1), LseOutPtrType: OptionalPointer = NullPointer[.float32], q_absorbed: Bool = False, latent_output: Bool = False, OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, selection_lag: Int = Int(0), dcp_rank: Int = Int(0), lse_out: LseOutPtrType = null_pointer[LseOutPtrType](), split_len_bound: Int = Int(0), row_invariant_verify: Bool = False, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))

BF16 MLA decode path.

Applies RoPE and RMSNorm, projects q_nope to latent space, concatenates with q_rope, and runs decode.

Parameters:

  • ​collection_t (KVCollectionT): Type of the KV collection (inferred).
  • ​KVInputFnType (def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred).
  • ​mask_str (StaticString): Mask variant.
  • ​sparse_mla (Bool): Whether to use sparse MLA (defaults to False).
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions (defaults to True); see flare_mla_decoding.
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each row by its valid length (defaults to plain top-k).
  • ​skip_masked_tiles (Bool): Whether each query token stops at the slots its causal key count covers, reading topk_lengths per token (defaults to False); see flare_mla_decoding.
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​dcp_size (Int): Ranks of the decode context-parallel group sharing each request's KV positions (defaults to 1, which stores them all).
  • ​dcp_interleave (Int): Consecutive positions a rank owns before the next rank's turn (defaults to 1).
  • ​LseOutPtrType (OptionalPointer): OptionalPointer type of lse_out (inferred, defaults to NullPointer).
  • ​q_absorbed (Bool): Whether q already holds each head's latent query, [q_nope @ w_uk, q_rope] with q_rope not yet rotated, so w_uk is not read (defaults to False).
  • ​latent_output (Bool): Whether output receives the attention's latent output, before w_uv, so w_uv is not read (defaults to False).
  • ​OutputGatePtrType (OptionalPointer): OptionalPointer type of output_gate (inferred, defaults to NullPointer); a null type applies no gate.
  • ​kernels (MLAGraphKernels): The vendor's MLA graph kernels; see MLAGraphKernels.

Args:

Was this page helpful?