IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_decode_branch_bf16

def mla_decode_branch_bf16[collection_t: KVCollectionT, //, mask_str: StringSpan[ImmStaticOrigin], kv_input_fn: def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width], target: StringSpan[ImmStaticOrigin] = StringSpan("cpu"), sparse_mla: Bool = False, physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32]](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], ctx: DeviceContextBase, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, selection_lag: Int = Int(0))

BF16 MLA decode path.

Applies RoPE and RMSNorm, projects q_nope to latent space, concatenates with q_rope, and runs decode.

Parameters:

  • ​collection_t (KVCollectionT): Type of the KV collection (inferred).
  • ​mask_str (StringSpan[ImmStaticOrigin]): Mask variant.
  • ​kv_input_fn (def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width]): Input lambda function to load the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank
    • qk_rope_head_dim.
  • ​target (StringSpan[ImmStaticOrigin]): Target device (defaults to "cpu").
  • ​sparse_mla (Bool): Whether to use sparse MLA (defaults to False).
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions (defaults to True); see flare_mla_decoding.
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each row by its valid length (defaults to plain top-k).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).

Args:

Was this page helpful?