IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_prefill_decode_graph_bf16

def mla_prefill_decode_graph_bf16[collection_t: KVCollectionT, //, mask_str: StringSpan[ImmStaticOrigin], kv_input_fn: def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width], target: StringSpan[ImmStaticOrigin] = StringSpan("cpu"), sparse_mla: Bool = False, sparse_indices_stride: Int = Int(0), physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32]](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, max_seq_len: Int, w_k: TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], ctx: DeviceContextBase, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, prefill_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0))

BF16 MLA prefill/decode graph.

Dispatches to prefill or decode based on max sequence length in the batch.

Parameters:

  • ​collection_t (KVCollectionT): Type of the KV collection (inferred).
  • ​mask_str (StringSpan[ImmStaticOrigin]): Mask variant.
  • ​kv_input_fn (def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width]): Input lambda function to load the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank
    • qk_rope_head_dim.
  • ​target (StringSpan[ImmStaticOrigin]): Target device (defaults to "cpu").
  • ​sparse_mla (Bool): Whether to use sparse MLA (defaults to False).
  • ​sparse_indices_stride (Int): Row stride of the sparse decode index buffer (defaults to 0).
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions for the decode branch (defaults to True); see flare_mla_decoding and prefill_d_indices.
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each decode row by its valid length (defaults to plain top-k).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).

Args:

Was this page helpful?