IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_prefill_decode_graph_bf16

def mla_prefill_decode_graph_bf16[collection_t: KVCollectionT & Deinitable, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, sparse_mla: Bool = False, sparse_indices_stride: Int = Int(0), physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0), OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels, allow_split: Bool = True](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, max_seq_len: Int, w_k: TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_uk: TileTensor[.bfloat16, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), num_partitions_in: Optional[Int] = None, prefill_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), mixed_split: Optional[MixedSplit] = None, decode_num_partitions: Optional[Int] = None, row_invariant_verify: Bool = False, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))

BF16 MLA prefill/decode graph.

Dispatches to prefill or decode based on max sequence length in the batch.

Parameters:

  • ​collection_t (KVCollectionT & Deinitable): Type of the KV collection (inferred).
  • ​KVInputFnType (def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred).
  • ​mask_str (StaticString): Mask variant.
  • ​sparse_mla (Bool): Whether to use sparse MLA (defaults to False).
  • ​sparse_indices_stride (Int): Row stride of the sparse decode index buffer (defaults to 0).
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions for the decode branch (defaults to True); see flare_mla_decoding and prefill_d_indices.
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each decode row by its valid length (defaults to plain top-k).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​nvfp4_convert_ratio (Int): Most context rows per query token at which sparse prefill over an NVFP4 cache converts the context to FP8 once (defaults to 0, never); see nvfp4_prefill_converts_once.
  • ​nvfp4_convert_max_rows (Int): Most rows that conversion may allocate (defaults to 0).
  • ​OutputGatePtrType (OptionalPointer): OptionalPointer type of output_gate (inferred, defaults to NullPointer); a null type applies no gate.
  • ​kernels (MLAGraphKernels): The vendor's MLA graph kernels; see MLAGraphKernels.
  • ​allow_split (Bool): Whether a passed mixed_split may run as two self-calls (defaults to True); the per-view calls pass False.

Args:

Was this page helpful?