IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_prefill_decode_graph_fp8

def mla_prefill_decode_graph_fp8[dtype: DType, fp8_dtype: DType, fp8_scale_dtype: DType, collection_t: KVCollectionT & Deinitable, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, m_scale_granularity: Int, n_scale_granularity: Int, k_scale_granularity: Int, mask_str: StaticString, sparse_mla: Bool = False, sparse_indices_stride: Int = Int(0), fold_shared_index: Bool = False, fp8_q: Bool = True, has_extra_k: Bool = False, physical_indices: Bool = True, sparse_row: SparseRowLayout = SparseRowLayout(), AttnSinkPtrType: OptionalPointer = NullPointer[.float32], TopkLengthsPtrType: OptionalPointer = NullPointer[.int32], attn_sink_inert: Bool = False, *, kernels: MLAGraphKernels, allow_split: Bool = True](output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, max_seq_len: Int, w_k: TileTensor[fp8_dtype, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_k_scale: TileTensor[fp8_scale_dtype, Engine=w_k_scale.Engine, linear_idx_type=w_k_scale.linear_idx_type], w_uk: TileTensor[fp8_dtype, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uk_scale: TileTensor[fp8_scale_dtype, Engine=w_uk_scale.Engine, linear_idx_type=w_uk_scale.linear_idx_type], w_uv: TileTensor[fp8_dtype, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], w_uv_scale: TileTensor[fp8_scale_dtype, Engine=w_uv_scale.Engine, linear_idx_type=w_uv_scale.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, indices_stride: Int = Int(0), topk_lengths: TopkLengthsPtrType = null_pointer[TopkLengthsPtrType](), attn_sink_ptr: AttnSinkPtrType = null_pointer[AttnSinkPtrType](), extra_k: OptionalReg[collection_t.CacheType] = None, extra_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, extra_indices_stride: Int = Int(0), extra_topk_lengths: TopkLengthsPtrType = unread_pointer[TopkLengthsPtrType](), extra_scales_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]] = None, num_partitions_in: Optional[Int] = None, prefill_d_indices: OptionalReg[Pointer[Int32, MutAnyOrigin]] = None, selection_lag: Int = Int(0), mixed_split: Optional[MixedSplit] = None, decode_num_partitions: Optional[Int] = None)

This is a manually fused kernel that performs the following operations: - Perform MLA prefill or decode based on the maximum sequence length.

Parameters:

  • ​dtype (DType): Data type of the input and output tensors (inferred).
  • ​fp8_dtype (DType): Data type of the fp8 input and output tensors (inferred).
  • ​fp8_scale_dtype (DType): Data type of the fp8 scale input and output tensors (inferred).
  • ​collection_t (KVCollectionT & Deinitable): Type of the KV collection (inferred).
  • ​KVInputFnType (def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred).
  • ​m_scale_granularity (Int): Granularity of the scale for M dimension of the matrix multiplication.
  • ​n_scale_granularity (Int): Granularity of the scale for N dimension of the matrix multiplication.
  • ​k_scale_granularity (Int): Granularity of the scale for K dimension of the matrix multiplication.
  • ​mask_str (StaticString): Mask variant.
  • ​sparse_mla (Bool): Whether to use sparse MLA (defaults to False).
  • ​sparse_indices_stride (Int): Row stride of the sparse decode index buffer (defaults to 0).
  • ​fold_shared_index (Bool): Whether to use the read-once shared-index MTP fold threaded to flare_mla_decoding (defaults to False).
  • ​fp8_q (Bool): Whether to stage Q in the FP8 cache dtype on the sparse decode path (defaults to True); see mla_decode_branch_fp8.
  • ​has_extra_k (Bool): Whether extra_k is supplied (defaults to False); see flare_mla_decoding.
  • ​physical_indices (Bool): Whether d_indices holds paged physical rows rather than token positions for the decode branch (defaults to True); see flare_mla_decoding and prefill_d_indices.
  • ​sparse_row (SparseRowLayout): How the sparse index producer packs each row, which bounds each decode row by its valid length (defaults to plain top-k).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred, defaults to NullPointer).
  • ​TopkLengthsPtrType (OptionalPointer): OptionalPointer type of topk_lengths and extra_topk_lengths (inferred, defaults to NullPointer).
  • ​attn_sink_inert (Bool): Whether the caller guarantees an inert attention sink; see mla_decode_branch_fp8.
  • ​kernels (MLAGraphKernels): The vendor's MLA graph kernels; see MLAGraphKernels.
  • ​allow_split (Bool): Whether a passed mixed_split may run as two self-calls (defaults to True); the per-view calls pass False.

Args:

Was this page helpful?