IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_prefill_branch_sparse_fp8

def mla_prefill_branch_sparse_fp8[dtype: DType, fp8_dtype: DType, fp8_scale_dtype: DType, collection_t: KVCollectionT, //, m_scale_granularity: Int, n_scale_granularity: Int, k_scale_granularity: Int, kv_input_fn: def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width], indices_stride: Int, target: StringSpan[ImmStaticOrigin] = StringSpan("cpu"), attn_sink_inert: Bool = False](output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk: TileTensor[fp8_dtype, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uk_scale: TileTensor[fp8_scale_dtype, Engine=w_uk_scale.Engine, linear_idx_type=w_uk_scale.linear_idx_type], w_uv: TileTensor[fp8_dtype, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], w_uv_scale: TileTensor[fp8_scale_dtype, Engine=w_uv_scale.Engine, linear_idx_type=w_uv_scale.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], ctx: DeviceContextBase, d_indices: Pointer[Int32, MutAnyOrigin], topk_lengths: Pointer[Int32, MutAnyOrigin], attn_sink_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]])

Sparse MLA prefill branch (DSv3.2/GLM absorbed shape, FP8 weights).

Reuses mla_decode_branch_fp8's absorbed-Q construction (q_nope up-proj via w_uk + RoPE/RMSNorm) and the identical w_uv output up-projection. On SM100 the attention call swaps to mla_sm100_prefill_sparse[_fp8] over the paged latent cache; the caller (.fp8.sparse op) has already remapped d_indices from logical to physical rows, so they are passed straight through. On AMD (no native gather4/TMA sparse prefill kernel), it instead runs the dense AMD decode kernel over each query token's selected rows, read in place through SparseRowsMHAOperand. d_indices/topk_lengths are already [total_q, indices_stride]/[total_q] (per query token, not per sequence), so every row of q runs as its own single-token decode batch, with no prefill-specific kernel.

Was this page helpful?