For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_prefill_branch_sparse_fp8
def mla_prefill_branch_sparse_fp8[dtype: DType, fp8_dtype: DType, fp8_scale_dtype: DType, collection_t: KVCollectionT, //, m_scale_granularity: Int, n_scale_granularity: Int, k_scale_granularity: Int, kv_input_fn: def[width: Int](IndexList[Int(2)]) capturing thin -> SIMD[.bfloat16, width], indices_stride: Int, target: StringSpan[ImmStaticOrigin] = StringSpan("cpu"), attn_sink_inert: Bool = False](output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk: TileTensor[fp8_dtype, Engine=w_uk.Engine, linear_idx_type=w_uk.linear_idx_type], w_uk_scale: TileTensor[fp8_scale_dtype, Engine=w_uk_scale.Engine, linear_idx_type=w_uk_scale.linear_idx_type], w_uv: TileTensor[fp8_dtype, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], w_uv_scale: TileTensor[fp8_scale_dtype, Engine=w_uv_scale.Engine, linear_idx_type=w_uv_scale.linear_idx_type], scalar_args_buf: TileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], ctx: DeviceContextBase, d_indices: Pointer[Int32, MutAnyOrigin], topk_lengths: Pointer[Int32, MutAnyOrigin], attn_sink_ptr: OptionalReg[Pointer[Float32, MutAnyOrigin]])
Sparse MLA prefill branch (DSv3.2/GLM absorbed shape, FP8 weights).
Reuses mla_decode_branch_fp8's absorbed-Q construction (q_nope up-proj via
w_uk + RoPE/RMSNorm) and the identical w_uv output up-projection. On
SM100 the attention call swaps to mla_sm100_prefill_sparse[_fp8] over the
paged latent cache; the caller (.fp8.sparse op) has already remapped
d_indices from logical to physical rows, so they are passed straight
through. On AMD (no native gather4/TMA sparse prefill kernel), it instead
runs the dense AMD decode kernel over each query token's selected rows,
read in place through SparseRowsMHAOperand. d_indices/topk_lengths
are already [total_q, indices_stride]/[total_q] (per query token, not
per sequence), so every row of q runs as its own single-token decode
batch, with no prefill-specific kernel.