For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mla_decode_sparse_qkv_fp8
Sparse MLA decode kernel for SM100 (B200) -- unified gather.
The MMAs run in the Q dtype (mma_type): FP8 Q runs
tcgen05.mma.kind::f8f6f4 and BF16 Q runs kind::f16. The K/V front end
is chosen by the config, and the compute core behind it is shared:
- Direct gather (
config.direct_gather; every BF16 form, the FP8 Layout E form and the routed Layout G forms): WG3 TMA-gathers K/V straight into the 64-column SW128 MMA ring. Under Layout E a 2-byte operand's Q and two K/V stages leave no room for a P ring, so P lives in the rope block of its K/V stage (MLA_SM100_Decode_Config.p_in_kv); FP8 keeps its own P ring. - Linear ring plus re-swizzle (packed NVFP4 at 64-key tiles, and FP8 Layout
E under
MLA_SPARSE_DIRECT_GATHER_E=0): warp 8 gathers whole rows with one INT64/SWIZZLE_NONE gather4 per four rows into a linear ring, and WG3 writes each tile into the SW64 layout the FP8 operand descriptor expects. A gather4 under SW64 has a 64-byte box, so gathering straight into SW64 would take nine times as many gather4 instructions for the same bytes. For packed NVFP4 the load warp also copies each row's E4M3 scales into the stage, and WG3 dequantizese2m1 * sfto FP8 on the way. - In-place convert (
config.kv_convert; BF16 Q over an FP8 cache and packed NVFP4 at 128-key tiles): each tile lands at the top of its own MMA stage and WG3 converts it in place, so the gather ring and the MMA ring are one ring.
At 128 heads the FP8 Layout E form runs as a CTA pair (cta_group == 2):
one cta_group::2 QK MMA over both CTAs' 64-head Q tiles and 64-key K
halves (the Z ring), and a PV MMA split on N so each CTA supplies the V
depth columns it owns (the V ring).
Roles, 4 warpgroups: WG0 (warps 0-3): Softmax. WG1 (warps 4-7): Correction. WG2 (warps 8-11): Load (warp 8: Q, plus K/V under the linear front end) / MMA QK (warp 9) / MMA PV (warp 10) / index producer and store (warp 11). WG3 (warps 12-15): K/V front end: the direct gather, the re-swizzle, or the in-place convert.
Under config.half_rows with at most 16 softmax rows, two threads share each
row: thread t of a warp loads TMEM lane t & 15 and 8 of its warp's 16 key
columns through tcgen05.ld.16x32bx2, the pair combines its row max with one
shuffle, and P rows 16-31 are zeroed once instead of written per tile.
The gather ring and the MMA ring do not share a depth: PV_REL(i) frees the
MMA slot that WG3 refills for tile i + M (M = num_kv_mma_stages), and that
WAR edge is the back-edge of the loop that sets the steady-state rate, so
leftover SMEM buys depth on the MMA side. Barriers: the fixed pipeline
barriers, the output barriers, the index ring (idx_bars) and
cvt_blk_bars: the in-place convert's per-block barriers, plus the
q_in_tmem hold on the last MMA stage.
An NVFP4 index stage holds the tile's value rows, then its scale rows.
Structsβ
- β
MLA_SM100_Decode_Sparse_QKV_FP8: Sparse MLA decode for SM100 (B200), unified gather, FP8 or BF16 MMAs.
Functionsβ
- β
sparse_qkv_fp8_carve_bytes: Returns the end ofMLA_SM100_Decode_Sparse_QKV_FP8's SMEM carve.