For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mamba2_ssd_chunked
Chunked Mamba-2 SSD prefill scan (varlen, in-place state pool).
The serial kernel in mamba2_ssd_scan.mojo walks every token of a sequence
one after another, which leaves a long prefill latency bound. This module
computes the same recurrence with the chunked state-space-duality algorithm of
the Mamba-2 paper (the same decomposition as mamba_chunk_scan_combined):
Each sequence is cut into chunks of CHUNK = 256 tokens. For a chunk, with
a_t = A * dt_t and cs_i = sum_{j <= i} a_j (the within-chunk cumulative
log decay, in log2 units):
ssd_chunk_cumsum:dt_t(softplus ofdt + dt_bias) andcs.ssd_chunk_state: the state each chunk adds on its own,S_c[p, n] = sum_t exp(cs_last - cs_t) * dt_t * x[t, p] * B[t, n].ssd_state_passing: a short serial scan over the chunks of a sequence,S'_c = exp(cs_last_c) * S'_{c-1} + S_c, seeded from the pool slot when the sequence has an initial state. It stores the state entering each chunk and writes the final state back to the pool.ssd_chunk_output:y = exp(cs_i) * C_i S'_{c-1}+ sum_{j <= i} (C_i . B_j) exp(cs_i - cs_j) dt_j x_j + D x_i. The first term is the carry from earlier chunks and the second is the masked-decay attention inside the chunk.
The GEMMs run on tensor cores (m16n8k16, bf16 operands, fp32 accumulation).
x, B and C are already bf16; the decayed score matrix and the chunk
states are rounded to bf16 for their GEMMs, as in the Triton reference
kernels. The decays, dt, the cumulative sums and the state-passing scan stay
in fp32, and the pool is rounded once at the final write-back like the serial
kernel. The serial kernel stays in charge of decode, verify and short
launches.
comptime values
CHUNK
comptime CHUNK = 256
Tokens per chunk.
F32_VEC
comptime F32_VEC = 4
Elements of one 16-byte fp32 vector load or store.
FRAG_C
comptime FRAG_C = 4
Accumulator values per lane of an m16n8k16 MMA.
GROUPED_CB_AUTO
comptime GROUPED_CB_AUTO = 0
Selects grouped score reuse only for the validated amortization regime.
GROUPED_CB_OFF
comptime GROUPED_CB_OFF = 1
Disables grouped score precomputation.
GROUPED_CB_ON
comptime GROUPED_CB_ON = 2
Forces grouped score precomputation for explicit comparisons.
HALF
comptime HALF = 8
Offset of the second half (rows g + 8, columns + 8) of a fragment.
LDM_ROWS
comptime LDM_ROWS = 8
Rows of one 8x8 ldmatrix tile (lanes l % 8 address its rows).
MIN_AVG_SEQ_LEN
comptime MIN_AVG_SEQ_LEN = 128
Average sequence length from which the chunked scan beats the serial one.
MMA_K
comptime MMA_K = 16
Reduction depth of an m16n8k16 MMA.
MMA_M
comptime MMA_M = 16
Rows of an m16n8k16 MMA (and the rows of the tile each warp owns).
MMA_N
comptime MMA_N = 8
Columns of an m16n8k16 MMA.
NUM_THREADS
comptime NUM_THREADS = 256
Threads of the state-passing kernel.
NUM_WARPS
comptime NUM_WARPS = (Int(128) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
OUT_THREADS
comptime OUT_THREADS = 128
Threads of the tensor-core kernels (4 warps).
QUAD
comptime QUAD = 4
Lanes that share a fragment row; lane l holds row l // QUAD.
SLAB_ALIGN
comptime SLAB_ALIGN = 256
Bytes between the regions of the scratch slab, the alignment of a fresh allocation.
SMEM_PAD
comptime SMEM_PAD = 8
Row padding (bf16 elements) of the shared tiles, against bank conflicts.
TILE
comptime TILE = 64
Head-dim and row tile of a thread block.
VEC
comptime VEC = 8
Elements of one 16-byte bf16 vector load or store.
Functions
-
chunked_slots: Returns an upper bound on the number of chunks of a launch. -
mamba2_ssd_chunked_fwd_inplace_gpu: Runs the chunked SSD scan over a varlen batch. -
ssd_chunk_cb: Computes undecayed fp32C B^Tonce per chunk and B/C group. -
ssd_chunk_cumsum: Writesdtand the inclusive cumulative log2 decay of every chunk. -
ssd_chunk_map: Builds the chunk table of the launch (one thread). -
ssd_chunk_output: Computesyof one 64-row tile of a chunk for one head on tensor cores. -
ssd_chunk_state: Computes the state each chunk contributes by itself, on tensor cores. -
ssd_state_passing: Passes the state across the chunks of each sequence. -
use_chunked_ssd: Returns whether a launch should take the chunked scan over the serial one. -
use_grouped_cb_ssd: Returns shape eligibility for amortized grouped score precomputation. -
use_mi355_ssd: Returns whether the MI355 chunk state and output kernels can take a launch: P 64, N 128, an even number of heads per group, x, B, C and y as plain rows read and written 16 bytes a lane (16-byte aligned, row strides a multiple of 8 elements), a packed row-major y (y_packed: strides (H P, P, 1), since the output kernel writes y as dense [T, H, P]), and D, if any, contiguous (d_stride1). They are the chunked scan's only path on AMD, so a launch they cannot take stays on the serial scan.