IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mamba2_ssd_chunked

Chunked Mamba-2 SSD prefill scan (varlen, in-place state pool).

The serial kernel in mamba2_ssd_scan.mojo walks every token of a sequence one after another, which leaves a long prefill latency bound. This module computes the same recurrence with the chunked state-space-duality algorithm of the Mamba-2 paper (the same decomposition as mamba_chunk_scan_combined):

Each sequence is cut into chunks of CHUNK = 256 tokens. For a chunk, with a_t = A * dt_t and cs_i = sum_{j <= i} a_j (the within-chunk cumulative log decay, in log2 units):

  1. ssd_chunk_cumsum: dt_t (softplus of dt + dt_bias) and cs.
  2. ssd_chunk_state: the state each chunk adds on its own, S_c[p, n] = sum_t exp(cs_last - cs_t) * dt_t * x[t, p] * B[t, n].
  3. ssd_state_passing: a short serial scan over the chunks of a sequence, S'_c = exp(cs_last_c) * S'_{c-1} + S_c, seeded from the pool slot when the sequence has an initial state. It stores the state entering each chunk and writes the final state back to the pool.
  4. ssd_chunk_output: y = exp(cs_i) * C_i S'_{c-1} + sum_{j <= i} (C_i . B_j) exp(cs_i - cs_j) dt_j x_j + D x_i. The first term is the carry from earlier chunks and the second is the masked-decay attention inside the chunk.

The GEMMs run on tensor cores (m16n8k16, bf16 operands, fp32 accumulation). x, B and C are already bf16; the decayed score matrix and the chunk states are rounded to bf16 for their GEMMs, as in the Triton reference kernels. The decays, dt, the cumulative sums and the state-passing scan stay in fp32, and the pool is rounded once at the final write-back like the serial kernel. The serial kernel stays in charge of decode, verify and short launches.

comptime values​

CHUNK​

comptime CHUNK = 256

Tokens per chunk.

F32_VEC​

comptime F32_VEC = 4

Elements of one 16-byte fp32 vector load or store.

FRAG_C​

comptime FRAG_C = 4

Accumulator values per lane of an m16n8k16 MMA.

GROUPED_CB_AUTO​

comptime GROUPED_CB_AUTO = 0

Selects grouped score reuse only for the validated amortization regime.

GROUPED_CB_OFF​

comptime GROUPED_CB_OFF = 1

Disables grouped score precomputation.

GROUPED_CB_ON​

comptime GROUPED_CB_ON = 2

Forces grouped score precomputation for explicit comparisons.

HALF​

comptime HALF = 8

Offset of the second half (rows g + 8, columns + 8) of a fragment.

LDM_ROWS​

comptime LDM_ROWS = 8

Rows of one 8x8 ldmatrix tile (lanes l % 8 address its rows).

MIN_AVG_SEQ_LEN​

comptime MIN_AVG_SEQ_LEN = 128

Average sequence length from which the chunked scan beats the serial one.

MMA_K​

comptime MMA_K = 16

Reduction depth of an m16n8k16 MMA.

MMA_M​

comptime MMA_M = 16

Rows of an m16n8k16 MMA (and the rows of the tile each warp owns).

MMA_N​

comptime MMA_N = 8

Columns of an m16n8k16 MMA.

NUM_THREADS​

comptime NUM_THREADS = 256

Threads of the state-passing kernel.

NUM_WARPS​

comptime NUM_WARPS = (Int(128) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

OUT_THREADS​

comptime OUT_THREADS = 128

Threads of the tensor-core kernels (4 warps).

QUAD​

comptime QUAD = 4

Lanes that share a fragment row; lane l holds row l // QUAD.

SLAB_ALIGN​

comptime SLAB_ALIGN = 256

Bytes between the regions of the scratch slab, the alignment of a fresh allocation.

SMEM_PAD​

comptime SMEM_PAD = 8

Row padding (bf16 elements) of the shared tiles, against bank conflicts.

TILE​

comptime TILE = 64

Head-dim and row tile of a thread block.

VEC​

comptime VEC = 8

Elements of one 16-byte bf16 vector load or store.

Functions​

  • ​chunked_slots: Returns an upper bound on the number of chunks of a launch.
  • ​mamba2_ssd_chunked_fwd_inplace_gpu: Runs the chunked SSD scan over a varlen batch.
  • ​ssd_chunk_cb: Computes undecayed fp32 C B^T once per chunk and B/C group.
  • ​ssd_chunk_cumsum: Writes dt and the inclusive cumulative log2 decay of every chunk.
  • ​ssd_chunk_map: Builds the chunk table of the launch (one thread).
  • ​ssd_chunk_output: Computes y of one 64-row tile of a chunk for one head on tensor cores.
  • ​ssd_chunk_state: Computes the state each chunk contributes by itself, on tensor cores.
  • ​ssd_state_passing: Passes the state across the chunks of each sequence.
  • ​use_chunked_ssd: Returns whether a launch should take the chunked scan over the serial one.
  • ​use_grouped_cb_ssd: Returns shape eligibility for amortized grouped score precomputation.
  • ​use_mi355_ssd: Returns whether the MI355 chunk state and output kernels can take a launch: P 64, N 128, an even number of heads per group, x, B, C and y as plain rows read and written 16 bytes a lane (16-byte aligned, row strides a multiple of 8 elements), a packed row-major y (y_packed: strides (H P, P, 1), since the output kernel writes y as dense [T, H, P]), and D, if any, contiguous (d_stride 1). They are the chunked scan's only path on AMD, so a launch they cannot take stays on the serial scan.

Was this page helpful?