For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mamba2_ssd_out_sm100
The chunked Mamba-2 scan's output stage on SM100 (B200).
For each chunk slot of CHUNK tokens and head, with a the log2 decay
cumsum cs of the slot and h the state entering it:
y_i = exp2(cs_i) C_i h^T + sum_{j <= i} (C_i . B_j) exp2(cs_i - cs_j) dt_j x_j + D x_i
on the chunked scan's varlen contract (mamba2_ssd_chunked): x, B and C are
rows of the packed conv output read by TMA at the slot's first token, dt and
cs are its fp32 [H, slots, CHUNK] vectors, h is bf16 [slots, H, P, N],
and y is written only on the slot's own rows.
A persistent CTA walks work items of one slot and two heads of one group, which share C, B and S = C B^T. Five warp roles hand slots to each other through shared-memory barriers:
- load (1 warp): TMA copies in the order the MMA uses them (C0, h, B0, C1, B1, X0, X1) through a ring of six 32 KB slots, and the item's dt and cs.
- mma (1 warp): the carry C [hA; hB]^T into Y, S = C B^T per 128 x 128 tile, and Y += P X with P read from tensor memory.
- mask (8 warps): each S tile into both heads' P (bf16) in the tile's own TMEM columns, with D on the diagonal. Below a row's own 32-key group the decay factors as exp2(cs_i - cs_r) exp2(cs_r - cs_j), r the next group's first key, both at most 1, so one exp2 a group and a per-key factor replace an exp2 per element.
- epilogue (4 warps): the carry's row scale exp2(cs_i) in place before P X accumulates, then Y to bf16 and out by TMA, or row by row from registers for a slot's partial last 128 rows so a chunk never writes the next sequence's rows.
Loads past a sequence's end read the next sequence's rows (or TMA's zero fill): they reach only rows never stored and keys the causal mask zeroes.
This is the Moka component Kernels/test/gpu/Moka/kernels/mamba2/ compute_out_varlen_plan.mojo (after Joe Todd's compute_out_plan.mojo),
statement for statement, with no Moka in it.
comptime values
B_DC
comptime B_DC = 12
B_OP
comptime B_OP = 0
B_P
comptime B_P = 20
B_S
comptime B_S = 16
B_TFREE
comptime B_TFREE = 28
B_XF
comptime B_XF = 29
The x0 / x1 tiles land here; the fixup then publishes them on the ring.
B_Y
comptime B_Y = 24
HP
comptime HP = 2
Heads of one group a work item takes.
L
comptime L = 256
Tokens of a chunk slot.
LOC_CS
comptime LOC_CS = 1
LOC_DT
comptime LOC_DT = 2
LOC_N
comptime LOC_N = 512
LOC_V
comptime LOC_V = 0
N
comptime N = 128
State dim: C, B and the state are two 64-column spans along it.
NB_EPI
comptime NB_EPI = 4
NB_FIX
comptime NB_FIX = 5
NB_MASK
comptime NB_MASK = 3
NB_OSTORE
comptime NB_OSTORE = 1
NB_TPTR
comptime NB_TPTR = 2
NBARS
comptime NBARS = 31
NRB
comptime NRB = 2
NS
comptime NS = 6
Operand ring slots of 32 KB.
OFF_BARS
comptime OFF_BARS = 229376
OFF_CS
comptime OFF_CS = 201728
OFF_DT
comptime OFF_DT = 196608
OFF_T
comptime OFF_T = 0
OFF_TMEM
comptime OFF_TMEM = 229632
OFF_V
comptime OFF_V = 223232
OFF_Y
comptime OFF_Y = 206848
P
comptime P = 64
Head dim: one head's x, y and state rows are one 128-byte swizzle span.
ParamsPtr
comptime ParamsPtr = Pointer[SsdOutParams, ImmUntrackedOrigin]
REGS_EPI
comptime REGS_EPI = 88
REGS_MASK
comptime REGS_MASK = 192
REGS_REST
comptime REGS_REST = 40
ROWS
comptime ROWS = 128
Tokens of a row block: one TMEM lane each.
SLOT
comptime SLOT = 32768
SMEM_BYTES
comptime SMEM_BYTES = 229648
SMS
comptime SMS = 148
SPtr
comptime SPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.SHARED]
THREADS
comptime THREADS = 512
TM_S
comptime TM_S = 0
S (and P in its columns), two stages of 128 columns.
TM_Y
comptime TM_Y = 256
Y, two stages of 64 HP columns.
TMEM_COLS
comptime TMEM_COLS = 512
VEC_BOX
comptime VEC_BOX = 160
VEC_ROW
comptime VEC_ROW = 320
The fp32 elements of one head's row in a vector slot: two copy boxes.
VEC_SLOT
comptime VEC_SLOT = 2560
Structs
-
Bars: -
Cursor: One side's place in a ring: the next slot and its phase parity. -
Item: -
SsdOutParams: TMA descriptors and uniform scalars, passed as one grid constant.
Functions
-
ssd_out_sm100: Computes y for every used chunk slot and head. -
ssd_out_sm100_kernel: Grid min(SMS, items), block THREADS: warps 0-7 mask, 8-11 epilogue, 12 mma, 13 load, 14-15 tail fixup. -
ssd_out_sm100_supported: Returns whether a launch fits the kernel: P 64, N 128, and an even number of heads per group.