IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mamba2_ssd_out_sm100

The chunked Mamba-2 scan's output stage on SM100 (B200).

For each chunk slot of CHUNK tokens and head, with a the log2 decay cumsum cs of the slot and h the state entering it:

y_i = exp2(cs_i) C_i h^T + sum_{j <= i} (C_i . B_j) exp2(cs_i - cs_j) dt_j x_j + D x_i

on the chunked scan's varlen contract (mamba2_ssd_chunked): x, B and C are rows of the packed conv output read by TMA at the slot's first token, dt and cs are its fp32 [H, slots, CHUNK] vectors, h is bf16 [slots, H, P, N], and y is written only on the slot's own rows.

A persistent CTA walks work items of one slot and two heads of one group, which share C, B and S = C B^T. Five warp roles hand slots to each other through shared-memory barriers:

  • load (1 warp): TMA copies in the order the MMA uses them (C0, h, B0, C1, B1, X0, X1) through a ring of six 32 KB slots, and the item's dt and cs.
  • mma (1 warp): the carry C [hA; hB]^T into Y, S = C B^T per 128 x 128 tile, and Y += P X with P read from tensor memory.
  • mask (8 warps): each S tile into both heads' P (bf16) in the tile's own TMEM columns, with D on the diagonal. Below a row's own 32-key group the decay factors as exp2(cs_i - cs_r) exp2(cs_r - cs_j), r the next group's first key, both at most 1, so one exp2 a group and a per-key factor replace an exp2 per element.
  • epilogue (4 warps): the carry's row scale exp2(cs_i) in place before P X accumulates, then Y to bf16 and out by TMA, or row by row from registers for a slot's partial last 128 rows so a chunk never writes the next sequence's rows.

Loads past a sequence's end read the next sequence's rows (or TMA's zero fill): they reach only rows never stored and keys the causal mask zeroes.

This is the Moka component Kernels/test/gpu/Moka/kernels/mamba2/ compute_out_varlen_plan.mojo (after Joe Todd's compute_out_plan.mojo), statement for statement, with no Moka in it.

comptime values​

B_DC​

comptime B_DC = 12

B_OP​

comptime B_OP = 0

B_P​

comptime B_P = 20

B_S​

comptime B_S = 16

B_TFREE​

comptime B_TFREE = 28

B_XF​

comptime B_XF = 29

The x0 / x1 tiles land here; the fixup then publishes them on the ring.

B_Y​

comptime B_Y = 24

HP​

comptime HP = 2

Heads of one group a work item takes.

L​

comptime L = 256

Tokens of a chunk slot.

LOC_CS​

comptime LOC_CS = 1

LOC_DT​

comptime LOC_DT = 2

LOC_N​

comptime LOC_N = 512

LOC_V​

comptime LOC_V = 0

N​

comptime N = 128

State dim: C, B and the state are two 64-column spans along it.

NB_EPI​

comptime NB_EPI = 4

NB_FIX​

comptime NB_FIX = 5

NB_MASK​

comptime NB_MASK = 3

NB_OSTORE​

comptime NB_OSTORE = 1

NB_TPTR​

comptime NB_TPTR = 2

NBARS​

comptime NBARS = 31

NRB​

comptime NRB = 2

NS​

comptime NS = 6

Operand ring slots of 32 KB.

OFF_BARS​

comptime OFF_BARS = 229376

OFF_CS​

comptime OFF_CS = 201728

OFF_DT​

comptime OFF_DT = 196608

OFF_T​

comptime OFF_T = 0

OFF_TMEM​

comptime OFF_TMEM = 229632

OFF_V​

comptime OFF_V = 223232

OFF_Y​

comptime OFF_Y = 206848

P​

comptime P = 64

Head dim: one head's x, y and state rows are one 128-byte swizzle span.

ParamsPtr​

comptime ParamsPtr = Pointer[SsdOutParams, ImmUntrackedOrigin]

REGS_EPI​

comptime REGS_EPI = 88

REGS_MASK​

comptime REGS_MASK = 192

REGS_REST​

comptime REGS_REST = 40

ROWS​

comptime ROWS = 128

Tokens of a row block: one TMEM lane each.

SLOT​

comptime SLOT = 32768

SMEM_BYTES​

comptime SMEM_BYTES = 229648

SMS​

comptime SMS = 148

SPtr​

comptime SPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.SHARED]

THREADS​

comptime THREADS = 512

TM_S​

comptime TM_S = 0

S (and P in its columns), two stages of 128 columns.

TM_Y​

comptime TM_Y = 256

Y, two stages of 64 HP columns.

TMEM_COLS​

comptime TMEM_COLS = 512

VEC_BOX​

comptime VEC_BOX = 160

VEC_ROW​

comptime VEC_ROW = 320

The fp32 elements of one head's row in a vector slot: two copy boxes.

VEC_SLOT​

comptime VEC_SLOT = 2560

Structs​

Functions​

Was this page helpful?