For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mla_stamp
Debug-only per-warp clock stamps for sparse MLA decode (-D MLA_STAMP=1).
Lane 0 of a warp writes one clock64 word to
buf[((cta * STAMP_WARPS) + warp) * STAMP_SLOTS + slot] for CTAs with a
linear index below STAMP_MAX_CTAS. The buffer address is parked in the last
16 bytes of dynamic smem (the CTA record below it), which the launch reserves
only when stamping is on, so role functions need no extra argument. Off, every
call elaborates to nothing.
Warp 8 (the Q load) carries no role stamps, so its row holds the CTA-level
stamps thread 0 writes at the CTA's exit: entry at ST_START, exit at
ST_END, the walk's key count at ST_NTILES, the block and grid indices at
ST_BLOCK/ST_GRID, and the prologue and epilogue sync points as tile-0
events (CTA_*). They are on whenever MLA_STAMP is, whatever the role mask.
comptime values
CTA_ALLOC
comptime CTA_ALLOC = 5
The TMEM-alloc warp, after its alloc.
CTA_INDEX_START
comptime CTA_INDEX_START = 7
The index warp, at its role's start.
CTA_INITS
comptime CTA_INITS = 4
The barrier-init thread, after its inits, fence and prefetches.
CTA_INITS_UNFENCED
comptime CTA_INITS_UNFENCED = 8
The barrier-init thread, after its inits, before their fence.
CTA_LATE_DEALLOC
comptime CTA_LATE_DEALLOC = 10
Written by warp 9 itself after thread 0's exit: after the TMEM dealloc.
CTA_LATE_INBOUND
comptime CTA_LATE_INBOUND = 9
Written by warp 9 itself after thread 0's exit: after the pair's tail waits.
CTA_LATE_RELEASED
comptime CTA_LATE_RELEASED = 11
Written by warp 9 itself after thread 0's exit: after the allocation-permit release, before the TMEM dealloc.
CTA_OFFSETS
comptime CTA_OFFSETS = 3
Thread 0, once the offsets are known and the early exits are passed.
CTA_ROLES_DONE
comptime CTA_ROLES_DONE = 2
After the CTA barrier every role reaches when its walk is done.
CTA_SETUP
comptime CTA_SETUP = 0
After the CTA barrier that publishes the barrier inits and TMEM base.
CTA_SYNC_IN
comptime CTA_SYNC_IN = 1
After the pair's entry cluster sync.
CTA_ZEROED
comptime CTA_ZEROED = 6
Thread 0, after its share of the NoPE-tail zeroing.
MLA_STAMP
comptime MLA_STAMP = (get_defined_int[StringSpan("MLA_STAMP"), Int(0)]() != Int(0))
ST_BLOCK
comptime ST_BLOCK = 6
CTA row only: block_idx x | y << 16 | z << 40; ST_GRID packs grid_dim the same way.
ST_END
comptime ST_END = 1
ST_END_NS
comptime ST_END_NS = 5
ST_EVENTS
comptime ST_EVENTS = 4
ST_GRID
comptime ST_GRID = 7
ST_MAX_TILES
comptime ST_MAX_TILES = (Int((add get_defined_int[StringSpan("MLA_STAMP_SLOTS"), Int(256)](), -8)) // Int(4))
ST_NTILES
comptime ST_NTILES = 3
ST_SMID
comptime ST_SMID = 2
ST_START
comptime ST_START = 0
ST_START_NS
comptime ST_START_NS = 4
ST_TILE0
comptime ST_TILE0 = 8
Event e of tile t lands at ST_TILE0 + t * ST_EVENTS + e.
STAMP_CTA_WARP
comptime STAMP_CTA_WARP = 8
STAMP_MAX_CTAS
comptime STAMP_MAX_CTAS = get_defined_int[StringSpan("MLA_STAMP_MAX_CTAS"), Int(1024)]()
STAMP_ROLES
comptime STAMP_ROLES = get_defined_int[StringSpan("MLA_STAMP_ROLES"), Int(127)]()
Bit r enables role r: 0 softmax, 1 correction, 2 gather, 3 QK, 4 PV, 5 index, 6 re-swizzle.
STAMP_SLOTS
comptime STAMP_SLOTS = get_defined_int[StringSpan("MLA_STAMP_SLOTS"), Int(256)]()
STAMP_SMEM_BYTES
comptime STAMP_SMEM_BYTES = Int(128) if MLA_STAMP else Int(0)
STAMP_SMX
comptime STAMP_SMX = (get_defined_int[StringSpan("MLA_STAMP_SMX"), Int(0)]() != Int(0))
Softmax (role 0) stamps that sum in shared memory (-D MLA_STAMP_SMX=1).
One elected lane of each softmax warp adds the 32-bit clock of tile event
e to word e of its row (red.shared.add), so each tile costs four
fire-and-forget adds and no global store; the per-tile mean of interval
e -> e + 1 is (word e + 1 - word e) / tiles. The exit copies the row
to the warp's stamp slots ST_TILE0 + e, with ST_START left 0 (see
stamp_finish). The per-tile global stores, and their 64-bit base held
across the loop, perturb the softmax too much to time it. The rows take
the CTA record's event words (the Layout G carve has no room for more), so
the CTA events are off under this knob.
STAMP_WARPS
comptime STAMP_WARPS = 16
Functions
-
stamp_begin: Stamps the role start and returns this warp's stamp base address. -
stamp_cta_entry: Thread 0 records the CTA entry;stamp_cta_exitwrites it out. -
stamp_cta_event: -
stamp_cta_event_here: Records a CTA event from the calling thread; one thread calls it. -
stamp_cta_event_warp: Records a CTA event from lane 0 of the one warp that calls it. -
stamp_cta_exit: Thread 0 writes the CTA row: its record, then the exit stamps. -
stamp_cta_keys: Records the CTA's walk length; called once the offsets are known. -
stamp_cta_late: Lane 0 writes a CTA-row event straight to global memory. -
stamp_finish: -
stamp_park: Thread 0 parks the buffer address; a CTA barrier must follow. -
stamp_tile: