For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mamba2_ssd_out_mi355
The chunked Mamba-2 scan's output stage on MI355 (gfx950).
Y = diag(exp2(cs)) C h^T + ((C B^T) .* M) (dt X) + D X, M[i, j] = (j <= i) exp2(cs_i - cs_j)
per chunk slot and head, on the chunked scan's varlen contract
(mamba2_ssd_chunked): x, B and C rows of the packed conv output at the
slot's first token, fp32 dt and log2 cumsum [H, slots, CHUNK], the bf16
state entering each slot [slots, H, P, N], bf16 D [H] or none, and y
[T, H, P] written only on each slot's own rows. P is 64, N 128.
A persistent workgroup walks work items of the used slots: a slot, two heads
of one group (they share C B^T), and one half of the slot's rows. Two load
waves stream the item's per-key vectors (cs, dt, the factored decay V and
D), the entering state h, and 32-token key blocks (B's rows and both heads'
x rows; rows past the slot's end are zeros) into LDS rings; eight MFMA waves
each own 16 rows. A wave starts Y^T with the carry exp2(cs_i) C_i h^T, then
per 32-key group at or below its rows computes S^T = B C^T with 16x16x32
bf16 MFMAs. A lane then holds S for its row at keys {4q.., 16 + 4q..},
exactly an MFMA B operand in that key order, so P = S times the decay (the
exact one in the rows' own group with D on the diagonal, the factored one
below) goes to Y^T += X^T P^T from the registers, X^T read transposed
(ds_read_tr16_b64) in the same order. Y leaves through a per-wave LDS
tile as whole 256-byte rows.
This is the Moka component Kernels/test/gpu/Moka/amd/moka_amd/kernels/ ssd_varlen.mojo (ssd_out_amd_plan), statement for statement, with no
Moka in it: its rings are LDS counters, as the component binds them.
comptime values
Acc
comptime Acc = SIMD[.float32, 4]
B_CHUNKS
comptime B_CHUNKS = 512
F4
comptime F4 = SIMD[.float32, 4]
Frag
comptime Frag = SIMD[.bfloat16, 8]
GMutPtr
comptime GMutPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.GLOBAL]
GPtr
comptime GPtr = Pointer[T, ImmUntrackedOrigin, address_space=AddressSpace.GLOBAL]
GROUP
comptime GROUP = 32
Keys of one decay group: one MFMA k step.
H_BYTES
comptime H_BYTES = 34816
H_PITCH
comptime H_PITCH = OB_PITCH
KB
comptime KB = 32
Tokens of one key block, the unit the load waves stream.
KEY_CHUNKS
comptime KEY_CHUNKS = 1024
16-byte chunks of a key block: B's rows, then both heads' x rows.
KEY_DEPTH
comptime KEY_DEPTH = 4
L
comptime L = 256
Chunk length: the chunked scan's CHUNK.
Lds
comptime Lds = LPtr[UInt8]
LdsAtomic
comptime LdsAtomic = Atomic[UInt32, scope="workgroup"]
LOAD_THREADS
comptime LOAD_THREADS = (Int(2) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
LOAD_WAVES
comptime LOAD_WAVES = 2
LOC_CS
comptime LOC_CS = 1
LOC_D
comptime LOC_D = 1536
LOC_DT
comptime LOC_DT = 2
LOC_V
comptime LOC_V = 0
LPtr
comptime LPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.SHARED]
MMA_WAVES
comptime MMA_WAVES = 8
N
comptime N = 128
NKB
comptime NKB = 8
NWORDS
comptime NWORDS = 14
OB_PITCH
comptime OB_PITCH = 272
OFF_H
comptime OFF_H = 84000
OFF_KEYS
comptime OFF_KEYS = 0
OFF_VEC
comptime OFF_VEC = 71680
OFF_WORDS
comptime OFF_WORDS = 153632
OFF_Y
comptime OFF_Y = 118816
OKEY_BYTES
comptime OKEY_BYTES = 17920
OUT_LDS
comptime OUT_LDS = 153688
OX_OFF
comptime OX_OFF = 8704
OX_PITCH
comptime OX_PITCH = 144
P
comptime P = 64
THREADS
comptime THREADS = (Int(10) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
VEC_BYTES
comptime VEC_BYTES = 6160
VEC_DEPTH
comptime VEC_DEPTH = 2
W_H
comptime W_H = 12
W_KEYS
comptime W_KEYS = 0
W_VEC
comptime W_VEC = 8
WAVES
comptime WAVES = 10
Y_BYTES
comptime Y_BYTES = 4352
Y_PITCH
comptime Y_PITCH = 272
Structs
-
Item: A work item: chunk slotslot, headsh0andh0 + 1of groupg, the slot's first token and length (0: unused), and the output's half of the slot's rows (0 for the state). -
Ring: One side's cursor over a ring ofdepthslots whose FULL count each ofproducerswaves and EMPTY count each ofconsumerswaves add 1 to per trip. -
SsdMi355Params: Addresses and uniform scalars of both MI355 stages. -
Table: A wave's walk over its items, with the chunk table entry of its next item in flight:fetchuses it when the tags match, then prefetches the following one.
Functions
-
arrive: The wave's LDS operations complete, then its leader adds 1. What a ring slot carries is LDS that threads wrote or read, and LDS runs a wave's operations in order, so device loads and stores stay in flight. -
gfrag: 16 bytes of device memory; a global (not generic) access, so an LDS wait never drains it. -
lds_at: -
load_keys: Key block rows [t0, t0 + KB) of the item intoslot: B's rows atb_pitch, then each head's x rows atx_pitch; rows fromliveon are zeros. Every load issues before the first LDS write. -
resident_workgroups: The workgroups ofkernelthe device holds at once. -
ssd_out_mi355: Y for every used chunk slot's rows into y ([T, H, P] bf16). x, B and C start at their first element, rowsx_ldandbc_ldapart;his the bf16 state entering each slot ([slots, H, P, N]) andd_addrD's address (bf16 [H]) or 0. -
ssd_out_mi355_kernel: Persistent: workgroup c takes items c, c + grid, ... Waves 0-1 load, 2-9 compute. -
ssd_out_mi355_supported: P 64, N 128, and an even number of heads per group. -
tr_frag: An MFMA operand from a token-major LDS tile: lane l of quad q gets columncol + l % 16of rowsrow0 + 4 q.. androw1 + 4 q.. (4 each) for its 8 k values.ds_read_tr16_b64hands lane s of each 16-lane group column s of the 4 x 16 block its lanes address, lane s pointing at row s / 4, columns 4 (s % 4).. -
wait_count: Spins until the count reachestarget, sleeping between polls so a waiting wave leaves the LDS and the issue slots to the working ones. -
zero_words: Every ring count to 0, before any wave touches a ring.