IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mamba2_ssd_out_mi355

The chunked Mamba-2 scan's output stage on MI355 (gfx950).

Y = diag(exp2(cs)) C h^T + ((C B^T) .* M) (dt X) + D X, M[i, j] = (j <= i) exp2(cs_i - cs_j)

per chunk slot and head, on the chunked scan's varlen contract (mamba2_ssd_chunked): x, B and C rows of the packed conv output at the slot's first token, fp32 dt and log2 cumsum [H, slots, CHUNK], the bf16 state entering each slot [slots, H, P, N], bf16 D [H] or none, and y [T, H, P] written only on each slot's own rows. P is 64, N 128.

A persistent workgroup walks work items of the used slots: a slot, two heads of one group (they share C B^T), and one half of the slot's rows. Two load waves stream the item's per-key vectors (cs, dt, the factored decay V and D), the entering state h, and 32-token key blocks (B's rows and both heads' x rows; rows past the slot's end are zeros) into LDS rings; eight MFMA waves each own 16 rows. A wave starts Y^T with the carry exp2(cs_i) C_i h^T, then per 32-key group at or below its rows computes S^T = B C^T with 16x16x32 bf16 MFMAs. A lane then holds S for its row at keys {4q.., 16 + 4q..}, exactly an MFMA B operand in that key order, so P = S times the decay (the exact one in the rows' own group with D on the diagonal, the factored one below) goes to Y^T += X^T P^T from the registers, X^T read transposed (ds_read_tr16_b64) in the same order. Y leaves through a per-wave LDS tile as whole 256-byte rows.

This is the Moka component Kernels/test/gpu/Moka/amd/moka_amd/kernels/ ssd_varlen.mojo (ssd_out_amd_plan), statement for statement, with no Moka in it: its rings are LDS counters, as the component binds them.

comptime values​

Acc​

comptime Acc = SIMD[.float32, 4]

B_CHUNKS​

comptime B_CHUNKS = 512

F4​

comptime F4 = SIMD[.float32, 4]

Frag​

comptime Frag = SIMD[.bfloat16, 8]

GMutPtr​

comptime GMutPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.GLOBAL]

GPtr​

comptime GPtr = Pointer[T, ImmUntrackedOrigin, address_space=AddressSpace.GLOBAL]

GROUP​

comptime GROUP = 32

Keys of one decay group: one MFMA k step.

H_BYTES​

comptime H_BYTES = 34816

H_PITCH​

comptime H_PITCH = OB_PITCH

KB​

comptime KB = 32

Tokens of one key block, the unit the load waves stream.

KEY_CHUNKS​

comptime KEY_CHUNKS = 1024

16-byte chunks of a key block: B's rows, then both heads' x rows.

KEY_DEPTH​

comptime KEY_DEPTH = 4

L​

comptime L = 256

Chunk length: the chunked scan's CHUNK.

Lds​

comptime Lds = LPtr[UInt8]

LdsAtomic​

comptime LdsAtomic = Atomic[UInt32, scope="workgroup"]

LOAD_THREADS​

comptime LOAD_THREADS = (Int(2) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

LOAD_WAVES​

comptime LOAD_WAVES = 2

LOC_CS​

comptime LOC_CS = 1

LOC_D​

comptime LOC_D = 1536

LOC_DT​

comptime LOC_DT = 2

LOC_V​

comptime LOC_V = 0

LPtr​

comptime LPtr = Pointer[T, MutUntrackedOrigin, address_space=AddressSpace.SHARED]

MMA_WAVES​

comptime MMA_WAVES = 8

N​

comptime N = 128

NKB​

comptime NKB = 8

NWORDS​

comptime NWORDS = 14

OB_PITCH​

comptime OB_PITCH = 272

OFF_H​

comptime OFF_H = 84000

OFF_KEYS​

comptime OFF_KEYS = 0

OFF_VEC​

comptime OFF_VEC = 71680

OFF_WORDS​

comptime OFF_WORDS = 153632

OFF_Y​

comptime OFF_Y = 118816

OKEY_BYTES​

comptime OKEY_BYTES = 17920

OUT_LDS​

comptime OUT_LDS = 153688

OX_OFF​

comptime OX_OFF = 8704

OX_PITCH​

comptime OX_PITCH = 144

P​

comptime P = 64

THREADS​

comptime THREADS = (Int(10) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

VEC_BYTES​

comptime VEC_BYTES = 6160

VEC_DEPTH​

comptime VEC_DEPTH = 2

W_H​

comptime W_H = 12

W_KEYS​

comptime W_KEYS = 0

W_VEC​

comptime W_VEC = 8

WAVES​

comptime WAVES = 10

Y_BYTES​

comptime Y_BYTES = 4352

Y_PITCH​

comptime Y_PITCH = 272

Structs​

  • ​Item: A work item: chunk slot slot, heads h0 and h0 + 1 of group g, the slot's first token and length (0: unused), and the output's half of the slot's rows (0 for the state).
  • ​Ring: One side's cursor over a ring of depth slots whose FULL count each of producers waves and EMPTY count each of consumers waves add 1 to per trip.
  • ​SsdMi355Params: Addresses and uniform scalars of both MI355 stages.
  • ​Table: A wave's walk over its items, with the chunk table entry of its next item in flight: fetch uses it when the tags match, then prefetches the following one.

Functions​

  • ​arrive: The wave's LDS operations complete, then its leader adds 1. What a ring slot carries is LDS that threads wrote or read, and LDS runs a wave's operations in order, so device loads and stores stay in flight.
  • ​gfrag: 16 bytes of device memory; a global (not generic) access, so an LDS wait never drains it.
  • ​lds_at:
  • ​load_keys: Key block rows [t0, t0 + KB) of the item into slot: B's rows at b_pitch, then each head's x rows at x_pitch; rows from live on are zeros. Every load issues before the first LDS write.
  • ​resident_workgroups: The workgroups of kernel the device holds at once.
  • ​ssd_out_mi355: Y for every used chunk slot's rows into y ([T, H, P] bf16). x, B and C start at their first element, rows x_ld and bc_ld apart; h is the bf16 state entering each slot ([slots, H, P, N]) and d_addr D's address (bf16 [H]) or 0.
  • ​ssd_out_mi355_kernel: Persistent: workgroup c takes items c, c + grid, ... Waves 0-1 load, 2-9 compute.
  • ​ssd_out_mi355_supported: P 64, N 128, and an even number of heads per group.
  • ​tr_frag: An MFMA operand from a token-major LDS tile: lane l of quad q gets column col + l % 16 of rows row0 + 4 q.. and row1 + 4 q.. (4 each) for its 8 k values. ds_read_tr16_b64 hands lane s of each 16-lane group column s of the 4 x 16 block its lanes address, lane s pointing at row s / 4, columns 4 (s % 4)..
  • ​wait_count: Spins until the count reaches target, sleeping between polls so a waiting wave leaves the LDS and the issue slots to the working ones.
  • ​zero_words: Every ring count to 0, before any wave touches a ring.

Was this page helpful?