For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
blockwise_scaled_fp8_matmul_amd_large
Blockwise-scaled FP8 dense matmul for gfx950 (MI355X) at large M.
Computes C[m, n] = sum_k A[m, k] * B[n, k] * a_scale[k // 128, m] * b_scale[n // 128, k // 128] with e4m3 operands, fp32 scales per (1, 128,
128) block, transpose_b (B is [N, K]) and a bf16 C: the contract of
blockwise_scaled_fp8_matmul_amd, for the prefill shapes where that
kernel's one-wave 64 x 64 tiles leave the matrix cores mostly idle.
Structure. One workgroup of eight waves per CU (two a SIMD) computes a
128 x 256 output tile; wave w owns columns 32 w ..< 32 w + 32 of all
128 rows. The K loop walks 256-byte slots, two 16x16x128 MFMA K steps (two
scale blocks) each. A goes to an LDS double buffer by buffer_load ... lds,
B to a slot of LDS of each wave's own, both by DMAs of whole rows (4 rows a
DMA, each lane fetching the 16-byte granule an XOR-16 swizzle puts at its
address), beside the slot's fp32 A scales. Every lane then reads its MFMA
fragments from LDS; B's for the next slot are read right after the barrier
that ends a slot, which frees the wave's B slot for the next fill. One
barrier per slot orders the fill of A's next half against the reads of
this one.
Why both operands go through LDS. The vector memory path a CU's waves share is the bottleneck. A load in the MFMA's lane layout reads 64 bytes of each of 16 rows, and that path spends about three times as long on it as on a DMA of whole lines; B through LDS costs LDS bandwidth, which has room.
Scales. The hardware block-scaled MFMA only takes E8M0 scales, so the fp32
scales are applied in software: each MFMA (one K step, exactly one scale
block) writes a fresh temporary, and acc += tmp * a_s * b_s folds it into
the fp32 accumulator FOLD_LAG MFMAs later, so the fold's VALU work
overlaps the matrix core instead of waiting on it. The MFMA's output lane
holds four columns of one row, so the fold multiplies by one per-lane
scalar. B's scale is wave-uniform (32 columns lie in one 128-column block)
and comes through the scalar cache. A's rows are stored permuted in LDS,
MFMA row r of row tile m being tile row 8 r + m, so a lane's eight
row scales are contiguous: two ds_read_b128s instead of one read per row
tile.
Every lane-derived offset reads the lane id through fresh_lane(): as a
plain intrinsic it is hoisted out of the loop and everything derived from it
spilled, and each spill reload drains every load in flight (gfx9 has one
in-order vmcnt).
comptime values
A_DMAS
comptime A_DMAS = ((Int(128) // (Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, 16)) // Int(256))) // Int(8))
A DMAs a wave issues per slot.
A_SLOT
comptime A_SLOT = 32768
LDS bytes of one half of A's double buffer.
Acc
comptime Acc = SIMD[.float32, 4]
B_DMAS
comptime B_DMAS = (Int(32) // (Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, 16)) // Int(256)))
B DMAs a wave issues per slot.
B_OFF
comptime B_OFF = 65536
B_SLOT
comptime B_SLOT = 8192
LDS bytes of a wave's B slot.
BK
comptime BK = 256
K bytes of a slot: two 16x16x128 K steps, two scale blocks.
BM
comptime BM = 128
Rows of a tile.
BN
comptime BN = 256
DMA_BYTES
comptime DMA_BYTES = (DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size * Int(16))
Bytes one buffer_load ... lds fills: 16 a lane, 4 rows of a slot.
DMA_ROWS
comptime DMA_ROWS = (Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, 16)) // Int(256))
F8
comptime F8 = Float8_e4m3fn
FOLD_LAG
comptime FOLD_LAG = 4
MFMAs between one writing its temporary and the fold reading it: enough issue slots that the fold never waits out the MFMA's result latency.
Frag
comptime Frag = SIMD[.uint8, 32]
GROUP_M
comptime GROUP_M = 8
Row tiles a group of the tile walk spans (see the kernel).
KS
comptime KS = 2
LDS_BYTES
comptime LDS_BYTES = 133120
LdsBytes
comptime LdsBytes = Pointer[UInt8, MutUntrackedOrigin, address_space=AddressSpace.SHARED]
MAX_UNROLL
comptime MAX_UNROLL = 24
Slots of K the loop unrolls fully; longer K loops over slot pairs.
MT
comptime MT = 8
NT
comptime NT = 2
SA_OFF
comptime SA_OFF = 131072
SA_SLOT
comptime SA_SLOT = 1024
LDS bytes of one half's A scales: [KS][BM] fp32, in tile-row order.
SA_WAVES
comptime SA_WAVES = (Int(1024) // Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, 4)))
Waves that each fill 64 of a slot's A scales with one 4-byte DMA.
SCALE_BLOCK
comptime SCALE_BLOCK = 128
THREADS
comptime THREADS = (Int(8) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
WAVES
comptime WAVES = 8
WN
comptime WN = 32
Columns of a wave: two 16-column MFMA tiles, inside one 128-column scale block.
XCDS
comptime XCDS = 8
Functions
-
blockwise_scaled_fp8_matmul_amd_large: Enqueues the large-M blockwise FP8 matmul. -
blockwise_scaled_fp8_matmul_amd_large_kernel: One workgroup's 128 x 256 output tile. See the module docstring. -
blockwise_scaled_fp8_matmul_amd_large_supported: Whether the large-M kernel covers a (bf16C, N, K): whole waves of 32 columns and whole pairs of 256-byte K slots. -
fold:acc + tmp * scale. -
fresh_lane: The lane id, read where it is used: as a plain intrinsic it is hoisted out of the loop, and the offsets derived from it are spilled and reloaded one at a time, each reload draining every load in flight. -
mfma_fp8: One16x16x128e4m3 MFMA into a zero accumulator: output lane(r, q)holds columns4 q ..< 4 q + 4of B's fragment for rowrof A's. Zero scale operands select the unscaled instruction. -
read_a_scales: The A scales of tile rows8 r ..< 8 r + 8for K steps: the row scale of each of the lane's row tiles. -
read_frag: Row tilem's fragment for K stepsof a slot atbase: the bytes a lane feeds the MFMA, granules8 s + qand8 s + 4 + qof row16 m + r, stored under the XOR of the row mod 16 so the reads spread over the banks. -
sched_fence: -
store_c: The tile's firstrowsrows (at most BM) of the wave's 32 columns as bf16;caddresses the tile's first row. -
tile_gemm: One tile's K into the wave's accumulators: rowsm_tile_row ..< m_tile_row + BMof A, of which the firsta_rowsexist (the rest read as zeros), columnsn_wave ..< n_wave + WNof B.sa_strideis the row stride of A's scales.ACTIVE == False: the wave owns no columns of this tile (pastN), and only fills its share of A and meets the barriers.