For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_grouped_matmul_block_scaled_amd
struct Struct_grouped_matmul_block_scaled_amd[preshuffled_b: Bool = False, lane_bytes: Int = Int(16)]
MOGG wrapper for grouped block-scaled matrix multiplication.
Provides graph compiler integration for block-scaled grouped matmul operations used in Mixture of Experts (MoE) layers on AMD GPUs.
Parameters
- preshuffled_b (
Bool): When True, dispatches toblock_scaled_grouped_matmul_amd_prebwhich expects B in the 5D preshuffled layout fromShuffler.preshuffle_b_5d(typically produced by the model's weight adapter at load time, e.g. Kimi K2.5). When False (default), dispatches to the denseblock_scaled_grouped_matmul_amdkernel that reads B row-major. The caller is responsible for preparing B in the matching layout. - lane_bytes (
Int): Element packing of A and B — 16 for MXFP4 (default) or 32 for MXFP8. The kernel readsa/bas raw bytes, so this rather than the operand dtype selects the format, and with it the K extent (Kat MXFP8,K // 2at MXFP4). Preshuffled-B path only. Dense mixed W4A8 derives independent formats from operand dtypes.
Implemented traits
Methods
execute
static def execute(c: T, a: T, b: T, a_scales: T, b_scales: T, expert_start_indices: T, expert_ids: T, max_num_tokens_per_expert: UInt32, num_active_experts: UInt32, estimated_total_m: UInt32, decode_grid_m_cap: UInt32, decode_grid_m_rows: UInt32, context: DeviceContext)
Executes grouped block-scaled matrix multiplication.
Computes C = A @ B^T for multiple expert groups where A and B are block-scaled (e.g. MXFP4: 4-bit floating point packed as uint8).
Args:
- c (
T): The output tensor of shape (total_tokens, N). - a (
T): The input tensor of shape (total_tokens, K // 2) as packed uint8 MXFP4, or (total_tokens, K) as float8_e4m3fn MXFP8 or W4A8 activations. - b (
T): The weight tensor of shape (num_experts, N, K // 2) as packed uint8 MXFP4 (W4A4 or W4A8), or (num_experts, N, K) as float8_e4m3fn MXFP8. - a_scales (
T): The A scale factors in 2D layout. - b_scales (
T): The B scale factors in 3D layout. - expert_start_indices (
T): The starting token index for each expert. - expert_ids (
T): The expert ID for each group. - max_num_tokens_per_expert (
UInt32): The maximum token count for any expert. - num_active_experts (
UInt32): The number of active experts. - estimated_total_m (
UInt32): Estimated total received tokens for this GPU, used by the preb dispatcher to pick the persistent vs direct kernel path. Pass 0 to default to persistent. Ignored whenpreshuffled_b == False. - decode_grid_m_cap (
UInt32): Decode-band gate selecting the direct kernel over the persistent one; 0 disables. Ignored unlesspreshuffled_b. - decode_grid_m_rows (
UInt32): Rows grid.y must cover per expert on the decode bands. Ignored unlesspreshuffled_b. - context (
DeviceContext): The device context pointer.