For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_grouped_matmul_block_scaled
struct Struct_grouped_matmul_block_scaled
MOGG wrapper for grouped block-scaled matrix multiplication.
Provides graph compiler integration for block-scaled grouped matmul operations used in Mixture of Experts (MoE) layers on SM100 GPUs.
Implemented traits
Methods
execute
static def execute[has_a_row_scales: Bool](c: T, a: T, b: T, a_scales: T, b_scales: T, expert_start_indices: T, expert_ids: T, a_scale_offsets: T, expert_scales: T, a_row_scales: T, estimated_total_m: UInt32, num_active_experts: UInt32, context: DeviceContext)
Executes grouped block-scaled matrix multiplication.
Computes C = A @ B^T for multiple expert groups where A and B are
block-scaled. uint8 operands are nibble-packed 4-bit E2M1, so their
rows are K // 2 wide; float8_e4m3fn operands are unpacked. The
operand and scale dtypes together select the UMMA kind (see
block_scaled_umma_kind): NVFP4, MXFP4, MXFP8, or the mixed W4A8 pair.
Parameters:
- has_a_row_scales (
Bool): Whethera_row_scalesholds per-row input scales. Constraints: NVFP4 only.
Args:
- c (
T): The output tensor of shape (total_tokens, N). - a (
T): The input tensor of shape (total_tokens, K) forfloat8_e4m3fnor (total_tokens, K // 2) for packeduint8. - b (
T): The weight tensor of shape (num_experts, N, K) forfloat8_e4m3fnor (num_experts, N, K // 2) for packeduint8. - a_scales (
T): The A scale factors in tcgen05 5D layout. - b_scales (
T): The B scale factors in tcgen05 6D layout. - expert_start_indices (
T): The starting token index for each expert. - expert_ids (
T): The expert ID for each group. - a_scale_offsets (
T): The starting scale index for each expert. - expert_scales (
T): The per-expert scaling factors for the epilogue. - a_row_scales (
T): Per-row input scales of shape (total_tokens,), multiplied into each output row with its expert scale. Unread unlesshas_a_row_scales. - estimated_total_m (
UInt32): The estimated total number of tokens. - num_active_experts (
UInt32): The number of active experts. - context (
DeviceContext): The device context pointer.