IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_grouped_matmul_block_scaled

struct Struct_grouped_matmul_block_scaled

MOGG wrapper for grouped block-scaled matrix multiplication.

Provides graph compiler integration for block-scaled grouped matmul operations used in Mixture of Experts (MoE) layers on SM100 GPUs.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[has_a_row_scales: Bool](c: T, a: T, b: T, a_scales: T, b_scales: T, expert_start_indices: T, expert_ids: T, a_scale_offsets: T, expert_scales: T, a_row_scales: T, estimated_total_m: UInt32, num_active_experts: UInt32, context: DeviceContext)

Executes grouped block-scaled matrix multiplication.

Computes C = A @ B^T for multiple expert groups where A and B are block-scaled. uint8 operands are nibble-packed 4-bit E2M1, so their rows are K // 2 wide; float8_e4m3fn operands are unpacked. The operand and scale dtypes together select the UMMA kind (see block_scaled_umma_kind): NVFP4, MXFP4, MXFP8, or the mixed W4A8 pair.

Parameters:

  • ​has_a_row_scales (Bool): Whether a_row_scales holds per-row input scales. Constraints: NVFP4 only.

Args:

  • ​c (T): The output tensor of shape (total_tokens, N).
  • ​a (T): The input tensor of shape (total_tokens, K) for float8_e4m3fn or (total_tokens, K // 2) for packed uint8.
  • ​b (T): The weight tensor of shape (num_experts, N, K) for float8_e4m3fn or (num_experts, N, K // 2) for packed uint8.
  • ​a_scales (T): The A scale factors in tcgen05 5D layout.
  • ​b_scales (T): The B scale factors in tcgen05 6D layout.
  • ​expert_start_indices (T): The starting token index for each expert.
  • ​expert_ids (T): The expert ID for each group.
  • ​a_scale_offsets (T): The starting scale index for each expert.
  • ​expert_scales (T): The per-expert scaling factors for the epilogue.
  • ​a_row_scales (T): Per-row input scales of shape (total_tokens,), multiplied into each output row with its expert scale. Unread unless has_a_row_scales.
  • ​estimated_total_m (UInt32): The estimated total number of tokens.
  • ​num_active_experts (UInt32): The number of active experts.
  • ​context (DeviceContext): The device context pointer.

Was this page helpful?