IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_matmul_rowwise_scaled_fp8_kernel

def grouped_matmul_rowwise_scaled_fp8_kernel[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, accum_type: DType, CLayout: TensorLayout, ALayout: TensorLayout, BLayout: TensorLayout, AScalesLayout: TensorLayout, BScalesLayout: TensorLayout, AOffsetsLayout: TensorLayout, ExpertIdsLayout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, a_scales_engine: TensorEngine, b_scales_engine: TensorEngine, a_offsets_engine: TensorEngine, expert_ids_engine: TensorEngine, *, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c: TileTensor[c_type, CLayout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, ALayout, ImmUnsafeAnyOrigin, Engine=a_engine], b: TileTensor[b_type, BLayout, ImmUnsafeAnyOrigin, Engine=b_engine], a_scales: TileTensor[a_scales_type, AScalesLayout, ImmUnsafeAnyOrigin, Engine=a_scales_engine], b_scales: TileTensor[b_scales_type, BScalesLayout, ImmUnsafeAnyOrigin, Engine=b_scales_engine], a_offsets: TileTensor[.uint32, AOffsetsLayout, ImmUnsafeAnyOrigin, Engine=a_offsets_engine], expert_ids: TileTensor[.int32, ExpertIdsLayout, ImmUnsafeAnyOrigin, Engine=expert_ids_engine])

Computes the tiled grouped FP8 matmul with rowwise weight scales and per-token activation scales, accumulating in fp32 and applying a single post-reduction scale.

For each token t in group g's row range and output channel n, computes out[t, n] = (sum_k a[t, k] * b[expert, n, k]) * a_scale[t] * b_scale[expert, n]. The a_scale is indexed by the global ragged row and b_scale by the real expert id, per the correctness invariants documented in the file header. Skips the matmul for expert == -1 (inactive LoRA blocks), which writes zeros like an all-zero reduction does.

Each thread owns a _ROW_TILE_REG_M x _ROW_TILE_REG_N block of the CTA's output tile and walks K one device vector at a time, widening each fp8 vector once and reusing it across its register block, so the K loop spends two multiply-adds per widened operand instead of two scalar loads and two converts. The thread block's M extent is a runtime value (block_dim.y): the launcher shrinks it for a decode-shaped expert so lanes are not spent on rows no token reaches.

Was this page helpful?