IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_matmul_rowwise_dynamic_scaled_fp8

def grouped_matmul_rowwise_dynamic_scaled_fp8[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, //, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel.ON](c: TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[.uint32, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, ctx: DeviceContext)

Grouped (ragged MoE) FP8 matmul with rowwise weight + per-token act scales.

Computes, for each token t in group g's row range and each output channel n::

out[t, n] = (sum_k a[t, k] * b[expert_ids[g], n, k])
            * a_scale[t] * b_scale[expert_ids[g], n]

On SM100 with a bf16 output and static expert shapes this runs the persistent tensor-core grouped GEMM and applies both scales in its compute epilogue, so the scale is applied to the bf16-rounded sum. Everywhere else it runs a naive CUDA-core kernel that accumulates in fp32 and applies a single post-reduction scale.

Parameters:

  • ​c_type (DType): Output dtype (typically bfloat16).
  • ​a_type (DType): Activation dtype (float8_e4m3fn).
  • ​b_type (DType): Weight dtype (float8_e4m3fn).
  • ​a_scales_type (DType): Per-token activation scale dtype (float32).
  • ​b_scales_type (DType): Per-channel weight scale dtype (float32).
  • ​transpose_b (Bool): Must be True; b is [E, N, K].
  • ​pdl_level (PDLLevel): Programmatic dependent launch level; above PDLLevel.OFF the kernel waits on its predecessor grid before its first read and releases dependent grids when it finishes. PDLLevel.NO_WAIT_OVERLAP_AT_END is rejected, since every input of this op is written by the grids before it (defaults to PDLLevel.ON).

Args:

Was this page helpful?