IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_matmul_rowwise_dynamic_scaled_fp8

def grouped_matmul_rowwise_dynamic_scaled_fp8[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, a_offsets_type: DType, expert_ids_type: DType, //, transpose_b: Bool = True, target: StringSpan[ImmStaticOrigin] = StringSpan("cpu")](c: TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[a_offsets_type, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[expert_ids_type, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, ctx: DeviceContextBase)

Grouped (ragged MoE) FP8 matmul with rowwise weight + per-token act scales.

Target: NVIDIA SM100 (B200). Correctness-first naive grouped kernel; no persistent / TMA path. Computes, for each token t in group g's row range and each output channel n::

out[t, n] = (sum_k a[t, k] * b[expert_ids[g], n, k])
            * a_scale[t] * b_scale[expert_ids[g], n]

accumulated in fp32 with a single post-reduction scale.

Parameters:

  • ​c_type (DType): Output dtype (typically bfloat16).
  • ​a_type (DType): Activation dtype (float8_e4m3fn).
  • ​b_type (DType): Weight dtype (float8_e4m3fn).
  • ​a_scales_type (DType): Per-token activation scale dtype (float32).
  • ​b_scales_type (DType): Per-channel weight scale dtype (float32).
  • ​a_offsets_type (DType): Ragged-offset dtype (uint32).
  • ​expert_ids_type (DType): Expert-id dtype (int32).
  • ​transpose_b (Bool): Must be True; b is [E, N, K].
  • ​target (StringSpan[ImmStaticOrigin]): Compilation target string.

Args:

Was this page helpful?