For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
grouped_matmul_rowwise_dynamic_scaled_fp8
def grouped_matmul_rowwise_dynamic_scaled_fp8[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, //, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel.ON](c: TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[.uint32, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, ctx: DeviceContext)
Grouped (ragged MoE) FP8 matmul with rowwise weight + per-token act scales.
Computes, for each token t in group g's row range and each output
channel n::
out[t, n] = (sum_k a[t, k] * b[expert_ids[g], n, k])
* a_scale[t] * b_scale[expert_ids[g], n]On SM100 with a bf16 output and static expert shapes this runs the persistent tensor-core grouped GEMM and applies both scales in its compute epilogue, so the scale is applied to the bf16-rounded sum. Everywhere else it runs a naive CUDA-core kernel that accumulates in fp32 and applies a single post-reduction scale.
Parameters:
- c_type (
DType): Output dtype (typicallybfloat16). - a_type (
DType): Activation dtype (float8_e4m3fn). - b_type (
DType): Weight dtype (float8_e4m3fn). - a_scales_type (
DType): Per-token activation scale dtype (float32). - b_scales_type (
DType): Per-channel weight scale dtype (float32). - transpose_b (
Bool): Must beTrue;bis[E, N, K]. - pdl_level (
PDLLevel): Programmatic dependent launch level; abovePDLLevel.OFFthe kernel waits on its predecessor grid before its first read and releases dependent grids when it finishes.PDLLevel.NO_WAIT_OVERLAP_AT_ENDis rejected, since every input of this op is written by the grids before it (defaults toPDLLevel.ON).
Args:
- c (
TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type]): Output[total_tokens, N]. - a (
TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type]): Activations[total_tokens, K]. - b (
TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type]): Weights[num_experts, N, K](already transposed; K innermost). - a_scales (
TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type]): Per-token activation scales[total_tokens, 1]. - b_scales (
TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type]): Per-channel weight scales[num_experts, N, 1]. - a_offsets (
TileTensor[.uint32, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type]): Ragged row offsets[num_active_experts + 1]. - expert_ids (
TileTensor[.int32, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type]): Real expert ids[num_active_experts]. - max_num_tokens_per_expert (
Int): Max tokens routed to any active expert. - num_active_experts (
Int): Number of active experts (groups). - ctx (
DeviceContext): Device context.