IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_matmul_sm100_blockwise_scaled_fp8

def grouped_matmul_sm100_blockwise_scaled_fp8[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, a_offsets_type: DType, expert_ids_type: DType, transpose_b: Bool, //, *, config: MatmulConfig[a_type, b_type, c_type, transpose_b]](c: TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[a_offsets_type, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[expert_ids_type, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, ctx: DeviceContext)

Launches the basic (non-persistent) SM100 blockwise-scaled FP8 grouped GEMM kernel.

Builds TMA descriptors for A and B and enqueues matmul_sm100_grouped_blockwise_scaled_fp8_1d2d_kernel with a grid of (N/BN, max_tokens/BM, num_active_experts) blocks.

Parameters:

  • ​c_type (DType): Element type of the output C tensor (inferred).
  • ​a_type (DType): Element type of the input A tensor (inferred). Must be float8_e4m3fn.
  • ​b_type (DType): Element type of the input B tensor (inferred). Must be float8_e4m3fn.
  • ​a_scales_type (DType): Element type of the a_scales tensor (inferred).
  • ​b_scales_type (DType): Element type of the b_scales tensor (inferred).
  • ​a_offsets_type (DType): Element type of the a_offsets tensor (inferred).
  • ​expert_ids_type (DType): Element type of the expert_ids tensor (inferred).
  • ​transpose_b (Bool): Whether B is stored transposed (inferred). Must be True.
  • ​config (MatmulConfig[a_type, b_type, c_type, transpose_b]): Matmul configuration specifying block tile shape, MMA shape, and TMA swizzle modes.

Args:

def grouped_matmul_sm100_blockwise_scaled_fp8[EpilogueFnType: ElementwiseEpilogueFn, c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, a_offsets_type: DType, expert_ids_type: DType, transpose_b: Bool, //, *, config: MatmulConfig[a_type, b_type, c_type, transpose_b]](c: TileTensor[c_type, Engine=c.Engine, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Engine=a_scales.Engine, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Engine=b_scales.Engine, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[a_offsets_type, Engine=a_offsets.Engine, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[expert_ids_type, Engine=expert_ids.Engine, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, epilogue_fn: EpilogueFnType, ctx: DeviceContext)

Same as the plain overload, storing each output through epilogue_fn.

The kernel never writes c; it only supplies the output shape.

Parameters:

  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​c_type (DType): Element type of the output C tensor (inferred).
  • ​a_type (DType): Element type of the input A tensor (inferred).
  • ​b_type (DType): Element type of the input B tensor (inferred).
  • ​a_scales_type (DType): Element type of the a_scales tensor (inferred).
  • ​b_scales_type (DType): Element type of the b_scales tensor (inferred).
  • ​a_offsets_type (DType): Element type of the a_offsets tensor (inferred).
  • ​expert_ids_type (DType): Element type of the expert_ids tensor (inferred).
  • ​transpose_b (Bool): Whether B is stored transposed (inferred).
  • ​config (MatmulConfig[a_type, b_type, c_type, transpose_b]): Matmul configuration specifying block tile shape, MMA shape, and TMA swizzle modes.

Args:

Was this page helpful?