For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
naive_blockwise_scaled_fp8_grouped_matmul_kernel
def naive_blockwise_scaled_fp8_grouped_matmul_kernel[c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, a_scale_layout: TensorLayout, b_scale_layout: TensorLayout, a_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, a_offsets_type: DType, expert_ids_type: DType, accum_type: DType, transpose_b: Bool = True, scales_granularity_mnk: Optional[Coord[Int, Int, Int]] = None](c: TileTensor[c_type, c_layout, MutAnyOrigin], a: TileTensor[a_type, a_layout, ImmutAnyOrigin], b: TileTensor[b_type, b_layout, ImmutAnyOrigin], a_offsets: TileTensor[a_offsets_type, a_offsets_layout, ImmutAnyOrigin], expert_ids: TileTensor[expert_ids_type, expert_ids_layout, ImmutAnyOrigin], a_scales: TileTensor[a_scales_type, a_scale_layout, ImmutAnyOrigin], b_scales: TileTensor[b_scales_type, b_scale_layout, ImmutAnyOrigin])
Computes one output element per thread for the naive blockwise scaled FP8 grouped matmul GPU kernel.
Each thread handles a single (m_local, n) output element within one
expert's tile, determined by block_idx.z and the a_offsets prefix
sum. The thread loops over the K dimension, loading the expert's a row
and b slice along with their per-block scale factors, accumulates the
scaled product, and applies the optional elementwise epilogue before
storing the result.
Args:
- c (
TileTensor[c_type, c_layout, MutAnyOrigin]): Rank-2 output accumulator tensor holding all expert outputs. - a (
TileTensor[a_type, a_layout, ImmutAnyOrigin]): Rank-2 FP8 input matrix in K-major format. - b (
TileTensor[b_type, b_layout, ImmutAnyOrigin]): Rank-3 FP8 weight tensor indexed by expert, in K-major format. - a_offsets (
TileTensor[a_offsets_type, a_offsets_layout, ImmutAnyOrigin]): Prefix-sum offsets delimiting each expert's rows ina. - expert_ids (
TileTensor[expert_ids_type, expert_ids_layout, ImmutAnyOrigin]): Expert id (or-1to skip) for each grid-Z slice. - a_scales (
TileTensor[a_scales_type, a_scale_layout, ImmutAnyOrigin]): Per-block scales forain M-major format. - b_scales (
TileTensor[b_scales_type, b_scale_layout, ImmutAnyOrigin]): Per-block scales forbindexed by expert.