For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
naive_blockwise_scaled_fp8_grouped_matmul
def naive_blockwise_scaled_fp8_grouped_matmul[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, a_offsets_type: DType, expert_ids_type: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, a_scale_layout: TensorLayout, b_scale_layout: TensorLayout, a_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, //, BLOCK_DIM_N: Int = Int(32), BLOCK_DIM_M: Int = Int(16), transpose_b: Bool = True, scales_granularity_mnk: Optional[Coord[Int, Int, Int]] = None](c: TileTensor[c_type, c_layout, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, a_layout, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, b_layout, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, a_scale_layout, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, b_scale_layout, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], a_offsets: TileTensor[a_offsets_type, a_offsets_layout, Engine=a_offsets.Engine, address_space=a_offsets.address_space, linear_idx_type=a_offsets.linear_idx_type], expert_ids: TileTensor[expert_ids_type, expert_ids_layout, Engine=expert_ids.Engine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], max_num_tokens_per_expert: Int, num_active_experts: Int, ctx: DeviceContext)
Dispatches the naive blockwise scaled FP8 grouped matmul kernel on the GPU.
Enqueues naive_blockwise_scaled_fp8_grouped_matmul_kernel with one
expert per grid-Z slice, tiling the per-expert M_local x N output
with BLOCK_DIM_M x BLOCK_DIM_N thread blocks.
Args:
- c (
TileTensor[c_type, c_layout, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Rank-2 output accumulator tensor holding all expert outputs. - a (
TileTensor[a_type, a_layout, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): Rank-2 FP8 input matrix in K-major format. - b (
TileTensor[b_type, b_layout, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): Rank-3 FP8 weight tensor indexed by expert, in K-major format. - a_scales (
TileTensor[a_scales_type, a_scale_layout, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]): Per-block scales forain M-major format. - b_scales (
TileTensor[b_scales_type, b_scale_layout, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]): Per-block scales forbindexed by expert. - a_offsets (
TileTensor[a_offsets_type, a_offsets_layout, Engine=a_offsets.Engine, address_space=a_offsets.address_space, linear_idx_type=a_offsets.linear_idx_type]): Prefix-sum offsets delimiting each expert's rows ina. - expert_ids (
TileTensor[expert_ids_type, expert_ids_layout, Engine=expert_ids.Engine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type]): Expert id (or-1to skip) for each grid-Z slice. - max_num_tokens_per_expert (
Int): Maximum row count assigned to any single expert. - num_active_experts (
Int): Number of active experts to dispatch. - ctx (
DeviceContext): Device context used to enqueue the kernel.