For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
blackwell_tma_umma_warp_specialized_kernel
def blackwell_tma_umma_warp_specialized_kernel[a_type: DType, b_type: DType, c_type: DType, expert_m: Int, a_tile_shape: Coord[*a_tile_shape.element_types], a_desc_shape: Coord[*a_desc_shape.element_types], b_tile_shape: Coord[*b_tile_shape.element_types], b_desc_shape: Coord[*b_desc_shape.element_types], c_tile_shape: Coord[*c_tile_shape.element_types], c_desc_shape: Coord[*c_desc_shape.element_types], block_tile_shape: Coord[Int, Int, Int], mma_shape: Coord[Int, Int, Int], cluster_shape: Coord[Int, Int, Int], num_pipeline_stages: Int, num_accum_pipeline_stages: Int, num_output_stages: Int = Int(2), output_tile_shape: Coord[*output_tile_shape.element_types] = Coord(), transpose_b: Bool = True, a_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, b_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, c_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, cta_group: Int = Int(2), EpilogueFnType: ElementwiseEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"_no_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], has_epilogue_fn: Bool = False, ComputeFnType: ElementwiseComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"identity_compute_fn[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], has_compute_fn: Bool = False, a_plane_splits: Tuple[Int, Int] = Tuple(Int(0), Int(0)), transpose_c: Bool = False, use_tma: Bool = True, K_actual: Int = Int(0), a_gmem_layout: TensorLayout = Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], b_gmem_layout: TensorLayout = Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], pdl_level: PDLLevel = PDLLevel()](expert_usage_stats: ImmPointer[UInt32, ImmutAnyOrigin], a_tma_op: TMATensorTile[a_type, a_tile_shape, a_desc_shape], expert_ids: ImmPointer[Int32, ImmutAnyOrigin], b_tma_op: TMATensorTile[b_type, b_tile_shape, b_desc_shape], b_offsets: ImmPointer[UInt32, ImmutAnyOrigin], c_tma_op: TMATensorTile[c_type, c_tile_shape, c_desc_shape], c_ptr: MutPointer[Scalar[c_type], MutAnyOrigin], mnk: StaticTuple[UInt32, Int(3)], a_gmem: TileTensor[a_type, a_gmem_layout, ImmutAnyOrigin], b_gmem: TileTensor[b_type, b_gmem_layout, ImmutAnyOrigin], epilogue_fn: EpilogueFnType, compute_fn: ComputeFnType)
Implements the warp-specialized persistent grouped GEMM kernel for SM100.
Splits the CTA warps into main-load, MMA, and epilogue roles that cooperate through mbarriers and a tile scheduler to iterate over grouped GEMM work tiles, streaming A and B through shared memory via TMA and accumulating into tensor memory before storing C back to global memory.
Parameters:
- a_type (
DType): Element type of the A operand matrix. - b_type (
DType): Element type of the B operand matrix. - c_type (
DType): Element type of the C output matrix. - expert_m (
Int): Static M dimension of each expert's local slice, used as the C stride and row bound for contiguous row-major C output. - a_tile_shape (
Coord[*element_types]): Element shape of one A TMA tile. - a_desc_shape (
Coord[*element_types]): Descriptor shape of the A TMA tile, used to compute per- load element counts and row stride. - b_tile_shape (
Coord[*element_types]): Element shape of one B TMA tile. - b_desc_shape (
Coord[*element_types]): Descriptor shape of the B TMA tile, used to compute per- load element counts and row stride. - c_tile_shape (
Coord[*element_types]): Element shape of one C TMA tile. - c_desc_shape (
Coord[*element_types]): Descriptor shape of the C TMA tile. - block_tile_shape (
Coord[Int, Int, Int]): Block tile shape[BM, BN, BK]partitioning the GEMM into work tiles. - mma_shape (
Coord[Int, Int, Int]): MMA instruction shape[MMA_M, MMA_N, MMA_K]. - cluster_shape (
Coord[Int, Int, Int]): Thread block cluster shape[CLUSTER_M, CLUSTER_N, CLUSTER_D]. - num_pipeline_stages (
Int): Number of TMA producer and consumer pipeline stages used to double-buffer A and B shared-memory tiles. - num_accum_pipeline_stages (
Int): Number of accumulator pipeline stages used to double-buffer TMEM-to-global-memory C stores. - num_output_stages (
Int): Number of output shared-memory buffer stages for C (defaults to 2). - output_tile_shape (
Coord[*element_types]): Output tile shape[OUT_M, OUT_N]for one C shared- memory stage (defaults tocoord[128, 32]). - transpose_b (
Bool): Whether B is stored transposed in global memory (defaults to True). - a_swizzle (
TensorMapSwizzle): TMA swizzle mode applied to A shared-memory tiles (defaults toSWIZZLE_128B). - b_swizzle (
TensorMapSwizzle): TMA swizzle mode applied to B shared-memory tiles (defaults toSWIZZLE_128B). - c_swizzle (
TensorMapSwizzle): TMA swizzle mode applied to C shared-memory tiles (defaults toSWIZZLE_128B). - cta_group (
Int): Number of CTAs cooperating per MMA along the M dimension (defaults to 2). - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn. - has_epilogue_fn (
Bool): Whetherepilogue_fnstores C in place of the TMA store (defaults to False). Launch withepilogue_fnas the firsthost_argselement. - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn. - has_compute_fn (
Bool): Whethercompute_fnmaps C in registers ahead of the TMA store (defaults to False). Launch withcompute_fnas the secondhost_argselement. - a_plane_splits (
Tuple[Int, Int]): Per-plane split sizes for fused LoRA QKV A-plane row offsetting;(0, 0)disables it (defaults to(0, 0)). - transpose_c (
Bool): Whether to transpose the C output tile in shared memory before the TMA store (defaults to False). - use_tma (
Bool): Whether to use TMA for A and B loads; when False, use the CUDA-core fallback (defaults to True). - K_actual (
Int): Actual K dimension in elements for the CUDA-core fallback whenuse_tmais False (defaults to 0). - a_gmem_layout (
TensorLayout): Layout of the A global-memory tensor, used only by the CUDA-core fallback (defaults totype_of(row_major[1, 1]())). - b_gmem_layout (
TensorLayout): Layout of the B global-memory tensor, used only by the CUDA-core fallback (defaults totype_of(row_major[1, 1]())). - pdl_level (
PDLLevel): Programmatic dependent launch level; abovePDLLevel.OFFthe kernel waits for the predecessor grid immediately before readingexpert_usage_stats.PDLLevel.OVERLAP_AT_BEGINNINGreleases dependent grids after the TMEM allocation, any other level at the end of the body (defaults toPDLLevel.OVERLAP_AT_BEGINNING).
Args:
- expert_usage_stats (
ImmPointer[UInt32, ImmutAnyOrigin]): Pointer to per-expert usage stats; index 1 holds the active expert count. - a_tma_op (
TMATensorTile[a_type, a_tile_shape, a_desc_shape]): TMA tile descriptor for loading A tiles from global to shared memory. - expert_ids (
ImmPointer[Int32, ImmutAnyOrigin]): Pointer to the per-group expert id array; negative entries skip the tile. - b_tma_op (
TMATensorTile[b_type, b_tile_shape, b_desc_shape]): TMA tile descriptor for loading B tiles from global to shared memory. - b_offsets (
ImmPointer[UInt32, ImmutAnyOrigin]): Pointer to the per-expert B offset prefix sum, of lengthnum_active_experts + 1. - c_tma_op (
TMATensorTile[c_type, c_tile_shape, c_desc_shape]): TMA tile descriptor for storing C tiles from shared to global memory. - c_ptr (
MutPointer[Scalar[c_type], MutAnyOrigin]): Base pointer to the row-major C output tensor in global memory. - mnk (
StaticTuple[UInt32, Int(3)]):[M, N, K]dimensions of the grouped GEMM as unsigned 32-bit integers. - a_gmem (
TileTensor[a_type, a_gmem_layout, ImmutAnyOrigin]): A operand tensor used only by the CUDA-core fallback whenuse_tmais False. - b_gmem (
TileTensor[b_type, b_gmem_layout, ImmutAnyOrigin]): B operand tensor used only by the CUDA-core fallback whenuse_tmais False. - epilogue_fn (
EpilogueFnType): Stores C whenhas_epilogue_fnis set. - compute_fn (
ComputeFnType): Maps C whenhas_compute_fnis set.