IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

blackwell_tma_umma_warp_specialized_kernel

def blackwell_tma_umma_warp_specialized_kernel[a_type: DType, b_type: DType, c_type: DType, expert_m: Int, a_tile_shape: Coord[*a_tile_shape.element_types], a_desc_shape: Coord[*a_desc_shape.element_types], b_tile_shape: Coord[*b_tile_shape.element_types], b_desc_shape: Coord[*b_desc_shape.element_types], c_tile_shape: Coord[*c_tile_shape.element_types], c_desc_shape: Coord[*c_desc_shape.element_types], block_tile_shape: Coord[Int, Int, Int], mma_shape: Coord[Int, Int, Int], cluster_shape: Coord[Int, Int, Int], num_pipeline_stages: Int, num_accum_pipeline_stages: Int, num_output_stages: Int = Int(2), output_tile_shape: Coord[*output_tile_shape.element_types] = Coord(), transpose_b: Bool = True, a_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, b_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, c_swizzle: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_128B, cta_group: Int = Int(2), EpilogueFnType: ElementwiseEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"_no_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], has_epilogue_fn: Bool = False, ComputeFnType: ElementwiseComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"identity_compute_fn[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], has_compute_fn: Bool = False, a_plane_splits: Tuple[Int, Int] = Tuple(Int(0), Int(0)), transpose_c: Bool = False, use_tma: Bool = True, K_actual: Int = Int(0), a_gmem_layout: TensorLayout = Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], b_gmem_layout: TensorLayout = Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], pdl_level: PDLLevel = PDLLevel()](expert_usage_stats: ImmPointer[UInt32, ImmutAnyOrigin], a_tma_op: TMATensorTile[a_type, a_tile_shape, a_desc_shape], expert_ids: ImmPointer[Int32, ImmutAnyOrigin], b_tma_op: TMATensorTile[b_type, b_tile_shape, b_desc_shape], b_offsets: ImmPointer[UInt32, ImmutAnyOrigin], c_tma_op: TMATensorTile[c_type, c_tile_shape, c_desc_shape], c_ptr: MutPointer[Scalar[c_type], MutAnyOrigin], mnk: StaticTuple[UInt32, Int(3)], a_gmem: TileTensor[a_type, a_gmem_layout, ImmutAnyOrigin], b_gmem: TileTensor[b_type, b_gmem_layout, ImmutAnyOrigin], epilogue_fn: EpilogueFnType, compute_fn: ComputeFnType)

Implements the warp-specialized persistent grouped GEMM kernel for SM100.

Splits the CTA warps into main-load, MMA, and epilogue roles that cooperate through mbarriers and a tile scheduler to iterate over grouped GEMM work tiles, streaming A and B through shared memory via TMA and accumulating into tensor memory before storing C back to global memory.

Parameters:

  • ​a_type (DType): Element type of the A operand matrix.
  • ​b_type (DType): Element type of the B operand matrix.
  • ​c_type (DType): Element type of the C output matrix.
  • ​expert_m (Int): Static M dimension of each expert's local slice, used as the C stride and row bound for contiguous row-major C output.
  • ​a_tile_shape (Coord[*element_types]): Element shape of one A TMA tile.
  • ​a_desc_shape (Coord[*element_types]): Descriptor shape of the A TMA tile, used to compute per- load element counts and row stride.
  • ​b_tile_shape (Coord[*element_types]): Element shape of one B TMA tile.
  • ​b_desc_shape (Coord[*element_types]): Descriptor shape of the B TMA tile, used to compute per- load element counts and row stride.
  • ​c_tile_shape (Coord[*element_types]): Element shape of one C TMA tile.
  • ​c_desc_shape (Coord[*element_types]): Descriptor shape of the C TMA tile.
  • ​block_tile_shape (Coord[Int, Int, Int]): Block tile shape [BM, BN, BK] partitioning the GEMM into work tiles.
  • ​mma_shape (Coord[Int, Int, Int]): MMA instruction shape [MMA_M, MMA_N, MMA_K].
  • ​cluster_shape (Coord[Int, Int, Int]): Thread block cluster shape [CLUSTER_M, CLUSTER_N, CLUSTER_D].
  • ​num_pipeline_stages (Int): Number of TMA producer and consumer pipeline stages used to double-buffer A and B shared-memory tiles.
  • ​num_accum_pipeline_stages (Int): Number of accumulator pipeline stages used to double-buffer TMEM-to-global-memory C stores.
  • ​num_output_stages (Int): Number of output shared-memory buffer stages for C (defaults to 2).
  • ​output_tile_shape (Coord[*element_types]): Output tile shape [OUT_M, OUT_N] for one C shared- memory stage (defaults to coord[128, 32]).
  • ​transpose_b (Bool): Whether B is stored transposed in global memory (defaults to True).
  • ​a_swizzle (TensorMapSwizzle): TMA swizzle mode applied to A shared-memory tiles (defaults to SWIZZLE_128B).
  • ​b_swizzle (TensorMapSwizzle): TMA swizzle mode applied to B shared-memory tiles (defaults to SWIZZLE_128B).
  • ​c_swizzle (TensorMapSwizzle): TMA swizzle mode applied to C shared-memory tiles (defaults to SWIZZLE_128B).
  • ​cta_group (Int): Number of CTAs cooperating per MMA along the M dimension (defaults to 2).
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn.
  • ​has_epilogue_fn (Bool): Whether epilogue_fn stores C in place of the TMA store (defaults to False). Launch with epilogue_fn as the first host_args element.
  • ​ComputeFnType (ElementwiseComputeFn): Type of compute_fn.
  • ​has_compute_fn (Bool): Whether compute_fn maps C in registers ahead of the TMA store (defaults to False). Launch with compute_fn as the second host_args element.
  • ​a_plane_splits (Tuple[Int, Int]): Per-plane split sizes for fused LoRA QKV A-plane row offsetting; (0, 0) disables it (defaults to (0, 0)).
  • ​transpose_c (Bool): Whether to transpose the C output tile in shared memory before the TMA store (defaults to False).
  • ​use_tma (Bool): Whether to use TMA for A and B loads; when False, use the CUDA-core fallback (defaults to True).
  • ​K_actual (Int): Actual K dimension in elements for the CUDA-core fallback when use_tma is False (defaults to 0).
  • ​a_gmem_layout (TensorLayout): Layout of the A global-memory tensor, used only by the CUDA-core fallback (defaults to type_of(row_major[1, 1]())).
  • ​b_gmem_layout (TensorLayout): Layout of the B global-memory tensor, used only by the CUDA-core fallback (defaults to type_of(row_major[1, 1]())).
  • ​pdl_level (PDLLevel): Programmatic dependent launch level; above PDLLevel.OFF the kernel waits for the predecessor grid immediately before reading expert_usage_stats. PDLLevel.OVERLAP_AT_BEGINNING releases dependent grids after the TMEM allocation, any other level at the end of the body (defaults to PDLLevel.OVERLAP_AT_BEGINNING).

Args:

Was this page helpful?