For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
TileScheduler
struct TileScheduler[num_stages: Int, reduction_tile_shape: IndexList[Int(3)], cluster_shape: Coord[Int, Int, Int] = Coord(Int(1), Int(1), Int(1)), rasterize_order: RasterOrder = RasterOrder.AlongM, block_swizzle_size: Int = Int(8), num_split_k: Int = Int(1)]
Tile scheduler that partitions the K dimension into multiple splits.
Wraps the B200 tile scheduler and remaps its work info into split-K coordinates, managing per-split workspace reductions and lock-based synchronization so that CTAs contributing to the same output tile accumulate correctly.
Parameters
- num_stages (
Int): Number of CLC pipeline stages for work distribution. - reduction_tile_shape (
IndexList[Int(3)]): The(BM, MMA_N, BK)per-block reduction tile shape used for the split-K workspace layout. - cluster_shape (
Coord[Int, Int, Int]): Cluster tile counts as(m, n, k)(defaults to(1, 1, 1)). - rasterize_order (
RasterOrder): Order CLC rasterizes tiles across the cluster grid (defaults toRasterOrder.AlongM). - block_swizzle_size (
Int): Block swizzle factor for tile remapping, one of 0, 1, 2, 4, or 8 (defaults to 8). - num_split_k (
Int): Number of splits the K dimension is partitioned into (defaults to 1).
Fields
- locks_ptr (
MutPointer[Int32, MutAnyOrigin]): - scheduler (
TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].UnderlyingScheduler): - total_k_tiles (
UInt32): - k_tiles_per_split (
UInt32): - throttle_pipeline (
TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].ThrottlePipeline):
Implemented traits
AnyType,
Copyable,
Deinitable,
ImplicitlyCopyable,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
BK
comptime BK = reduction_tile_shape[Int(2)]
BM
comptime BM = reduction_tile_shape[Int(0)]
ClcBarrierArray
comptime ClcBarrierArray = TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].UnderlyingScheduler.ClcBarrierArray
ClcResponseArray
comptime ClcResponseArray = TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].UnderlyingScheduler.ClcResponseArray
MMA_N
comptime MMA_N = reduction_tile_shape[Int(1)]
PARALLEL_BARRIER_ID
comptime PARALLEL_BARRIER_ID = 4
Named barrier of the epilogue warpgroup in the parallel reduction (ids 0 and 1 belong to the output writer and the MMA handshake).
ROW_SIZE
comptime ROW_SIZE = reduction_tile_shape[Int(1)] if (reduction_tile_shape[Int(0)] == Int(128)) else (reduction_tile_shape[Int(1)] // Int(2))
ThrottleBarrierArray
comptime ThrottleBarrierArray = TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].UnderlyingScheduler.ThrottleBarrierArray
ThrottlePipeline
comptime ThrottlePipeline = TileScheduler[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k].UnderlyingScheduler.ThrottlePipeline
UnderlyingScheduler
comptime UnderlyingScheduler = TileScheduler[num_stages, cluster_shape, rasterize_order, block_swizzle_size]
WorkspaceTileLayout
comptime WorkspaceTileLayout = static_row_major[reduction_tile_shape[Int(0)], reduction_tile_shape[Int(1)]]
Methods
__init__
def __init__(cluster_dim: StaticTuple[Int32, Int(3)], mnk: StaticTuple[UInt32, Int(3)], clc_response: SMemArray[UInt128, num_stages], clc_full: SMemArray[SharedMemBarrier, num_stages], clc_empty: SMemArray[SharedMemBarrier, num_stages], clc_throttle: SMemArray[SharedMemBarrier, (num_stages * Int(2))], locks_ptr: MutPointer[UInt8, MutAnyOrigin], k_group_size: Int = Int(1)) -> Self
Initialize from typed barrier arrays.
Args:
- cluster_dim (
StaticTuple[Int32, Int(3)]): Grid cluster dimensions as(m, n, k)used for fast division rasterization. - mnk (
StaticTuple[UInt32, Int(3)]): Problem dimensions as(M, N, K);Ksets the total K tile count. - clc_response (
SMemArray[UInt128, num_stages]): Shared memory array storing CLC response payloads. - clc_full (
SMemArray[SharedMemBarrier, num_stages]): Barriers signaled when CLC response data is ready. - clc_empty (
SMemArray[SharedMemBarrier, num_stages]): Barriers signaled when a response slot is available. - clc_throttle (
SMemArray[SharedMemBarrier, (num_stages * Int(2))]): Barriers for the throttle pipeline pacing the scheduler against the load warp. - locks_ptr (
MutPointer[UInt8, MutAnyOrigin]): Pointer to the per-output-tile lock buffer used to synchronize split-K reductions across CTAs. - k_group_size (
Int): K tiles the producer and MMA loops consume per step; every split but the last covers a whole number of these.
init_throttle_barriers
static def init_throttle_barriers(storage_ptr: SMemPtr[SharedMemBarrier], producer_arv_count: Int32, consumer_arv_count: Int32)
Initialize throttle pipeline barriers. Called once by elect_one thread.
Args:
- storage_ptr (
SMemPtr[SharedMemBarrier]): Pointer to shared memory backing the throttle barriers. - producer_arv_count (
Int32): Arrival count the producer waits for per stage. - consumer_arv_count (
Int32): Arrival count the consumer waits for per stage.
convert_to_splitk_work_info
initial_work_info
advance_to_next_work
def advance_to_next_work(self, mut clc_state: PipelineState[num_stages]) -> PipelineState[num_stages]
Returns:
fetch_next_work
def fetch_next_work(self, work_info: WorkInfo, consumer_state: PipelineState[num_stages]) -> WorkInfo
Returns:
throttle_signal
def throttle_signal(mut self, is_first_cta_in_cluster: Bool)
Signal CLC throttle if this is the first CTA in cluster.
Args:
- is_first_cta_in_cluster (
Bool): Only first CTA signals to avoid duplicates.
wait_and_advance_work
def wait_and_advance_work[work_origin: MutOrigin, //](self, ref[num_stages] work_info: WorkInfo, mut consumer_state: PipelineState[num_stages]) -> WaitAndAdvanceContextSplitK[work_origin]
Wait for next work from CLC and advance (Split-K).
Encapsulates the CLC barrier wait (called on scheduler directly).
Usage: with scheduler.wait_and_advance_work(work_info, state) as current: do_mma(current) # After: work_info updated to next value
Parameters:
- work_origin (
MutOrigin): Memory origin of thework_inforeference (inferred).
Args:
- work_info (
WorkInfo): Reference to the current work descriptor, updated to the next work item on context exit. - consumer_state (
PipelineState[num_stages]): CLC consumer pipeline state, advanced after the fetch.
Returns:
work_iterator
def work_iterator(self) -> WorkIteratorSplitK[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k]
Create a per-warp work iterator that owns work_info internally. Throttle pipeline is obtained from the scheduler.
Returns:
scheduler_iterator
def scheduler_iterator(self) -> SchedulerWorkIteratorSplitK[num_stages, reduction_tile_shape, cluster_shape, rasterize_order, block_swizzle_size, num_split_k]
Create iterator for Scheduler warp (owns work_info and both states). Throttle pipeline is obtained from the scheduler.
Returns:
is_last_split
output_tile_index
store_to_workspace
def store_to_workspace[accum_type: DType, workspace_layout: TensorLayout, workspace_engine: TensorEngine, /, *, do_reduction: Bool = False, write_back: Bool = False](self, tmem: TmemAddress, reduction_workspace: TileTensor[accum_type, workspace_layout, MutAnyOrigin, Engine=workspace_engine], epilogue_thread_idx: Int, reduction_tile_idx: UInt32)
store_partial_slot
def store_partial_slot[accum_type: DType](self, tmem: TmemAddress, workspace: MutPointer[Scalar[accum_type], MutAnyOrigin], epilogue_thread_idx: Int, work_info: WorkInfo)
Stores this split's accumulator to its own workspace slot.
Slot tile * num_split_k + split holds the BM x MMA_N fp32 tile in
16-byte pieces, piece q * BM + row holding columns 4q .. 4q + 3 of
the row, so a warp's 32 rows of one piece column are contiguous.
Parameters:
- accum_type (
DType): Accumulator dtype.
Args:
- tmem (
TmemAddress): TMEM address of the accumulator stage. - workspace (
MutPointer[Scalar[accum_type], MutAnyOrigin]): Base of the fp32 workspace. - epilogue_thread_idx (
Int): Thread index in the epilogue warpgroup, which is the accumulator row it holds. - work_info (
WorkInfo): The split's work tile.
reduce_slice_to_output
def reduce_slice_to_output[accum_type: DType, c_type: DType, //, transpose_c: Bool, pieces_per_batch: Int](self, workspace: MutPointer[Scalar[accum_type], MutAnyOrigin], counters: SplitKTileCounters, c: MutPointer[Scalar[c_type], MutAnyOrigin], c_row_stride: Int, epilogue_thread_idx: Int, work_info: WorkInfo, shape: Tuple[UInt32, UInt32])
Publishes this split's slot, waits for the tile's other splits and writes slice split of the summed tile to C.
The tile's 16-byte pieces are cut into num_split_k slices, one per
split. Every split sums its slice over all the tile's slots in split
order, so the output does not depend on which split finished first,
and the reduction is spread over the tile's CTAs. The tile's counter
word counts arrivals in its low half and finished readers in its
high half; the last reader resets it, so it is zero again for the
next launch on this tile.
Parameters:
- accum_type (
DType): Accumulator dtype (inferred). - c_type (
DType): Output dtype (inferred). - transpose_c (
Bool): Whether the kernel computes C transposed (swapped A and B): accumulator rows are then output columns. - pieces_per_batch (
Int): Pieces per thread whose loads from every slot are in flight together; bounds the registers held.
Args:
- workspace (
MutPointer[Scalar[accum_type], MutAnyOrigin]): Base of the fp32 workspace. - counters (
SplitKTileCounters): The kernel's tile counters (splitk_tile_counters). - c (
MutPointer[Scalar[c_type], MutAnyOrigin]): Base of the row-major output. - c_row_stride (
Int): Elements between consecutive rows of the output. - epilogue_thread_idx (
Int): Thread index in the epilogue warpgroup. - work_info (
WorkInfo): The split's work tile. - shape (
Tuple[UInt32, UInt32]): The output's (rows, columns).
reduction
def reduction[accum_type: DType, workspace_layout: TensorLayout, workspace_engine: TensorEngine](self, reduction_workspace: TileTensor[accum_type, workspace_layout, MutAnyOrigin, Engine=workspace_engine], tmem: TmemAddress, epilogue_thread_idx: Int, work_info: WorkInfo) -> Bool
Returns:
wait_eq
static def wait_eq(lock_ptr: MutPointer[Int32, MutAnyOrigin], barrier_id: Int32, barrier_group_thread_idx: Int, lock_idx: UInt32, val: UInt32)
wait_lt
static def wait_lt(lock_ptr: MutPointer[Int32, MutAnyOrigin], barrier_id: Int32, barrier_group_thread_idx: Int, lock_idx: UInt32, count: UInt32)
arrive_set
static def arrive_set(lock_ptr: MutPointer[Int32, MutAnyOrigin], barrier_id: Int32, barrier_group_thread_idx: Int, lock_idx: UInt32, val: UInt32)