For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
TileWriter
struct TileWriter[tma_origin: ImmOrigin, c_type: DType, c_tile_shape: Coord[*c_tile_shape.element_types], c_desc_shape: Coord[*c_desc_shape.element_types], //, a_type: DType, accum_type: DType, block_tile_shape: Tuple[Int, Int, Int], mma_shape: Tuple[Int, Int, Int], opc: OutputPipelineConfig, c_swizzle: TensorMapSwizzle, transpose_c: Bool, c_smem_dim0: Int, c_smem_dim1: Int, num_output_stages: Int, num_output_warps: Int, register_based_epilogue: Bool = True, batched: Bool = False, problem_n: Int = Int(0), num_peers: Int = Int(1), c_store_dead: Bool = False, p5_direct_scatter: Bool = False, p5_row_cache: Bool = False, SinkT: EpiloguePeerSink = NullPeerSink]
Output tile writer for SM100 matmul epilogue.
Stores pointer to TMA descriptor. SMEM tiles passed per-call.
Parameters are passed explicitly to work with both MatmulConfig and BlockScaledMatmulConfig.
The opc (OutputPipelineConfig) parameter must match the config used when constructing the OutputTilePipeline that provides OutputStage instances to the write() method.
Parameters
- tma_origin (
ImmOrigin): Memory origin of the TMA descriptor pointer (inferred). - c_type (
DType): Element dtype of the C output tensor (inferred). - c_tile_shape (
Coord[*element_types]): Per-tile shape of the C output (inferred). - c_desc_shape (
Coord[*element_types]): TMA descriptor shape for C (inferred). - a_type (
DType): Element dtype of the A input matrix. - accum_type (
DType): Accumulator dtype stored in TMEM. - block_tile_shape (
Tuple[Int, Int, Int]): Block tile shape as (BM, BN, BK). - mma_shape (
Tuple[Int, Int, Int]): MMA instruction shape as (MMA_M, MMA_N, MMA_K). - opc (
OutputPipelineConfig): Output pipeline config bundling accumulator stages, stage stride, and CTA group. - c_swizzle (
TensorMapSwizzle): TMA swizzle pattern for the C SMEM layout. - transpose_c (
Bool): Whether C is stored transposed. - c_smem_dim0 (
Int): Row dimension of the C SMEM tile. - c_smem_dim1 (
Int): Column dimension of the C SMEM tile. - num_output_stages (
Int): Number of C SMEM pipeline stages. - num_output_warps (
Int): Number of warps driving the output pipeline. - register_based_epilogue (
Bool): Whether the compute epilogue runs in registers (true) or SMEM (false) (defaults to True). - batched (
Bool): Whether the output uses 3D batched coordinates with a batch index (defaults to False). - problem_n (
Int): Logical N dimension used for row-major bounds checking in the slow path; 0 disables the N check (defaults to 0). - num_peers (
Int): Number of TMA store descriptors in the array; 1 for a local epilogue (defaults to 1). - c_store_dead (
Bool): Whether nothing reads the local C output, so the store is dead for the whole launch. The caller then supplies an empty C descriptor and an unbacked C tensor, so neither may be touched here (defaults to False). - p5_direct_scatter (
Bool): Whether to compile in the in-epilogue EP-combine peer scatter-send. Still gated at runtime byp3_control(defaults to False). - p5_row_cache (
Bool): Whether the peer send resolves each row's destination once per launch throughP3PeerSendConfig.row_base_ptrinstead of once per (row, tile) (defaults to False). - SinkT (
EpiloguePeerSink): A second destination for the output tile, replacing the local store when itsEnabledis set. Stateless, so it costs nothing to carry; defaults toNullPeerSink, which keeps the local store.
Fields
- c_tma_op (
TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].TmaOpPtr):
Implemented traits
AnyType,
Copyable,
Deinitable,
ImplicitlyCopyable,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
accum_tile_layout
comptime accum_tile_layout = row_major[TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].BM, TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].stageN]()
AccumTmemArray
comptime AccumTmemArray = TmemArrayType[accum_type, Layout[TypeList[ComptimeInt[TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].BM], ComptimeInt[TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].stageN]](), TypeList[ComptimeInt[c_smem_dim0 if transpose_c else c_smem_dim1], ComptimeInt[Int(1)]]()], TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].accum_tile_layout, EpilogueConfig.create(MMA_M=(load_from_mem mma_shape.__getitem_param__[Int(0)]()), MMA_N=(load_from_mem mma_shape.__getitem_param__[Int(1)]()), stageN=c_smem_dim0 if transpose_c else c_smem_dim1, cta_group=opc.cta_group, transpose_c=transpose_c, BM=(load_from_mem block_tile_shape.__getitem_param__[Int(0)]()), BN=(load_from_mem block_tile_shape.__getitem_param__[Int(1)]())).num_stages, cta_group=opc.cta_group]
bits
comptime bits = 256
BM
comptime BM = (load_from_mem block_tile_shape.__getitem_param__[Int(0)]())
BN
comptime BN = (load_from_mem block_tile_shape.__getitem_param__[Int(1)]())
cta_group
comptime cta_group = opc.cta_group
CTileArray
comptime CTileArray = SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]
data_paths
comptime data_paths = 16
epc
comptime epc = EpilogueConfig.create(MMA_M=TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].MMA_M, MMA_N=TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].MMA_N, stageN=TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].stageN, cta_group=opc.cta_group, transpose_c=transpose_c, BM=TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].BM, BN=TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].BN)
epilogue_dtype
comptime epilogue_dtype = TileWriter.get_epilogue_dtype()
fragment_size
comptime fragment_size = (Int(128) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
is_lower_frag_required
comptime is_lower_frag_required = TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].epc.is_lower_frag_required
MMA_M
comptime MMA_M = (load_from_mem mma_shape.__getitem_param__[Int(0)]())
MMA_N
comptime MMA_N = (load_from_mem mma_shape.__getitem_param__[Int(1)]())
N_dim
comptime N_dim = Int(0) if transpose_c else Int(1)
needs_sync
comptime needs_sync = False
num_accum_pipeline_stages
comptime num_accum_pipeline_stages = opc.num_stages
num_stages
comptime num_stages = TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].epc.num_stages
P5_SEND_DRAIN_BARRIER_ID
comptime P5_SEND_DRAIN_BARRIER_ID = 6
p5_send_enabled
comptime p5_send_enabled = p5_direct_scatter if (get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)]() > Int(0)) else (get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)]() > Int(0))
P5_SEND_RELEASE_BARRIER_ID
comptime P5_SEND_RELEASE_BARRIER_ID = 5
P5_SEND_THREADS
comptime P5_SEND_THREADS = (get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)]() * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
p5_send_warps
comptime p5_send_warps = get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)]()
p5_stamp_window
comptime p5_stamp_window = get_defined_bool[StringSpan("P5_STAMP_WINDOW")]()
p5_trace_epi
comptime p5_trace_epi = get_defined_bool[StringSpan("P5_TRACE_EPI")]()
P5SendDrain
comptime P5SendDrain = WarpGroupBarrier[(Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, num_output_warps)) + Int((mul get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)](), DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size))), Int(6)]
P5SendRelease
comptime P5SendRelease = WarpGroupBarrier[(Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size, num_output_warps)) + Int((mul get_defined_int[StringSpan("P5_SEND_WARPS"), Int(0)](), DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size))), Int(5)]
rep
comptime rep = (TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].stageN // Int(8))
rep_frag_size
comptime rep_frag_size = ((Int(128) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size) * TileWriter[a_type, accum_type, block_tile_shape, mma_shape, opc, c_swizzle, transpose_c, c_smem_dim0, c_smem_dim1, num_output_stages, num_output_warps, register_based_epilogue, batched, problem_n, num_peers, c_store_dead, p5_direct_scatter, p5_row_cache, SinkT].rep)
Stage
comptime Stage = OutputStage[opc]
stage_contiguous_size
comptime stage_contiguous_size = c_smem_dim1
stage_stride_cols
comptime stage_stride_cols = opc.stage_stride_cols
stageN
comptime stageN = c_smem_dim0 if transpose_c else c_smem_dim1
TmaOp
comptime TmaOp = TMATensorTile[c_type, c_tile_shape, c_desc_shape]
TmaOpArray
comptime TmaOpArray = Array[TMATensorTile[c_type, c_tile_shape, c_desc_shape], num_peers]
TmaOpArrayPtr
comptime TmaOpArrayPtr = Pointer[Array[TMATensorTile[c_type, c_tile_shape, c_desc_shape], num_peers], tma_origin]
TmaOpPtr
comptime TmaOpPtr = Pointer[TMATensorTile[c_type, c_tile_shape, c_desc_shape], tma_origin]
WarpRole
comptime WarpRole = WarpRole1D1D[num_epi_warps=num_output_warps]
Methods
__init__
def __init__(c_tma_op: Pointer[TMATensorTile[c_type, c_tile_shape, c_desc_shape], tma_origin]) -> Self
Initialize with pointer to TMA descriptor.
Args:
- c_tma_op (
Pointer[TMATensorTile[c_type, c_tile_shape, c_desc_shape], tma_origin]): Pointer to the TMA store descriptor for C.
def __init__(c_tma_ops: Pointer[Array[TMATensorTile[c_type, c_tile_shape, c_desc_shape], num_peers], tma_origin]) -> Self
Initialize from the c_tma_ops array pointer (TileWriterLike).
The standard local store targets a single descriptor, so this uses
element [0] of the array. Unifies construction with the
reduce-scatter writer, which retains all num_peers descriptors.
Args:
- c_tma_ops (
Pointer[Array[TMATensorTile[c_type, c_tile_shape, c_desc_shape], num_peers], tma_origin]): Pointer to the array of TMA store descriptors for C; element[0]is used for the local store.
get_epilogue_dtype
write
def write(self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32], shape: Tuple[UInt32, UInt32], elect_one_warp: Bool)
Write accumulated results to global memory (2D coords).
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): SMEM tile array for the C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32]): (m_tile, n_tile) tile coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - elect_one_warp (
Bool): Whether this warp is elected for coordination.
def write[ComputeFnType: ElementwiseComputeFn](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32], shape: Tuple[UInt32, UInt32], elect_one_warp: Bool, compute_fn: ComputeFnType)
Write accumulated results to global memory, applying compute_fn.
Only the register-based epilogue is supported.
Parameters:
- ComputeFnType (
ElementwiseComputeFn): Type of the compute epilogue closure.
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): SMEM tile array for the C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32]): (m_tile, n_tile) tile coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - elect_one_warp (
Bool): Whether this warp is elected for coordination. - compute_fn (
ComputeFnType): Element-wise epilogue applied to each output value.
def write[EpilogueFnType: ElementwiseEpilogueFn](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32], shape: Tuple[UInt32, UInt32], elect_one_warp: Bool, epilogue_fn: EpilogueFnType)
Write accumulated results to global memory through epilogue_fn.
Parameters:
- EpilogueFnType (
ElementwiseEpilogueFn): Type of the elementwise epilogue closure.
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): SMEM tile array for the C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32]): (m_tile, n_tile) tile coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - elect_one_warp (
Bool): Whether this warp is elected for coordination. - epilogue_fn (
EpilogueFnType): Stores each output value to global memory.
write_batched
def write_batched(self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32, UInt32], shape: Tuple[UInt32, UInt32], alpha: Float32 = 1)
Write accumulated results to global memory (3D batched coords).
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): TileTensor-based SMEM tile array for C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32, UInt32]): (m_tile, n_tile, batch) coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - alpha (
Float32): Tensor scale factor (scalar).
def write_batched[ComputeFnType: ElementwiseComputeFn](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32, UInt32], shape: Tuple[UInt32, UInt32], compute_fn: ComputeFnType, alpha: Float32 = 1)
Write accumulated results to global memory (3D batched coords), applying compute_fn.
Parameters:
- ComputeFnType (
ElementwiseComputeFn): Type of the compute epilogue closure.
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): TileTensor-based SMEM tile array for C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32, UInt32]): (m_tile, n_tile, batch) coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - compute_fn (
ComputeFnType): Element-wise epilogue applied to each output value. - alpha (
Float32): Tensor scale factor (scalar).
def write_batched[EpilogueFnType: ElementwiseEpilogueFn](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], tile_coord: Tuple[UInt32, UInt32, UInt32], shape: Tuple[UInt32, UInt32], epilogue_fn: EpilogueFnType, alpha: Float32 = 1)
Write accumulated results to global memory (3D batched coords) through epilogue_fn.
Parameters:
- EpilogueFnType (
ElementwiseEpilogueFn): Type of the elementwise epilogue closure.
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): TileTensor-based SMEM tile array for C output. - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - tile_coord (
Tuple[UInt32, UInt32, UInt32]): (m_tile, n_tile, batch) coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - epilogue_fn (
EpilogueFnType): Stores each output value to global memory. - alpha (
Float32): Tensor scale factor (scalar).
write_splitk
def write_splitk[reduction_layout: TensorLayout, reduction_engine: TensorEngine](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], scheduler: TileScheduler, reduction_tensor: TileTensor[accum_type, reduction_layout, MutAnyOrigin, Engine=reduction_engine], work_info: WorkInfo, shape: Tuple[UInt32, UInt32], elect_one_warp: Bool)
Write with split-K reduction. Only last split writes to GMEM.
write_splitk_parallel
def write_splitk_parallel[pieces_per_batch: Int](self, stage: OutputStage[opc], scheduler: TileScheduler, workspace: MutPointer[Scalar[accum_type], MutAnyOrigin], counters: SplitKTileCounters, c: MutPointer[Scalar[c_type], MutAnyOrigin], c_row_stride: Int, work_info: WorkInfo, shape: Tuple[UInt32, UInt32])
Writes a split-K tile with the parallel reduction: this split's partial goes to its own workspace slot, the TMEM stage is released, then every split of the tile reduces its slice of all the slots and stores it to C (see TileScheduler.reduce_slice_to_output).
Parameters:
- pieces_per_batch (
Int): Pieces per thread whose slot loads are in flight together.
Args:
- stage (
OutputStage[opc]): The accumulator stage holding this split's partial. - scheduler (
TileScheduler): The split-K tile scheduler. - workspace (
MutPointer[Scalar[accum_type], MutAnyOrigin]): Base of the fp32 workspace, one slot per split. - counters (
SplitKTileCounters): The kernel's tile counters. - c (
MutPointer[Scalar[c_type], MutAnyOrigin]): Base of the row-major output. - c_row_stride (
Int): Elements between consecutive rows of the output. - work_info (
WorkInfo): The split's work tile. - shape (
Tuple[UInt32, UInt32]): The output's (rows, columns).
write_absolute_with_bounds_check
def write_absolute_with_bounds_check[c_tensor_layout: TensorLayout, c_tensor_engine: TensorEngine, RowScalesT: RowScales = NullRowScales](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], output_stage: OutputStage[opc], m_abs: UInt32, n_abs: UInt32, m_end: UInt32, expert_scale: Float32, c_tensor: TileTensor[c_type, c_tensor_layout, Engine=c_tensor_engine, address_space=c_tensor.address_space, linear_idx_type=c_tensor.linear_idx_type], p3_control: Int = Int(-1), p3_expert_id: Int32 = Int32(0), p3_cfg: P3PeerSendConfig = P3PeerSendConfig.disabled(), row_scales: RowScalesT = NullRowScales())
Write with absolute coordinates and bounds checking.
For 1D-1D grouped kernels where M coordinate is absolute.
Parameters:
- c_tensor_layout (
TensorLayout): Layout of the C tensor in GMEM (inferred). - c_tensor_engine (
TensorEngine): Engine of the C tensor in GMEM (inferred). - RowScalesT (
RowScales): Per-row output scales type. Defaults to the no-opNullRowScales.
Args:
- c_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): SMEM tile array for the C output. - output_stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - m_abs (
UInt32): Absolute M coordinate (start of tile in token space). - n_abs (
UInt32): Absolute N coordinate (start of tile). - m_end (
UInt32): End offset for bounds checking (exclusive). - expert_scale (
Float32): Per-expert output scaling factor. - c_tensor (
TileTensor[c_type, c_tensor_layout, Engine=c_tensor_engine, address_space=c_tensor.address_space, linear_idx_type=c_tensor.linear_idx_type]): C tensor in GMEM for bounds-checked stores. - p3_control (
Int): Peer-send gate;-1disables the send. - p3_expert_id (
Int32): Local expert id of the tile being written; the destination resolve keys its counter lookup on it. - p3_cfg (
P3PeerSendConfig): Bundled peer-send configuration (buffers, geometry and the destination-resolve tables). - row_scales (
RowScalesT): Per-row output scales, multiplied into each token row together withexpert_scale.
write_with_residual
def write_with_residual[pipeline_origin: MutOrigin, //, num_src_stages: Int](self, out_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], src_tile: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_src_stages], src_pipeline: Pointer[ProducerConsumerPipeline[num_src_stages], pipeline_origin], beta: Scalar[c_type], tile_coord: Tuple[UInt32, UInt32], shape: Tuple[UInt32, UInt32], elect_one_warp: Bool)
Write with residual: D = accum + beta * C.
Matches the CUTLASS sm100_epilogue_tma_warpspecialized lockstep
pattern: the epilogue load warp pre-fetches one source sub-tile per
inner epilogue stage into a num_src_stages-deep SMEM pipeline; this
method drives one wait_producer / use / consumer_release / step
cycle on src_pipeline per inner stage. The buffer index is read from
the pipeline's consumer_stage() rather than computed offline, so
producer and consumer stay synchronized exactly as in CUTLASS's
consumer_wait → copy(sC) → consumer_release per epi sub-tile.
Pipeline per inner stage:
- Load accum from TMEM to registers (epilogue dtype).
- Wait for source[k] via
src_pipeline.consume(); computeD = accum + beta * Creading from the SMEM buffer at the pipeline's current stage index; release source[k] on context exit. - Stage to output SMEM and TMA-store to GMEM.
Parameters:
- pipeline_origin (
MutOrigin): Mutability origin of the source pipeline ref. - num_src_stages (
Int): Number of source SMEM buffers; must equal the epi-load pipeline's stage count in the kernel.
Args:
- out_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): Output SMEM tile array (for D output). - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - src_tile (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_src_stages]): Source C SMEM tile array (num_src_stages buffers). - src_pipeline (
Pointer[ProducerConsumerPipeline[num_src_stages], pipeline_origin]): Pointer to the source producer/consumer pipeline. One acquire/release cycle is driven per inner epilogue stage. - beta (
Scalar[c_type]): Residual scale factor. - tile_coord (
Tuple[UInt32, UInt32]): (m_tile, n_tile) coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - elect_one_warp (
Bool): Whether this warp is elected for coordination.
def write_with_residual[EpilogueFnType: ElementwiseEpilogueFn, pipeline_origin: MutOrigin, //, num_src_stages: Int](self, out_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], stage: OutputStage[opc], src_tile: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_src_stages], src_pipeline: Pointer[ProducerConsumerPipeline[num_src_stages], pipeline_origin], beta: Scalar[c_type], tile_coord: Tuple[UInt32, UInt32], shape: Tuple[UInt32, UInt32], elect_one_warp: Bool, epilogue_fn: EpilogueFnType)
Write with residual, storing accum + beta * C through epilogue_fn.
Same pipeline as the overload without epilogue_fn, except that the
final step hands each post-residual fragment to epilogue_fn, which
owns the GMEM store, instead of staging it to SMEM for a TMA store.
Parameters:
- EpilogueFnType (
ElementwiseEpilogueFn): Type of the elementwise epilogue closure. - pipeline_origin (
MutOrigin): Mutability origin of the source pipeline ref. - num_src_stages (
Int): Number of source SMEM buffers; must equal the epi-load pipeline's stage count in the kernel.
Args:
- out_tiles (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages]): Output SMEM tile array (for D output). - stage (
OutputStage[opc]): OutputStage with pipeline, index, and TMEM handle. - src_tile (
SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_src_stages]): Source C SMEM tile array (num_src_stages buffers). - src_pipeline (
Pointer[ProducerConsumerPipeline[num_src_stages], pipeline_origin]): Pointer to the source producer/consumer pipeline. One acquire/release cycle is driven per inner epilogue stage. - beta (
Scalar[c_type]): Residual scale factor. - tile_coord (
Tuple[UInt32, UInt32]): (m_tile, n_tile) coordinates. - shape (
Tuple[UInt32, UInt32]): (M, N) problem dimensions. - elect_one_warp (
Bool): Whether this warp is elected for coordination. - epilogue_fn (
EpilogueFnType): Stores each post-residual output value to global memory.
write_batched_with_tma_epilogue_load
def write_batched_with_tma_epilogue_load[epi_load_swizzle: TensorMapSwizzle, epilogue_layout: TensorLayout](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], output_stage: OutputStage[opc], epilogue_tile: TileTensor[c_type, epilogue_layout, MutAnyOrigin, address_space=AddressSpace.SHARED], tile_coord: Tuple[UInt32, UInt32, UInt32], c_shape: Tuple[UInt32, UInt32])
Write accumulated results with epilogue tensor addition to global memory.
Pipeline: TMEM → Registers → (+epilogue from SMEM) → SMEM → GMEM (TMA).
write_batched_with_1d_bias
def write_batched_with_1d_bias[epilogue_layout: TensorLayout](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], output_stage: OutputStage[opc], epilogue_tile: TileTensor[c_type, epilogue_layout, MutAnyOrigin, address_space=AddressSpace.SHARED], tile_coord: Tuple[UInt32, UInt32, UInt32], c_shape: Tuple[UInt32, UInt32])
Write accumulated results with 1D bias addition to global memory.
Pipeline: TMEM -> Registers -> (+1D bias broadcast from SMEM) -> SMEM -> GMEM (TMA).
The bias SMEM tile is 1×MMA_N loaded via cp.async (linear layout, no swizzle) and then broadcast across all M rows.
write_batched_with_tma_epilogue_load_strips
def write_batched_with_tma_epilogue_load_strips[epi_load_swizzle: TensorMapSwizzle, num_epi_stages: Int](self, c_tiles: SMemTileArray2DRowMajor[c_type, c_smem_dim0, c_smem_dim1, num_output_stages], output_stage: OutputStage[opc], mut epilogue_pipeline: ProducerConsumerPipeline[num_epi_stages], epilogue_tiles_base: MutPointer[Scalar[c_type], MutAnyOrigin, address_space=AddressSpace.SHARED], epilogue_tile_elems: Int, tile_coord: Tuple[UInt32, UInt32, UInt32], c_shape: Tuple[UInt32, UInt32])
Write accumulated results with BM×stageN pipelined epilogue addition.
For non-AB_swapped configs. Each epilogue pipeline stage is one BM×stageN tile. Producer sends tiles in stage-outer / col_wg-inner order; consumer mirrors that structure so each TMEM stage is fully processed (load → add epilogue → write) before advancing to the next.