For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Row
struct Row[params: ContextParams, accum: DType, dtype: DType, axis: Int, rank: Int, AxisSize: CoordLike, keep_tile: Bool]
The body's handle to one logical row. See the module-level note.
Carries no input_fn field: it's threaded as a fresh value-closure
ARGUMENT to every method that needs it (a "forwarding bridge"),
rather than stored as a typed field. A field whose type is a
trait-bound closure that references a sibling struct parameter
(dtype) hits a Mojo limitation the moment the struct itself is
named generically elsewhere (e.g. as another method's parameter or
return type) — "lacking evidence to prove correctness" even when
the constraint holds by construction. Method-level trait-bound
params (fresh per call, not part of Self's persistent identity)
don't hit this; every other closure here (Contribute, G,
Write, ...) already uses that shape.
Parameters
- params (
ContextParams): Comptime tier parameters (target, tier, widths). - accum (
DType): Accumulator dtype the reduce closures compute in. - dtype (
DType): The primary (axis-walked) input dtype. - axis (
Int): Reduced axis (statically known). - rank (
Int): Tensor rank. - AxisSize (
CoordLike): The reduced-axis length'sCoordLiketype —ComptimeInt[N]when known at comptime (enables keeping the tile in registers) or a dynamicScalartype otherwise. One type carries both the comptime-known-ness and (via.value()) the runtime length, so the caller supplies a singleAxisSize-typed value instead of a separate comptime "is it static" signal. - keep_tile (
Bool): Whether this view may keep the row's tiles in registers between phases instead of loading them again — true for normalizing multi-phase bodies; false for single-pass true reductions where keeping them buys nothing.cachekeeps its values in registers only when the tile is kept.
Fields
- row_coord (
DynamicCoord[.int, rank]): The row's coords (reduced axis pinned to its base). - axis_size (
AxisSize): - ctx (
Context[params]):
Implemented traits
AnyType,
Copyable,
Deinitable,
Movable
Methods
__init__
def __init__[InputFn: def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable](out self, row_coords: Coord[*row_coords.element_types], axis_size: AxisSize, ctx: Context[params], input_fn: InputFn)
Builds the view, staging the row strip into registers when it keeps the tile there.
Parameters:
- InputFn (
def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofinput_fn.
Args:
- row_coords (
Coord[*row_coords.element_types]): The row's coords. - axis_size (
AxisSize): Length of the reduce axis, as anAxisSize-typed value (ComptimeInt[N]()when statically known, else a dynamicScalar). - ctx (
Context[params]): The dispatch bundle. - input_fn (
InputFn): Loads a tile of the primary input at a coord. Used only here (register-path staging); not retained as a field — later streaming-path methods take it again as an argument.
reduce
def reduce[M: ReduceOp, Contribute: def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[accum, w] & RegisterPassable & ImplicitlyCopyable, InputFn: def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable](mut self, contribute: Contribute, input_fn: InputFn) -> M
Runs one reduction phase: folds contribute(tile, idx) into M across the row, cross-thread-joins, and returns the joined state. The body reads M's result fields (e.g. ReduceSum.acc[0], Welford.mean[0], ArgMin.best_idx).
contribute is a value closure (its copy-captured state rides the
value); the tier loop's reduce_tile captures it directly.
input_fn (the primary-input loader) is likewise a fresh value
argument, used only on the streaming path — see the
module note above Row on why it isn't a stored field.
Parameters:
- M (
ReduceOp): The monoid for this phase (instantiated at the view's SIMD width, e.g.ReduceSum[accum, params.simd_width]). - Contribute (
def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[accum, w]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofcontribute. - InputFn (
def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofinput_fn.
Args:
- contribute (
Contribute): Per-element math; maps the loaded primary tile + its coord to the accumulator-dtype contribution. - input_fn (
InputFn): Loads a tile of the primary input at a coord (streaming path only).
Returns:
M: The joined M, broadcast to every participant.
def reduce[T: DType, shared: Bool, //, M: ReduceOp, Contribute: def[w: Int](SIMD[T, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[accum, w] & RegisterPassable & ImplicitlyCopyable, InputFn: def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, Compute: def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w] & RegisterPassable & ImplicitlyCopyable](mut self, over: RowCache[params, T, dtype, axis, rank, ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))) if keep_tile and (xor is_cpu[params.target](), True) and (params._tier != ReduceTier(Int(3))) and identical((add rank, -1), axis) and (xor identical((xor (lt AxisSize.static_value if AxisSize.is_static_value else Int(0), 1), True), False), True) and identical((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params), 0) and (xor identical((lt (mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width)))), 257), False), True) else Int(1), params.simd_width, AxisSize.static_value if AxisSize.is_static_value else Int(0), keep_tile and not is_cpu[params.target]().__bool__() and (params._tier != ReduceTier.Splitk) and (axis == (rank - Int(1))) and (AxisSize.static_value if AxisSize.is_static_value else Int(0) > Int(0)) and ((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params) == Int(0)) and (Int((mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))))) <= Int(256)), shared], contribute: Contribute, input_fn: InputFn, compute: Compute) -> M
Cached overload of reduce: folds contribute over a previously cached row instead of the primary input — reads the staged value at each column (registers on the fast path, recompute fallback otherwise). Used for a phase that reduces a computed cross-phase value (e.g. sum of the fused intermediate squared).
Resolved at compile time from plain Mojo function overloading —
whether the call site passes over — not a runtime-checked
Optional, so the cached-vs-plain choice costs nothing at
runtime.
contribute is a value closure (its copy-captured state rides the
value); the tier loop's reduce_tile captures it directly.
input_fn / compute (the same producer passed to the cache
call that built over) are likewise fresh value arguments, used
only on the recompute path — RowCache doesn't carry
them as fields (see its module note). axis_size / ctx are read
from self rather than threaded as separate arguments — the
owning view already holds both.
Parameters:
- T (
DType): The staged value's dtype (inferred fromover). - shared (
Bool): The staged handle's shmem flag (inferred). - M (
ReduceOp): The monoid for this phase. - Contribute (
def[w: Int](SIMD[T, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[accum, w]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofcontribute. - InputFn (
def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofinput_fn. - Compute (
def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofcompute.
Args:
Returns:
M: The joined M, broadcast to every participant.
elementwise
def elementwise[G: def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None & RegisterPassable & ImplicitlyCopyable, InputFn: def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable](mut self, g: G, input_fn: InputFn)
Per-element terminal (normalizing reductions): runs g(tile, idx) over the row; g computes and stores each output element. Reuses the tile kept in registers when present.
g is a value closure (its copy-captured state rides the value);
the tier loop's emit_tile captures it directly. input_fn is
likewise a fresh value argument (streaming path only) — see the
module note above Row.
Parameters:
- G (
def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None&RegisterPassable&ImplicitlyCopyable): The value-closure type ofg. - InputFn (
def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofinput_fn.
Args:
- g (
G): Per-element write; closes over the joined results and side inputs, and stores via the body's output closure. - input_fn (
InputFn): Loads a tile of the primary input at a coord (streaming path only).
def elementwise[T: DType, shared: Bool, //, G: def[w: Int](SIMD[T, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None & ImplicitlyCopyable, InputFn: def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, Compute: def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w] & RegisterPassable & ImplicitlyCopyable](mut self, over: RowCache[params, T, dtype, axis, rank, ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))) if keep_tile and (xor is_cpu[params.target](), True) and (params._tier != ReduceTier(Int(3))) and identical((add rank, -1), axis) and (xor identical((xor (lt AxisSize.static_value if AxisSize.is_static_value else Int(0), 1), True), False), True) and identical((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params), 0) and (xor identical((lt (mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width)))), 257), False), True) else Int(1), params.simd_width, AxisSize.static_value if AxisSize.is_static_value else Int(0), keep_tile and not is_cpu[params.target]().__bool__() and (params._tier != ReduceTier.Splitk) and (axis == (rank - Int(1))) and (AxisSize.static_value if AxisSize.is_static_value else Int(0) > Int(0)) and ((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params) == Int(0)) and (Int((mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))))) <= Int(256)), shared], g: G, input_fn: InputFn, compute: Compute)
Cached overload of elementwise: the per-element terminal g receives the staged value at each column (from over) instead of the primary input — registers on the fast path, recompute fallback otherwise. g may additionally read other columns of over via over.load (e.g. rope's rotate-half partner, staged in shmem).
Resolved at compile time from plain Mojo function overloading —
whether the call site passes over — not a runtime-checked
Optional, so the cached-vs-plain choice costs nothing at
runtime.
g is a value closure (its copy-captured state rides the value);
the tier loop's emit_tile captures it directly. input_fn /
compute (the same producer passed to the cache call that
built over) are likewise fresh value arguments, used only on
the recompute path — RowCache doesn't carry them as
fields (see its module note). axis_size / ctx are read from
self rather than threaded as separate arguments — the owning
view already holds both.
The bound on G is ImplicitlyCopyable (not RegisterPassable
like the other value overloads) because g may copy-capture
non-register-passable state (e.g. over itself, for a terminal
that also reads a different column via over.load); this overload
runs inside the launched kernel (not across a launch boundary),
so register-passability is not required.
Parameters:
- T (
DType): The staged value's dtype (inferred fromover). - shared (
Bool): The staged handle's shmem flag (inferred). - G (
def[w: Int](SIMD[T, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None&ImplicitlyCopyable): The value-closure type ofg. - InputFn (
def[width: Int, alignment: Int](Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofinput_fn. - Compute (
def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofcompute.
Args:
splat_lane
static def splat_lane[value_dtype: DType](value: SIMD[value_dtype, params.emit_tile_width]) -> SIMD[value_dtype, params.simd_width]
Spreads a per-row value, read with one lane per row, over the width of a fold's result: every lane when the participant handles one row, and unchanged on the tiled tier, where each lane is its own row.
Parameters:
- value_dtype (
DType): The value's element type.
Args:
- value (
SIMD[value_dtype, params.emit_tile_width]): The per-row value, one lane per row.
Returns:
SIMD[value_dtype, params.simd_width]: The value at simd_width lanes.
emit
def emit[Write: def(Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None & RegisterPassable & ImplicitlyCopyable](mut self, write: Write)
Per-row terminal (true reductions): runs write(out_coord) once on the canonical writer for this row. write closes over the joined results and stores them, and is taken as a value arg (its copy-captured state rides the value) rather than a comptime capturing parameter.
Unlike the per-element terminal (elementwise, plain or cached),
emit needs no comptime-form + value-shim pair: once is an
in-kernel thread predicate (not a launch boundary and not a
per-element loop), so the emit_once wrapper (which pins the
reduced axis and calls write) is itself a value closure passed
straight to once, with no per-element env-hoist risk.
Parameters:
- Write (
def(Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> None&RegisterPassable&ImplicitlyCopyable): The value-closure type ofwrite.
Args:
- write (
Write): Per-row terminal store, passed as a value. Stores the row's output(s) at the collapsed coordinate (reduced axis pinned to 0).
cache
def cache[T: DType, Compute: def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w] & RegisterPassable & ImplicitlyCopyable, shared: Bool = False](mut self, compute: Compute) -> RowCache[params, T, dtype, axis, rank, ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))) if keep_tile and (xor is_cpu[params.target](), True) and (params._tier != ReduceTier(Int(3))) and identical((add rank, -1), axis) and (xor identical((xor (lt AxisSize.static_value if AxisSize.is_static_value else Int(0), 1), True), False), True) and identical((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params), 0) and (xor identical((lt (mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width)))), 257), False), True) else Int(1), params.simd_width, AxisSize.static_value if AxisSize.is_static_value else Int(0), keep_tile and not is_cpu[params.target]().__bool__() and (params._tier != ReduceTier.Splitk) and (axis == (rank - Int(1))) and (AxisSize.static_value if AxisSize.is_static_value else Int(0) > Int(0)) and ((AxisSize.static_value if AxisSize.is_static_value else Int(0) % params) == Int(0)) and (Int((mul size_of[SIMD[dtype, params.simd_width]](), ceildiv(AxisSize.static_value if AxisSize.is_static_value else Int(0), Int((mul DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size if (params._tier == ReduceTier(Int(1))) else params.BLOCK_SIZE, params.simd_width))))) <= Int(256)), shared]
Caches a per-row value the later phases reuse: runs compute(tile, idx) once over the row and stages the result so the cached overloads of reduce / elementwise (and, with shared=True, load at any column) read it back instead of recomputing.
When the tile is kept in registers this writes each participant's
tiles into registers (and, when shared, publishes the row to a
shmem strip with one row-local sync); otherwise the returned handle
recomputes on access (always correct) — the caller re-supplies input_fn /
compute to the cached reduce / elementwise overloads on that
path, since RowCache doesn't carry them as fields (see its
module note).
compute runs exactly once per element on the fast path but may
re-run on the fallback, so any side effect it performs (e.g. a
residual emit) must be idempotent.
Parameters:
- T (
DType): The cached value's dtype. - Compute (
def[w: Int](SIMD[dtype, w], Coord[*#kgen.param_list.tabulate(rank, [idx: __mlir_type.index] Int)]) -> SIMD[T, w]&RegisterPassable&ImplicitlyCopyable): The value-closure type ofcompute. - shared (
Bool): Publish the row to shmem for cross-participantload.
Args:
- compute (
Compute): Per-element producer from the primary tile + coord.
Returns: