For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
twophase_reduce_kernel
def twophase_reduce_kernel[rank: Int, axis: Int, num_reductions: Int, BLOCK_SIZE: Int, InputFnType: def[dtype: DType, width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, OutputFnType: def[dtype: DType, width: SIMDLength](Coord[**?], StaticTuple[SIMD[dtype, width], num_reductions]) -> None & RegisterPassable & ImplicitlyCopyable, ReduceFnType: def[ty: DType, width: SIMDLength, reduction_idx: Int](SIMD[ty, width], SIMD[ty, width]) -> SIMD[ty, width] & RegisterPassable & ImplicitlyCopyable, dtype: DType, simd_width: Int, accum_type: DType = get_accum_type[dtype]()](shape: IndexList[rank], init: StaticTuple[Scalar[dtype], num_reductions], partials: Pointer[Scalar[accum_type], MutAnyOrigin], counters: Pointer[Int32, MutAnyOrigin], blocks_per_row: Int32, input_fn: InputFnType, output_fn: OutputFnType, reduce_fn: ReduceFnType)
GPU kernel for reductions when there are too few rows to saturate the device at one block per row. Assigns multiple blocks per row and uses a two-phase approach: each block reduces a chunk via cooperative block-level reduction, then the last block to finish (detected via a per-row atomic counter) reduces all partial results for its row.
Parameters:
- rank (
Int): The tensor rank. - axis (
Int): The axis along which to reduce. - num_reductions (
Int): The number of fused reductions to perform. - BLOCK_SIZE (
Int): The number of threads per block. - InputFnType (
def[dtype: DType, width: Int](Coord[**?]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The type ofinput_fn(inferred). - OutputFnType (
def[dtype: DType, width: SIMDLength](Coord[**?], StaticTuple[SIMD[dtype, width], num_reductions]) -> None&RegisterPassable&ImplicitlyCopyable): The type ofoutput_fn(inferred). - ReduceFnType (
def[ty: DType, width: SIMDLength, reduction_idx: Int](SIMD[ty, width], SIMD[ty, width]) -> SIMD[ty, width]&RegisterPassable&ImplicitlyCopyable): The type ofreduce_fn(inferred). - dtype (
DType): The data type of the elements. - simd_width (
Int): The SIMD vector width. - accum_type (
DType): The accumulator data type.
Args:
- shape (
IndexList[rank]): The shape of the input tensor. - init (
StaticTuple[Scalar[dtype], num_reductions]): The identity values for each reduction. - partials (
Pointer[Scalar[accum_type], MutAnyOrigin]): Global memory buffer for per-block partial results. Size: grid_dim.x * num_reductions elements of accum_type. - counters (
Pointer[Int32, MutAnyOrigin]): Global memory buffer for per-row atomic completion counters. Size: num_rows elements of int32, zero-initialized. - blocks_per_row (
Int32): The number of blocks assigned to each row. - input_fn (
InputFnType): The lambda to load input elements. - output_fn (
OutputFnType): The lambda to store output elements. - reduce_fn (
ReduceFnType): The binary reduction function.