IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

twophase_reduce_kernel

def twophase_reduce_kernel[rank: Int, axis: Int, num_reductions: Int, BLOCK_SIZE: Int, InputFnType: def[dtype: DType, width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, OutputFnType: def[dtype: DType, width: SIMDLength](Coord[**?], StaticTuple[SIMD[dtype, width], num_reductions]) -> None & RegisterPassable & ImplicitlyCopyable, ReduceFnType: def[ty: DType, width: SIMDLength, reduction_idx: Int](SIMD[ty, width], SIMD[ty, width]) -> SIMD[ty, width] & RegisterPassable & ImplicitlyCopyable, dtype: DType, simd_width: Int, accum_type: DType = get_accum_type[dtype]()](shape: IndexList[rank], init: StaticTuple[Scalar[dtype], num_reductions], partials: Pointer[Scalar[accum_type], MutAnyOrigin], counters: Pointer[Int32, MutAnyOrigin], blocks_per_row: Int32, input_fn: InputFnType, output_fn: OutputFnType, reduce_fn: ReduceFnType)

GPU kernel for reductions when there are too few rows to saturate the device at one block per row. Assigns multiple blocks per row and uses a two-phase approach: each block reduces a chunk via cooperative block-level reduction, then the last block to finish (detected via a per-row atomic counter) reduces all partial results for its row.

Parameters:

  • ​rank (Int): The tensor rank.
  • ​axis (Int): The axis along which to reduce.
  • ​num_reductions (Int): The number of fused reductions to perform.
  • ​BLOCK_SIZE (Int): The number of threads per block.
  • ​InputFnType (def[dtype: DType, width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable): The type of input_fn (inferred).
  • ​OutputFnType (def[dtype: DType, width: SIMDLength](Coord[**?], StaticTuple[SIMD[dtype, width], num_reductions]) -> None & RegisterPassable & ImplicitlyCopyable): The type of output_fn (inferred).
  • ​ReduceFnType (def[ty: DType, width: SIMDLength, reduction_idx: Int](SIMD[ty, width], SIMD[ty, width]) -> SIMD[ty, width] & RegisterPassable & ImplicitlyCopyable): The type of reduce_fn (inferred).
  • ​dtype (DType): The data type of the elements.
  • ​simd_width (Int): The SIMD vector width.
  • ​accum_type (DType): The accumulator data type.

Args:

  • ​shape (IndexList[rank]): The shape of the input tensor.
  • ​init (StaticTuple[Scalar[dtype], num_reductions]): The identity values for each reduction.
  • ​partials (Pointer[Scalar[accum_type], MutAnyOrigin]): Global memory buffer for per-block partial results. Size: grid_dim.x * num_reductions elements of accum_type.
  • ​counters (Pointer[Int32, MutAnyOrigin]): Global memory buffer for per-row atomic completion counters. Size: num_rows elements of int32, zero-initialized.
  • ​blocks_per_row (Int32): The number of blocks assigned to each row.
  • ​input_fn (InputFnType): The lambda to load input elements.
  • ​output_fn (OutputFnType): The lambda to store output elements.
  • ​reduce_fn (ReduceFnType): The binary reduction function.

Was this page helpful?