IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

allgather_rmsnorm_quant

def allgather_rmsnorm_quant[in_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], QuantEpilogueType: def[width: Int](row: Int, col: Int, val: SIMD[in_dtype, width]) -> None & RegisterPassable & ImplicitlyCopyable, //, group_size: Int = ngpus, pdl_level: PDLLevel = PDLLevel()](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin], ngpus], normed_out: TileTensor[in_dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type], sum_out: TileTensor[in_dtype, Engine=sum_out.Engine, address_space=sum_out.address_space, linear_idx_type=sum_out.linear_idx_type], gamma: TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Float32, weight_offset: Scalar[in_dtype], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, quant_epilogue: QuantEpilogueType, my_rank: Optional[Int] = None)

allgather_rmsnorm that also hands each normed value to an epilogue.

The epilogue sees the bf16 that lands in normed_out, so a folded-in quantizer emits the same bytes one launch fewer. Caller-supplied because comm cannot depend on linalg (which already depends on comm).

Parameters:

  • ​in_dtype (DType): Input/output data type (bf16).
  • ​ngpus (Int): Total number of devices in the world.
  • ​in_layout (TensorLayout): Layout of the input shard TileTensors.
  • ​in_origin (Origin[mut=mut]): Origin of the input shard TileTensors.
  • ​QuantEpilogueType (def[width: Int](row: Int, col: Int, val: SIMD[in_dtype, width]) -> None & RegisterPassable & ImplicitlyCopyable): The type of quant_epilogue (inferred).
  • ​group_size (Int): Number of devices per independent all-gather group; see _allgather_rmsnorm_impl.
  • ​pdl_level (PDLLevel): Programmatic dependent launch level. Above OFF the grid waits on its predecessor before it reads any input, including gamma, and releases its dependents after the end barrier that retires the peers' reads (ON) or right after that wait (OVERLAP_AT_BEGINNING). That level requires the next kernel not to write this kernel's inputs before its own PDL wait, because peers may still be reading them. ON is the level that is safe against a dependent that does.

Args:

Was this page helpful?