For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
reducescatter_quant
Row reduce-scatter whose peer traffic is quantized (lossy).
Each GPU quantizes the rows it contributes to its peers' shards into the
scratch that trails its Signal, in groups of 32 elements, then pulls its own
shard's rows out of the peers' scratch and sums them, dequantized, in f32 with
its own rows, which never leave the device and stay exact. Every element of a
peer's contribution carries that group's quantization error, so the sum is not
the bf16 reduce-scatter's; what it buys is fewer bytes over the links:
quant_bits == 8: E4M3 values with one E8M0 scale per group (MXFP8), 8.25 bits an element.quant_bits == 4: symmetric INT4 in [-7, 7] with one bf16 scale per group, 4.5 bits an element.
comptime values
RS_QUANT_GROUP
comptime RS_QUANT_GROUP = 32
Functions
-
reducescatter_quant: Reduce-scatters[rows, cols]row-wise with quantized peer traffic. -
rs_quant_row_bytes: Bytes one quantized row takes in the scratch: values, then scales. -
rs_quant_scratch_bytes: Scratch bytes a[rows, cols]quantized reduce-scatter needs per GPU.