For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
QuantizeDynamicScaledFloat8RowBounded
struct QuantizeDynamicScaledFloat8RowBounded
Registers the quantization.quantize_dynamic_scaled_float8.row_bounded graph op.
Same numerics as mo.composite.quantize_dynamic_scaled_fp8, but the rows it
touches stop at a count the GPU publishes rather than at the height of the
input tensor. The EP MoE down projection needs this: its activation buffer
is sized for the worst-case dispatch, the dispatch kernel writes the live
row count into row_offsets, and the host never learns that count.
A custom op rather than an operand on the composite, whose RMS-norm and
all-reduce fusion patterns read only its input and scale_ub.
Implemented traitsโ
Methodsโ
executeโ
static def execute[group_size_or_per_token: Int](output: T, scales: T, input: T, scale_ub: Float32, row_offsets: T, ctx: DeviceContext)