For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Mojo struct
Struct_matmul_weight_only_scaled_float8_apple
struct Struct_matmul_weight_only_scaled_float8_apple
Apple M5 weight-only FP8 (W8A16) matmul: out = a @ dequant(b)^T.
The FP8 sibling of mo.matmul.weight.only.block.scaled.apple. The activation
a stays in bf16 (NOT dynamically quantized to FP8), and the FP8-E4M3 weight
b is widened to f32/bf16 at the point of consumption (register-resident in
the M == 1 GEMV, or a transient bf16 buffer for the M > 1 interim). Unlike
the NVFP4 sibling there is NO per-block weight scale (modelopt static FP8
carries one per-tensor scalar weight_scale); that scalar is applied at the
graph level by the caller as a post-matmul multiply (the FP8 analog of NVFP4's
weight_scale_2), so it is NOT an input here. input_scale cancels for a
bf16 activation, so it is not an input either.
Implemented traits
Methods
execute
static def execute[c_type: DType, //, target: StringSlice[ImmStaticOrigin]](c: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c.static_spec], a: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a.static_spec], b: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b.static_spec], context: DeviceContext)
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!