For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
block_scaled_matmul
def block_scaled_matmul[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, EpilogueFnType: ElementwiseEpilogueFn = inflated$def[_no_epilogue_body[_, _, _, _, _]], ComputeFnType: ElementwiseComputeFn = inflated$def[identity_compute_fn[_, _, _, _, _]], //, *, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, transpose_a: Bool = False, has_epilogue_fn: Bool = False, has_compute_fn: Bool = False, pdl_level: PDLLevel = PDLLevel.OFF, _trace_description: StringSpan[ImmStaticOrigin] = StringSpan(""), target: StringSpan[ImmStaticOrigin] = StringSpan("cpu")](c_device: TileTensor[c_type, Engine=c_device.Engine, linear_idx_type=c_device.linear_idx_type], a_device: TileTensor[a_type, Engine=a_device.Engine, linear_idx_type=a_device.linear_idx_type], b_device: TileTensor[b_type, Engine=b_device.Engine, linear_idx_type=b_device.linear_idx_type], a_scales_device: TileTensor[scales_dtype, Engine=a_scales_device.Engine, linear_idx_type=a_scales_device.linear_idx_type], b_scales_device: TileTensor[scales_dtype, Engine=b_scales_device.Engine, linear_idx_type=b_scales_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContextBase, epilogue_fn: EpilogueFnType = no_epilogue_fn, compute_fn: ComputeFnType = no_compute_fn)
Dispatches the SM100 block-scaled matmul for NVFP4 or MXFP8, selecting between the Mojo heuristic kernel and the vendor fallback.
For small M or MXFP8 shapes the Mojo heuristic_and_outliers_dispatch
kernel is tried first; on a dispatch miss the vendor (cuBLASLt)
block-scaled matmul is used as a fallback.
Parameters:
- c_type (
DType): Element type of the output matrix (inferred). - a_type (
DType): Element type of the LHS input matrix (inferred). - b_type (
DType): Element type of the RHS input matrix (inferred). - scales_dtype (
DType): Element type of the block scale-factor tensors (inferred). - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn(inferred). - SF_VECTOR_SIZE (
Int): Number of elements covered by each block scale factor: 16 for NVFP4 or 32 for MXFP8. - transpose_b (
Bool): Whetherb_deviceis stored transposed (defaults toTrue). - transpose_a (
Bool): Whethera_deviceis stored transposed (defaults toFalse). - has_epilogue_fn (
Bool): Whetherepilogue_fnstores the output instead of the kernel writingc_device. - has_compute_fn (
Bool): Whethercompute_fnmaps each output value before it is stored. - pdl_level (
PDLLevel): Programmatic dependency launch level for the heuristic dispatch kernel (defaults toPDLLevel.OFF). - _trace_description (
StringSpan[ImmStaticOrigin]): Extra label appended to the trace event for profiling (defaults to""). - target (
StringSpan[ImmStaticOrigin]): Trace target string identifying the device in profiling output (defaults to"cpu").