For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
block_scaled_matmul
def block_scaled_matmul[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, EpilogueFnType: ElementwiseEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"_no_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], ComputeFnType: ElementwiseComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"identity_compute_fn[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], //, *, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, transpose_a: Bool = False, has_epilogue_fn: Bool = False, has_compute_fn: Bool = False, compute_fn_needs_vectors: Bool = False, compute_fn_col_period: Int = Int(0), compute_fn_col_start: Int = Int(0), pdl_level: PDLLevel = PDLLevel.ON, _trace_description: StaticString = ""](c_device: TileTensor[c_type, Engine=c_device.Engine, linear_idx_type=c_device.linear_idx_type], a_device: TileTensor[a_type, Engine=a_device.Engine, linear_idx_type=a_device.linear_idx_type], b_device: TileTensor[b_type, Engine=b_device.Engine, linear_idx_type=b_device.linear_idx_type], a_scales_device: TileTensor[scales_dtype, Engine=a_scales_device.Engine, linear_idx_type=a_scales_device.linear_idx_type], b_scales_device: TileTensor[scales_dtype, Engine=b_scales_device.Engine, linear_idx_type=b_scales_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext, epilogue_fn: EpilogueFnType = no_epilogue_fn, compute_fn: ComputeFnType = no_compute_fn)
Dispatches the SM100 block-scaled matmul for NVFP4 or MXFP8, selecting between the Mojo heuristic kernel and the vendor fallback.
For small M or MXFP8 shapes the Mojo heuristic_and_outliers_dispatch
kernel is tried first; on a dispatch miss the vendor (cuBLASLt)
block-scaled matmul is used as a fallback.
Parameters:
- c_type (
DType): Element type of the output matrix (inferred). - a_type (
DType): Element type of the LHS input matrix (inferred). - b_type (
DType): Element type of the RHS input matrix (inferred). - scales_dtype (
DType): Element type of the block scale-factor tensors (inferred). - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn(inferred). - SF_VECTOR_SIZE (
Int): Number of elements covered by each block scale factor: 16 for NVFP4 or 32 for MXFP8. - transpose_b (
Bool): Whetherb_deviceis stored transposed (defaults toTrue). - transpose_a (
Bool): Whethera_deviceis stored transposed (defaults toFalse). - has_epilogue_fn (
Bool): Whetherepilogue_fnstores the output instead of the kernel writingc_device. - has_compute_fn (
Bool): Whethercompute_fnmaps each output value before it is stored. - compute_fn_needs_vectors (
Bool): Whethercompute_fnreads each value with its neighbors in the row, so it must get consecutive columns rather than one element at a time. - compute_fn_col_period (
Int): When nonzero,compute_fnreturns its input unchanged at every column whose index modulo this period is belowcompute_fn_col_start, so an unfused pass visits only the remaining columns. - compute_fn_col_start (
Int): The first column of each period thatcompute_fnmay change. - pdl_level (
PDLLevel): Programmatic dependency launch level for the heuristic dispatch kernel. Defaults toON: the kernel waits for its predecessor at entry and releases its dependents at exit. - _trace_description (
StaticString): Extra label appended to the trace event for profiling (defaults to"").
Args:
- c_device (
TileTensor[c_type, Engine=c_device.Engine, linear_idx_type=c_device.linear_idx_type]): Output matrix of shape[M, N]. - a_device (
TileTensor[a_type, Engine=a_device.Engine, linear_idx_type=a_device.linear_idx_type]): LHS input matrix of shape[M, K]for FP8 or[M, K//2]for packed FP4uint8. - b_device (
TileTensor[b_type, Engine=b_device.Engine, linear_idx_type=b_device.linear_idx_type]): RHS input matrix of shape[N, K]for FP8 or[N, K//2]for packed FP4uint8, stored transposed. - a_scales_device (
TileTensor[scales_dtype, Engine=a_scales_device.Engine, linear_idx_type=a_scales_device.linear_idx_type]): Block scale factors fora_devicein the 5D TCGEN interleaved layout. - b_scales_device (
TileTensor[scales_dtype, Engine=b_scales_device.Engine, linear_idx_type=b_scales_device.linear_idx_type]): Block scale factors forb_devicein the 5D TCGEN interleaved layout. - tensor_sf (
Float32): Scalar multiplier applied to the matmul result. - ctx (
DeviceContext): Device context used to enqueue kernels. - epilogue_fn (
EpilogueFnType): Withhas_epilogue_fn, stores each output value at its(m, n).c_devicemay then be null, and the vendor fallback stages the product through a scratch buffer. - compute_fn (
ComputeFnType): Withhas_compute_fn, maps each output value before it is stored intoc_device.