IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

block_scaled_matmul

def block_scaled_matmul[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, EpilogueFnType: ElementwiseEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"_no_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], ComputeFnType: ElementwiseComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"identity_compute_fn[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], //, *, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, transpose_a: Bool = False, has_epilogue_fn: Bool = False, has_compute_fn: Bool = False, compute_fn_needs_vectors: Bool = False, compute_fn_col_period: Int = Int(0), compute_fn_col_start: Int = Int(0), pdl_level: PDLLevel = PDLLevel.ON, _trace_description: StaticString = ""](c_device: TileTensor[c_type, Engine=c_device.Engine, linear_idx_type=c_device.linear_idx_type], a_device: TileTensor[a_type, Engine=a_device.Engine, linear_idx_type=a_device.linear_idx_type], b_device: TileTensor[b_type, Engine=b_device.Engine, linear_idx_type=b_device.linear_idx_type], a_scales_device: TileTensor[scales_dtype, Engine=a_scales_device.Engine, linear_idx_type=a_scales_device.linear_idx_type], b_scales_device: TileTensor[scales_dtype, Engine=b_scales_device.Engine, linear_idx_type=b_scales_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext, epilogue_fn: EpilogueFnType = no_epilogue_fn, compute_fn: ComputeFnType = no_compute_fn)

Dispatches the SM100 block-scaled matmul for NVFP4 or MXFP8, selecting between the Mojo heuristic kernel and the vendor fallback.

For small M or MXFP8 shapes the Mojo heuristic_and_outliers_dispatch kernel is tried first; on a dispatch miss the vendor (cuBLASLt) block-scaled matmul is used as a fallback.

Parameters:

  • ​c_type (DType): Element type of the output matrix (inferred).
  • ​a_type (DType): Element type of the LHS input matrix (inferred).
  • ​b_type (DType): Element type of the RHS input matrix (inferred).
  • ​scales_dtype (DType): Element type of the block scale-factor tensors (inferred).
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​ComputeFnType (ElementwiseComputeFn): Type of compute_fn (inferred).
  • ​SF_VECTOR_SIZE (Int): Number of elements covered by each block scale factor: 16 for NVFP4 or 32 for MXFP8.
  • ​transpose_b (Bool): Whether b_device is stored transposed (defaults to True).
  • ​transpose_a (Bool): Whether a_device is stored transposed (defaults to False).
  • ​has_epilogue_fn (Bool): Whether epilogue_fn stores the output instead of the kernel writing c_device.
  • ​has_compute_fn (Bool): Whether compute_fn maps each output value before it is stored.
  • ​compute_fn_needs_vectors (Bool): Whether compute_fn reads each value with its neighbors in the row, so it must get consecutive columns rather than one element at a time.
  • ​compute_fn_col_period (Int): When nonzero, compute_fn returns its input unchanged at every column whose index modulo this period is below compute_fn_col_start, so an unfused pass visits only the remaining columns.
  • ​compute_fn_col_start (Int): The first column of each period that compute_fn may change.
  • ​pdl_level (PDLLevel): Programmatic dependency launch level for the heuristic dispatch kernel. Defaults to ON: the kernel waits for its predecessor at entry and releases its dependents at exit.
  • ​_trace_description (StaticString): Extra label appended to the trace event for profiling (defaults to "").

Args:

Was this page helpful?