For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
block_scaled_matmul_block_diagonal
def block_scaled_matmul_block_diagonal[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, //, *, groups: Int, SF_VECTOR_SIZE: Int, pdl_level: PDLLevel = PDLLevel.ON](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext)
Computes a block-diagonal block-scaled matmul on SM100.
Block g of groups multiplies A's columns [g * K, (g + 1) * K) by
B's rows [g * N, (g + 1) * N) and writes C's columns
[g * N, (g + 1) * N), where K = b.dim[1] and
N = c.dim[1] / groups. The blocks run as one batched matmul over
strided views, so A and C stay token-major and the scales stay in the
layouts the dense quantize and the weights already use.
Parameters:
- c_type (
DType): Element type ofc(inferred). - a_type (
DType): Element type ofa(inferred). - b_type (
DType): Element type ofb(inferred). - scales_dtype (
DType): Element type of both scale tensors (inferred). - groups (
Int): The number of diagonal blocks. - SF_VECTOR_SIZE (
Int): Elements per scale factor. - pdl_level (
PDLLevel): Programmatic dependent launch level.
Args:
- c (
TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output[M, groups * N]. - a (
TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]):[M, groups * K]operand. - b (
TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]):[groups * N, K]operand, K-major. - a_scales (
TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]):a's scales in the 5D SF-atom layout of the whole[M, groups * K]operand. - b_scales (
TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]):b's scales in the 5D SF-atom layout. - tensor_sf (
Float32): Scalar multiplier applied to the product. - ctx (
DeviceContext): Device context used to launch the kernel.
Raises:
If no SM100 kernel config covers the shape.