For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
TensorCore
struct TensorCore[out_type: DType, in_type: DType, shape: Coord[*shape.element_types], transpose_b: Bool = False]
TensorCore provides an abstraction for GPU tensor core hardware to perform optimized matrix operations.
This struct encapsulates the functionality required to efficiently map matrix operations to Tensor Cores on NVIDIA and AMD GPUs. It handles loading matrix fragments, performing matrix multiply-accumulate operations, and storing results with hardware-specific optimizations.
Note: Different shapes and data types are supported depending on the GPU hardware. For NVIDIA GPUs:
- float32: 16x8x8 or 16x8x4
- half-precision: 16x8x16
- float8: 16x8x32 For AMD GPUs:
- float32: 16x16x4
- half-precision: 16x16x16 or 32x32x8
Parameters
- out_type (
DType): The data type for output/accumulation operations. - in_type (
DType): The data type for input matrix elements. - shape (
Coord[*element_types]): The shape parameters for the matrix operation in the form [M, N, K] where MxN is the output shape and K is the inner dimension. - transpose_b (
Bool): Whether to transpose the B matrix before multiplication. Defaults to False.
Implemented traits
AnyType,
Copyable,
Defaultable,
Deinitable,
ImplicitlyCopyable,
Movable
comptime members
a_reg_type
comptime a_reg_type = SIMD[in_type, num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[2]]((load_from_mem shape.__getitem_param__[Int(2)]()))]()]
SIMD type for the A operand registers.
b_reg_type
comptime b_reg_type = SIMD[in_type, num_matrix_reg[SIMD[values[2]]((load_from_mem shape.__getitem_param__[Int(2)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()]
SIMD type for the B operand registers.
c_fragment_layout
comptime c_fragment_layout = row_major[Int(1), SIMD(SIMDLength(num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()))]()
Native scalar layout for the C register fragment.
c_fragment_type
comptime c_fragment_type = TileTensor[out_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[SIMDLength(num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]())]](), TypeList[ComptimeInt[num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Native local-memory type for the C register fragment.
c_reg_type
comptime c_reg_type = SIMD[out_type, num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()]
SIMD type for the C/accumulator operand registers.
supported_fp32
comptime supported_fp32 = (shape == Coord(Int(16), Int(8), Int(8))) if is_nvidia_gpu() else (shape == Coord(Int(16), Int(16), Int(4))) if (in_type == DType.float32) else (in_type == DType.float32)
Whether float32 is supported for this tensor core configuration.
supported_fp8
comptime supported_fp8 = (in_type in Tuple(DType.float8_e4m3fn, DType.float8_e5m2)) and (shape == shape_16x8x32) if is_nvidia_gpu() else (Optional(in_type) in Tuple(get_amd_fp8_dtype(), get_amd_bf8_dtype())) and (shape == shape_16x16x32) or (shape == shape_32x32x64)
Whether float8 is supported for this tensor core configuration.
supported_half
comptime supported_half = in_type.is_half_float() and (shape == shape_16x8x16) if is_nvidia_gpu() else (shape == shape_16x16x16) or (shape == shape_16x16x32) or (shape == shape_32x32x8) or (shape == shape_32x32x16)
Whether half-precision float is supported for this configuration.
Methods
__init__
def __init__(out self)
Initialize a new TensorCore instance.
get_shapes
static def get_shapes[_out_type: DType, _in_type: DType]() -> List[Coord[Int, Int, Int]]
Get supported shapes for given data types.
Returns a list of valid shapes for the specified output and input data types.
Note: The returned shapes are hardware-dependent. Different shapes are supported for different combinations of input and output types.
Parameters:
- _out_type (
DType): The output/accumulation data type. - _in_type (
DType): The input matrix data type.
Returns:
List[Coord[Int, Int, Int]]: List[DynamicCoord[.int, 3]]: Valid shapes for the matrix operations given the specified types.
load_a
def load_a[swizzle: Optional[Swizzle] = None](self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Loads a native matrix tile into hardware A registers.
Constraints:
NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require M to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA and swizzled loads are not supported.
Parameters:
- swizzle (
Optional[Swizzle]): Must beNone; the tile layout supplies the addressing.
Args:
- a (
TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): An M-by-K scalar tile with arbitrary strides and static K.
Returns:
def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0))
Loads A matrix fragments from a shared-memory warp tile.
NVIDIA loads use ldmatrix, with swizzle applied to the lane
addresses. AMD loads distribute one contiguous vector per lane and
apply swizzle to the vector index.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern to optimize memory bandwidth.
Args:
- warp_tile (
TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type]): The source data in shared memory. - fragments (
TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type]): The destination scalar tile in local memory, of shape (num_frags, MMA M-by-K registers per thread). - mma_tile_coord_k (
Int): The K coordinate of the MMA tile. Defaults to 0.
load_b
def load_b[swizzle: Optional[Swizzle] = None](self, b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], SIMD[shape.element_types.values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Loads a native matrix tile into hardware B registers.
Constraints:
NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require N to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA, swizzled loads, and transposed NVIDIA FP8 loads are not supported.
Parameters:
- swizzle (
Optional[Swizzle]): Must beNone; the tile layout supplies the addressing.
Args:
- b (
TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): A K-by-N scalar tile, or N-by-K whentranspose_bis enabled. Strides are arbitrary; K must be static.
Returns:
TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], SIMD[shape.element_types.values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.
def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))
Loads B matrix fragments from a shared-memory warp tile.
NVIDIA loads use ldmatrix with the swizzle derived from the warp
tile's row stride, except for the float32 and float8 non-transposed
forms, which gather the hardware fragment with scalar loads. AMD
loads distribute one vector per lane and apply swizzle to the
vector index.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must beNoneon NVIDIA GPUs.
Args:
- warp_tile (
TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type]): Source tile in shared memory containing the B matrix data. - fragments (
TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type]): The destination scalar tile in local memory, of shape (num_frags, MMA K-by-N registers per thread). - mma_tile_coord_k (
Int): K-dimension coordinate within the warp tile. Defaults to 0. - warp_tile_coord_n (
Int): N-dimension coordinate of the warp tile. Defaults to 0.
def load_b(self, warp_tile: TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type], scales: TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], mma_tile_coord_k: Int = Int(0))
Dequantizes repacked INT4 B operands into native bfloat16 registers.
Constraints:
Supports NVIDIA bfloat16 16-by-8-by-16 MMA only. The destination has an even number of fragment rows between two and eight. Packed words need only scalar alignment. This loader consumes the existing INT4 repack format, not FP4 or FP8.
Args:
- warp_tile (
TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type]): Shared-memory tile of packed int32 or uint32 words. - fragments (
TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type]): Local scalar tile with four bfloat16 registers per row. - scales (
TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type]): Local scalar tile with one bfloat16 scale per fragment row. - mma_tile_coord_k (
Int): Coordinate selecting a contiguous 128-word tile.
load_c
def load_c(self, c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type
Loads a scalar matrix tile into the MMA accumulator registers.
Constraints:
The tile must have scalar elements and exactly the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.
Args:
- c (
TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): The complete M-by-N accumulator tile, with arbitrary strides.
Returns:
Self.c_fragment_type: The register fragment accepted by mma_op.
store_d
def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type])
Stores native MMA accumulator registers into a scalar matrix tile.
Constraints:
The source must have contiguous scalar registers and shape (1, registers). The destination must have scalar elements and the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.
Args:
- d_dst (
TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type]): The complete M-by-N output tile, with arbitrary strides. - d_src (
TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type]): The native register fragment produced bymma_oporload_c.
mma_op
def mma_op(self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type
Multiplies native register fragments and adds the accumulator.
Constraints:
Each fragment must contain exactly one contiguous MMA register group.
Args:
- a (
TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): A scalar fragment of shape (1, A registers). - b (
TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): A scalar fragment of shape (B registers, 1). - c (
TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): A scalar fragment of shape (1, C registers).
Returns:
Self.c_fragment_type: A native local accumulator fragment in hardware register order.
mma
def mma(self, a_frag: TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type], b_frag: TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type], c_frag: TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type])
Accumulates packed native register groups in hardware order.
Constraints:
Each element must hold exactly one hardware register group. All fragments must reside in local memory. Outer strides may include padding between register groups.
Args:
- a_frag (
TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type]): A register groups of shape (M, 1). - b_frag (
TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type]): B register groups of shape (N, 1). - c_frag (
TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type]): Mutable accumulator groups of shape (M * N, 1), with group (m, n) stored at n * M + m.