IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

TensorCore

struct TensorCore[out_type: DType, in_type: DType, shape: Coord[*shape.element_types], transpose_b: Bool = False]

TensorCore provides an abstraction for GPU tensor core hardware to perform optimized matrix operations.

This struct encapsulates the functionality required to efficiently map matrix operations to Tensor Cores on NVIDIA and AMD GPUs. It handles loading matrix fragments, performing matrix multiply-accumulate operations, and storing results with hardware-specific optimizations.

Note: Different shapes and data types are supported depending on the GPU hardware. For NVIDIA GPUs:

  • float32: 16x8x8 or 16x8x4
  • half-precision: 16x8x16
  • float8: 16x8x32 For AMD GPUs:
  • float32: 16x16x4
  • half-precision: 16x16x16 or 32x32x8

Parameters​

  • ​out_type (DType): The data type for output/accumulation operations.
  • ​in_type (DType): The data type for input matrix elements.
  • ​shape (Coord[*element_types]): The shape parameters for the matrix operation in the form [M, N, K] where MxN is the output shape and K is the inner dimension.
  • ​transpose_b (Bool): Whether to transpose the B matrix before multiplication. Defaults to False.

Implemented traits​

AnyType, Copyable, Defaultable, Deinitable, ImplicitlyCopyable, Movable

comptime members​

a_reg_type​

comptime a_reg_type = SIMD[in_type, num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[2]]((load_from_mem shape.__getitem_param__[Int(2)]()))]()]

SIMD type for the A operand registers.

b_reg_type​

comptime b_reg_type = SIMD[in_type, num_matrix_reg[SIMD[values[2]]((load_from_mem shape.__getitem_param__[Int(2)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()]

SIMD type for the B operand registers.

c_fragment_layout​

comptime c_fragment_layout = row_major[Int(1), SIMD(SIMDLength(num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()))]()

Native scalar layout for the C register fragment.

c_fragment_type​

comptime c_fragment_type = TileTensor[out_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[SIMDLength(num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]())]](), TypeList[ComptimeInt[num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Native local-memory type for the C register fragment.

c_reg_type​

comptime c_reg_type = SIMD[out_type, num_matrix_reg[SIMD[values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), SIMD[values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()]

SIMD type for the C/accumulator operand registers.

supported_fp32​

comptime supported_fp32 = (shape == Coord(Int(16), Int(8), Int(8))) if is_nvidia_gpu() else (shape == Coord(Int(16), Int(16), Int(4))) if (in_type == DType.float32) else (in_type == DType.float32)

Whether float32 is supported for this tensor core configuration.

supported_fp8​

comptime supported_fp8 = (in_type in Tuple(DType.float8_e4m3fn, DType.float8_e5m2)) and (shape == shape_16x8x32) if is_nvidia_gpu() else (Optional(in_type) in Tuple(get_amd_fp8_dtype(), get_amd_bf8_dtype())) and (shape == shape_16x16x32) or (shape == shape_32x32x64)

Whether float8 is supported for this tensor core configuration.

supported_half​

comptime supported_half = in_type.is_half_float() and (shape == shape_16x8x16) if is_nvidia_gpu() else (shape == shape_16x16x16) or (shape == shape_16x16x32) or (shape == shape_32x32x8) or (shape == shape_32x32x16)

Whether half-precision float is supported for this configuration.

Methods​

__init__​

def __init__(out self)

Initialize a new TensorCore instance.

get_shapes​

static def get_shapes[_out_type: DType, _in_type: DType]() -> List[Coord[Int, Int, Int]]

Get supported shapes for given data types.

Returns a list of valid shapes for the specified output and input data types.

Note: The returned shapes are hardware-dependent. Different shapes are supported for different combinations of input and output types.

Parameters:

  • ​_out_type (DType): The output/accumulation data type.
  • ​_in_type (DType): The input matrix data type.

Returns:

List[Coord[Int, Int, Int]]: List[DynamicCoord[.int, 3]]: Valid shapes for the matrix operations given the specified types.

load_a​

def load_a[swizzle: Optional[Swizzle] = None](self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Loads a native matrix tile into hardware A registers.

Constraints:

NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require M to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA and swizzled loads are not supported.

Parameters:

  • ​swizzle (Optional[Swizzle]): Must be None; the tile layout supplies the addressing.

Args:

Returns:

TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[SIMD[shape.element_types.values[0]]((load_from_mem shape.__getitem_param__[Int(0)]())), a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.

def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0))

Loads A matrix fragments from a shared-memory warp tile.

NVIDIA loads use ldmatrix, with swizzle applied to the lane addresses. AMD loads distribute one contiguous vector per lane and apply swizzle to the vector index.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern to optimize memory bandwidth.

Args:

load_b​

def load_b[swizzle: Optional[Swizzle] = None](self, b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], SIMD[shape.element_types.values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Loads a native matrix tile into hardware B registers.

Constraints:

NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require N to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA, swizzled loads, and transposed NVIDIA FP8 loads are not supported.

Parameters:

  • ​swizzle (Optional[Swizzle]): Must be None; the tile layout supplies the addressing.

Args:

Returns:

TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], SIMD[shape.element_types.values[1]]((load_from_mem shape.__getitem_param__[Int(1)]()))]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.

def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))

Loads B matrix fragments from a shared-memory warp tile.

NVIDIA loads use ldmatrix with the swizzle derived from the warp tile's row stride, except for the float32 and float8 non-transposed forms, which gather the hardware fragment with scalar loads. AMD loads distribute one vector per lane and apply swizzle to the vector index.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must be None on NVIDIA GPUs.

Args:

def load_b(self, warp_tile: TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type], scales: TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], mma_tile_coord_k: Int = Int(0))

Dequantizes repacked INT4 B operands into native bfloat16 registers.

Constraints:

Supports NVIDIA bfloat16 16-by-8-by-16 MMA only. The destination has an even number of fragment rows between two and eight. Packed words need only scalar alignment. This loader consumes the existing INT4 repack format, not FP4 or FP8.

Args:

load_c​

def load_c(self, c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type

Loads a scalar matrix tile into the MMA accumulator registers.

Constraints:

The tile must have scalar elements and exactly the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.

Args:

Returns:

Self.c_fragment_type: The register fragment accepted by mma_op.

store_d​

def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type])

Stores native MMA accumulator registers into a scalar matrix tile.

Constraints:

The source must have contiguous scalar registers and shape (1, registers). The destination must have scalar elements and the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.

Args:

mma_op​

def mma_op(self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type

Multiplies native register fragments and adds the accumulator.

Constraints:

Each fragment must contain exactly one contiguous MMA register group.

Args:

Returns:

Self.c_fragment_type: A native local accumulator fragment in hardware register order.

mma​

def mma(self, a_frag: TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type], b_frag: TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type], c_frag: TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type])

Accumulates packed native register groups in hardware order.

Constraints:

Each element must hold exactly one hardware register group. All fragments must reside in local memory. Outer strides may include padding between register groups.

Args:

Was this page helpful?