IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

TensorCore

struct TensorCore[out_type: DType, in_type: DType, shape: IndexList[Int(3)], transpose_b: Bool = False]

TensorCore provides an abstraction for GPU tensor core hardware to perform optimized matrix operations.

This struct encapsulates the functionality required to efficiently map matrix operations to Tensor Cores on NVIDIA and AMD GPUs. It handles loading matrix fragments, performing matrix multiply-accumulate operations, and storing results with hardware-specific optimizations.

Note: Different shapes and data types are supported depending on the GPU hardware. For NVIDIA GPUs:

  • float32: 16x8x8 or 16x8x4
  • half-precision: 16x8x16
  • float8: 16x8x32 For AMD GPUs:
  • float32: 16x16x4
  • half-precision: 16x16x16 or 32x32x8

Parameters​

  • ​out_type (DType): The data type for output/accumulation operations.
  • ​in_type (DType): The data type for input matrix elements.
  • ​shape (IndexList[Int(3)]): The shape parameters for the matrix operation in the form [M, N, K] where MxN is the output shape and K is the inner dimension.
  • ​transpose_b (Bool): Whether to transpose the B matrix before multiplication. Defaults to False.

Implemented traits​

AnyType, Copyable, Defaultable, Deinitable, ImplicitlyCopyable, Movable

comptime members​

a_reg_type​

comptime a_reg_type = SIMD[in_type, num_matrix_reg[shape[Int(0)], shape[Int(2)]]()]

SIMD type for the A operand registers.

b_reg_type​

comptime b_reg_type = SIMD[in_type, num_matrix_reg[shape[Int(2)], shape[Int(1)]]()]

SIMD type for the B operand registers.

c_fragment_layout​

comptime c_fragment_layout = row_major[Int(1), SIMD(SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]()))]()

Native scalar layout for the C register fragment.

c_fragment_type​

comptime c_fragment_type = TileTensor[out_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]())]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Native local-memory type for the C register fragment.

c_reg_tile_type​

comptime c_reg_tile_type = LayoutTensor[out_type, Layout.col_major(Int(1), SIMD(SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]()))), MutAnyOrigin, address_space=AddressSpace.LOCAL]

LayoutTensor type for the C register tile.

c_reg_type​

comptime c_reg_type = SIMD[out_type, num_matrix_reg[shape[Int(0)], shape[Int(1)]]()]

SIMD type for the C/accumulator operand registers.

supported_fp32​

comptime supported_fp32 = (shape == IndexList(Int(16), Int(8), Int(8), __list_literal__=NoneType(None))) if is_nvidia_gpu() else (shape == IndexList(Int(16), Int(16), Int(4), __list_literal__=NoneType(None))) if (in_type == DType.float32) else (in_type == DType.float32)

Whether float32 is supported for this tensor core configuration.

supported_fp64​

comptime supported_fp64 = (out_type == DType.float64) if (in_type == DType.float64) else (in_type == DType.float64) and (shape in Tuple(shape_8x8x4, shape_16x8x4, shape_16x8x8, shape_16x8x16)) if is_nvidia_gpu() else False

Whether float64 is supported for this tensor core configuration.

supported_fp8​

comptime supported_fp8 = (in_type in Tuple(DType.float8_e4m3fn, DType.float8_e5m2)) and (shape == shape_16x8x32) if is_nvidia_gpu() else (Optional(in_type) in Tuple(get_amd_fp8_dtype(), get_amd_bf8_dtype())) and (shape in Tuple(shape_16x16x32, shape_32x32x64))

Whether float8 is supported for this tensor core configuration.

supported_half​

comptime supported_half = in_type.is_half_float() and (shape == shape_16x8x16) if is_nvidia_gpu() else (shape in Tuple(shape_16x16x16, shape_16x16x32, shape_32x32x8, shape_32x32x16))

Whether half-precision float is supported for this configuration.

Methods​

__init__​

def __init__(out self)

Initialize a new TensorCore instance.

get_shapes​

static def get_shapes[_out_type: DType, _in_type: DType]() -> List[IndexList[Int(3)]]

Get supported shapes for given data types.

Returns a list of valid shapes for the specified output and input data types.

Note: The returned shapes are hardware-dependent. Different shapes are supported for different combinations of input and output types.

Parameters:

  • ​_out_type (DType): The output/accumulation data type.
  • ​_in_type (DType): The input matrix data type.

Returns:

List[IndexList[Int(3)]]: List[IndexList[3]]: Valid shapes for the matrix operations given the specified types.

load_a​

def load_a[swizzle: Optional[Swizzle] = None](self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Loads a native matrix tile into hardware A registers.

Constraints:

NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require M to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA and swizzled loads are not supported.

Parameters:

  • ​swizzle (Optional[Swizzle]): Must be None; the tile layout supplies the addressing.

Args:

Returns:

TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.

def load_a[swizzle: Optional[Swizzle] = None](self, a: LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment]) -> LayoutTensor[in_type, _get_a_reg_tile_layout[a.layout, shape](), MutAnyOrigin, address_space=AddressSpace.LOCAL]

Load the A matrix fragments.

Loads matrix A from memory into a LayoutTensor suitable for tensor core operations.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional swizzle pattern for optimal memory access (AMD only).

Args:

Returns:

LayoutTensor[in_type, _get_a_reg_tile_layout[a.layout, shape](), MutAnyOrigin, address_space=AddressSpace.LOCAL]: The loaded matrix fragments as a LayoutTensor.

def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], mma_tile_coord_k: Int = Int(0))

Load A matrix fragments from shared memory.

Optimized version for loading A matrix fragments from shared memory.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern for to optimize memory bandwidth.

Args:

def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0))

Load A matrix fragments from a shared-memory warp tile.

TileTensor-native variant of the shared-memory load_a above: the warp tile and the fragment destination are TileTensors, and the swizzle must be passed explicitly (the legacy overload for LayoutTensors cannot infer it from a TileTensor layout). Bridges to the identical ldmatrix/distribute implementation, so the register ordering, swizzle pattern, and alignment requirements match the legacy overload exactly.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern to optimize memory bandwidth. Must be None on NVIDIA GPUs.

Args:

load_b​

def load_b[swizzle: Optional[Swizzle] = None](self, b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]

Loads a native matrix tile into hardware B registers.

Constraints:

NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require N to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA, swizzled loads, and transposed NVIDIA FP8 loads are not supported.

Parameters:

  • ​swizzle (Optional[Swizzle]): Must be None; the tile layout supplies the addressing.

Args:

Returns:

TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.

def load_b[swizzle: Optional[Swizzle] = None](self, b: LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment]) -> LayoutTensor[in_type, _get_b_reg_tile_layout[b.layout, shape, transpose_b](), MutAnyOrigin, address_space=AddressSpace.LOCAL]

Load the B matrix fragments.

Loads matrix B from memory into a LayoutTensor suitable for tensor core operations. The function handles different hardware architectures and memory access patterns.

Note: If transpose_b is True, the B matrix will be transposed during loading. This is more efficient than transposing the matrix in memory.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional swizzle pattern for optimal memory access (AMD only). Will cause an error if used with NVIDIA GPUs.

Args:

Returns:

LayoutTensor[in_type, _get_b_reg_tile_layout[b.layout, shape, transpose_b](), MutAnyOrigin, address_space=AddressSpace.LOCAL]: The loaded matrix fragments as a LayoutTensor.

def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))

Load B matrix fragments from shared memory into registers for tensor core operations.

This function loads matrix B fragments from a warp tile in shared memory into register fragments for use in tensor core matrix multiply operations. It handles hardware-specific optimizations for both NVIDIA and AMD GPUs.

Note: The warp_tile must be in shared memory. For NVIDIA GPUs, swizzle must be None. For AMD GPUs, providing an appropriate swizzle pattern can improve performance.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must be None when running on NVIDIA GPUs. For NVIDIA GPUs, swizzle is always on.

Args:

def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))

Load B matrix fragments from a shared-memory warp tile.

TileTensor-native variant of the shared-memory load_b above: the warp tile and the fragment destination are TileTensors, and the swizzle must be passed explicitly. Bridges to the identical ldmatrix/distribute implementation, so the register ordering, swizzle pattern, and alignment requirements match the legacy overload exactly.

Parameters:

  • ​swizzle (Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must be None on NVIDIA GPUs, where the swizzle is always derived from the warp tile's row stride.

Args:

def load_b(self, warp_tile: TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type], scales: TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], mma_tile_coord_k: Int = Int(0))

Dequantizes repacked INT4 B operands into native bfloat16 registers.

Constraints:

Supports NVIDIA bfloat16 16-by-8-by-16 MMA only. The destination has an even number of fragment rows between two and eight. Packed words need only scalar alignment. This loader consumes the existing INT4 repack format, not FP4 or FP8.

Args:

def load_b(self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], scales: LayoutTensor[address_space=scales.address_space, element_layout=scales.element_layout, layout_int_type=scales.layout_int_type, linear_idx_type=scales.linear_idx_type, masked=scales.masked, alignment=scales.alignment], mma_tile_coord_k: Int = Int(0))

Load quantized B matrix fragments from shared memory with dequantization.

This function loads int4 quantized matrix B fragments from shared memory, dequantizes them using the provided scales, and stores the result in register fragments for tensor core operations.

Notes:

  • The warp_tile must be in shared memory.
  • The fragments and scales must be in local memory.
  • The unpacker produces bfloat16 pairs from repacked INT4 words.
  • The quantized data is stored as int4 values packed into int32 elements.
  • Each thread processes multiple fragments by unpacking and dequantizing the int4 values.

Args:

load_c​

def load_c(self, c: LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]) -> Self.c_reg_tile_type

Load the C matrix fragments.

Loads matrix C from memory into a LayoutTensor suitable for tensor core operations. The function handles different hardware architectures and memory access patterns.

Args:

Returns:

Self.c_reg_tile_type: The loaded matrix fragments as a LayoutTensor.

def load_c(self, c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type

Loads a scalar matrix tile into the MMA accumulator registers.

Constraints:

The tile must have scalar elements and exactly the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.

Args:

Returns:

Self.c_fragment_type: The register fragment accepted by mma_op.

store_d​

def store_d(self, d_dst: LayoutTensor[address_space=d_dst.address_space, element_layout=d_dst.element_layout, layout_int_type=d_dst.layout_int_type, linear_idx_type=d_dst.linear_idx_type, masked=d_dst.masked, alignment=d_dst.alignment], d_src: LayoutTensor[address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment])

Store matrix D to destination memory.

Stores the result matrix D from tensor core computation to the destination memory.

Args:

def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: LayoutTensor[out_type, address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment])

Stores a legacy accumulator fragment into a scalar matrix tile.

Args:

def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type])

Stores native MMA accumulator registers into a scalar matrix tile.

Constraints:

The source must have contiguous scalar registers and shape (1, registers). The destination must have scalar elements and the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.

Args:

mma_op​

def mma_op(self, a: LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment], b: LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment], c: LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]) -> Self.c_reg_tile_type

Perform matrix multiply-accumulate operation (MMA).

Executes D = A * B + C using tensor cores.

Args:

Returns:

Self.c_reg_tile_type: Self.c_reg_tile_type: The result of the MMA operation.

def mma_op(self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type

Multiplies native register fragments and adds the accumulator.

Constraints:

Each fragment must contain exactly one contiguous MMA register group.

Args:

Returns:

Self.c_fragment_type: A native local accumulator fragment in hardware register order.

mma​

def mma(self, a_frag: LayoutTensor[address_space=a_frag.address_space, element_layout=a_frag.element_layout, layout_int_type=a_frag.layout_int_type, linear_idx_type=a_frag.linear_idx_type, masked=a_frag.masked, alignment=a_frag.alignment], b_frag: LayoutTensor[address_space=b_frag.address_space, element_layout=b_frag.element_layout, layout_int_type=b_frag.layout_int_type, linear_idx_type=b_frag.linear_idx_type, masked=b_frag.masked, alignment=b_frag.alignment], c_frag: LayoutTensor[address_space=c_frag.address_space, element_layout=c_frag.element_layout, layout_int_type=c_frag.layout_int_type, linear_idx_type=c_frag.linear_idx_type, masked=c_frag.masked, alignment=c_frag.alignment])

Perform matrix multiply-accumulate operation using tensor cores.

Executes C = A * B + C using tensor cores, where A, B, and C are matrix fragments stored in register memory. This function handles the mapping of fragments to hardware tensor core operations.

Notes:

  • All fragments must be properly loaded using the corresponding load functions.
  • The function assumes fragments are vectorized layout tensors with dimensions num_vectors x 1.
  • The c_frag shape[0] must equal num_m_mmas * num_n_mmas.
  • The result is accumulated in-place in c_frag.

Args:

def mma(self, a_frag: TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type], b_frag: TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type], c_frag: TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type])

Accumulates packed native register groups in hardware order.

Constraints:

Each element must hold exactly one hardware register group. All fragments must reside in local memory. Outer strides may include padding between register groups.

Args:

Was this page helpful?