For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
TensorCore
struct TensorCore[out_type: DType, in_type: DType, shape: IndexList[Int(3)], transpose_b: Bool = False]
TensorCore provides an abstraction for GPU tensor core hardware to perform optimized matrix operations.
This struct encapsulates the functionality required to efficiently map matrix operations to Tensor Cores on NVIDIA and AMD GPUs. It handles loading matrix fragments, performing matrix multiply-accumulate operations, and storing results with hardware-specific optimizations.
Note: Different shapes and data types are supported depending on the GPU hardware. For NVIDIA GPUs:
- float32: 16x8x8 or 16x8x4
- half-precision: 16x8x16
- float8: 16x8x32 For AMD GPUs:
- float32: 16x16x4
- half-precision: 16x16x16 or 32x32x8
Parameters
- out_type (
DType): The data type for output/accumulation operations. - in_type (
DType): The data type for input matrix elements. - shape (
IndexList[Int(3)]): The shape parameters for the matrix operation in the form [M, N, K] where MxN is the output shape and K is the inner dimension. - transpose_b (
Bool): Whether to transpose the B matrix before multiplication. Defaults to False.
Implemented traits
AnyType,
Copyable,
Defaultable,
Deinitable,
ImplicitlyCopyable,
Movable
comptime members
a_reg_type
comptime a_reg_type = SIMD[in_type, num_matrix_reg[shape[Int(0)], shape[Int(2)]]()]
SIMD type for the A operand registers.
b_reg_type
comptime b_reg_type = SIMD[in_type, num_matrix_reg[shape[Int(2)], shape[Int(1)]]()]
SIMD type for the B operand registers.
c_fragment_layout
comptime c_fragment_layout = row_major[Int(1), SIMD(SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]()))]()
Native scalar layout for the C register fragment.
c_fragment_type
comptime c_fragment_type = TileTensor[out_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]())]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Native local-memory type for the C register fragment.
c_reg_tile_type
comptime c_reg_tile_type = LayoutTensor[out_type, Layout.col_major(Int(1), SIMD(SIMDLength(num_matrix_reg[shape[Int(0)], shape[Int(1)]]()))), MutAnyOrigin, address_space=AddressSpace.LOCAL]
LayoutTensor type for the C register tile.
c_reg_type
comptime c_reg_type = SIMD[out_type, num_matrix_reg[shape[Int(0)], shape[Int(1)]]()]
SIMD type for the C/accumulator operand registers.
supported_fp32
comptime supported_fp32 = (shape == IndexList(Int(16), Int(8), Int(8), __list_literal__=NoneType(None))) if is_nvidia_gpu() else (shape == IndexList(Int(16), Int(16), Int(4), __list_literal__=NoneType(None))) if (in_type == DType.float32) else (in_type == DType.float32)
Whether float32 is supported for this tensor core configuration.
supported_fp64
comptime supported_fp64 = (out_type == DType.float64) if (in_type == DType.float64) else (in_type == DType.float64) and (shape in Tuple(shape_8x8x4, shape_16x8x4, shape_16x8x8, shape_16x8x16)) if is_nvidia_gpu() else False
Whether float64 is supported for this tensor core configuration.
supported_fp8
comptime supported_fp8 = (in_type in Tuple(DType.float8_e4m3fn, DType.float8_e5m2)) and (shape == shape_16x8x32) if is_nvidia_gpu() else (Optional(in_type) in Tuple(get_amd_fp8_dtype(), get_amd_bf8_dtype())) and (shape in Tuple(shape_16x16x32, shape_32x32x64))
Whether float8 is supported for this tensor core configuration.
supported_half
comptime supported_half = in_type.is_half_float() and (shape == shape_16x8x16) if is_nvidia_gpu() else (shape in Tuple(shape_16x16x16, shape_16x16x32, shape_32x32x8, shape_32x32x16))
Whether half-precision float is supported for this configuration.
Methods
__init__
def __init__(out self)
Initialize a new TensorCore instance.
get_shapes
static def get_shapes[_out_type: DType, _in_type: DType]() -> List[IndexList[Int(3)]]
Get supported shapes for given data types.
Returns a list of valid shapes for the specified output and input data types.
Note: The returned shapes are hardware-dependent. Different shapes are supported for different combinations of input and output types.
Parameters:
- _out_type (
DType): The output/accumulation data type. - _in_type (
DType): The input matrix data type.
Returns:
List[IndexList[Int(3)]]: List[IndexList[3]]: Valid shapes for the matrix operations given the specified types.
load_a
def load_a[swizzle: Optional[Swizzle] = None](self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Loads a native matrix tile into hardware A registers.
Constraints:
NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require M to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA and swizzled loads are not supported.
Parameters:
- swizzle (
Optional[Swizzle]): Must beNone; the tile layout supplies the addressing.
Args:
- a (
TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): An M-by-K scalar tile with arbitrary strides and static K.
Returns:
TileTensor[in_type, Layout[TypeList[ComptimeInt[Int(1)], ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()]](), TypeList[ComptimeInt[num_matrix_reg[shape[Int(0)], a.LayoutType.static_shape[Int(1)]]()], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.
def load_a[swizzle: Optional[Swizzle] = None](self, a: LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment]) -> LayoutTensor[in_type, _get_a_reg_tile_layout[a.layout, shape](), MutAnyOrigin, address_space=AddressSpace.LOCAL]
Load the A matrix fragments.
Loads matrix A from memory into a LayoutTensor suitable for tensor core operations.
Parameters:
- swizzle (
Optional[Swizzle]): Optional swizzle pattern for optimal memory access (AMD only).
Args:
- a (
LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment]): The source matrix A data.
Returns:
LayoutTensor[in_type, _get_a_reg_tile_layout[a.layout, shape](), MutAnyOrigin, address_space=AddressSpace.LOCAL]: The loaded matrix fragments as a LayoutTensor.
def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], mma_tile_coord_k: Int = Int(0))
Load A matrix fragments from shared memory.
Optimized version for loading A matrix fragments from shared memory.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern for to optimize memory bandwidth.
Args:
- warp_tile (
LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment]): The source data in shared memory. - fragments (
LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment]): The destination tensor for fragments. - mma_tile_coord_k (
Int): The K coordinate of the MMA tile. Defaults to 0.
def load_a[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0))
Load A matrix fragments from a shared-memory warp tile.
TileTensor-native variant of the shared-memory load_a above: the
warp tile and the fragment destination are TileTensors, and the
swizzle must be passed explicitly (the legacy overload for
LayoutTensors cannot infer it from a TileTensor layout).
Bridges to the identical ldmatrix/distribute implementation, so the
register ordering, swizzle pattern, and alignment requirements match
the legacy overload exactly.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern to optimize memory bandwidth. Must beNoneon NVIDIA GPUs.
Args:
- warp_tile (
TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type]): The source data in shared memory. - fragments (
TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type]): The destination scalar tile in local memory, of shape (num_frags, MMA M-by-K registers per thread). - mma_tile_coord_k (
Int): The K coordinate of the MMA tile. Defaults to 0.
load_b
def load_b[swizzle: Optional[Swizzle] = None](self, b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Loads a native matrix tile into hardware B registers.
Constraints:
NVIDIA loads require exactly one MMA K dimension. AMD CDNA loads require N to be 16 or 32 and K to be a positive multiple of MMA K; consecutive register groups are packed per lane. RDNA, swizzled loads, and transposed NVIDIA FP8 loads are not supported.
Parameters:
- swizzle (
Optional[Swizzle]): Must beNone; the tile layout supplies the addressing.
Args:
- b (
TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): A K-by-N scalar tile, or N-by-K whentranspose_bis enabled. Strides are arbitrary; K must be static.
Returns:
TileTensor[in_type, Layout[TypeList[ComptimeInt[num_matrix_reg[b.LayoutType.static_shape[Int(1) if transpose_b else Int(0)], shape[Int(1)]]()], ComptimeInt[Int(1)]](), TypeList[ComptimeInt[Int(1)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]: The native register fragment accepted by mma_op.
def load_b[swizzle: Optional[Swizzle] = None](self, b: LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment]) -> LayoutTensor[in_type, _get_b_reg_tile_layout[b.layout, shape, transpose_b](), MutAnyOrigin, address_space=AddressSpace.LOCAL]
Load the B matrix fragments.
Loads matrix B from memory into a LayoutTensor suitable for tensor core operations.
The function handles different hardware architectures and memory access patterns.
Note:
If transpose_b is True, the B matrix will be transposed during loading.
This is more efficient than transposing the matrix in memory.
Parameters:
- swizzle (
Optional[Swizzle]): Optional swizzle pattern for optimal memory access (AMD only). Will cause an error if used with NVIDIA GPUs.
Args:
- b (
LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment]): The source matrix B data.
Returns:
LayoutTensor[in_type, _get_b_reg_tile_layout[b.layout, shape, transpose_b](), MutAnyOrigin, address_space=AddressSpace.LOCAL]: The loaded matrix fragments as a LayoutTensor.
def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))
Load B matrix fragments from shared memory into registers for tensor core operations.
This function loads matrix B fragments from a warp tile in shared memory into register fragments for use in tensor core matrix multiply operations. It handles hardware-specific optimizations for both NVIDIA and AMD GPUs.
Note:
The warp_tile must be in shared memory. For NVIDIA GPUs, swizzle must be None.
For AMD GPUs, providing an appropriate swizzle pattern can improve performance.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must be None when running on NVIDIA GPUs. For NVIDIA GPUs, swizzle is always on.
Args:
- warp_tile (
LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment]): SourceLayoutTensorin shared memory containing the B matrix data. - fragments (
LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment]): DestinationLayoutTensorto store the loaded matrix fragments. - mma_tile_coord_k (
Int): K-dimension coordinate within the warp tile. Defaults to 0. - warp_tile_coord_n (
Int): N-dimension coordinate within the warp tile. Defaults to 0.
def load_b[swizzle: Optional[Swizzle] = None](self, warp_tile: TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type], mma_tile_coord_k: Int = Int(0), warp_tile_coord_n: Int = Int(0))
Load B matrix fragments from a shared-memory warp tile.
TileTensor-native variant of the shared-memory load_b above: the
warp tile and the fragment destination are TileTensors, and the
swizzle must be passed explicitly. Bridges to the identical
ldmatrix/distribute implementation, so the register ordering,
swizzle pattern, and alignment requirements match the legacy
overload exactly.
Parameters:
- swizzle (
Optional[Swizzle]): Optional memory access pattern for AMD GPUs to optimize memory bandwidth. Must beNoneon NVIDIA GPUs, where the swizzle is always derived from the warp tile's row stride.
Args:
- warp_tile (
TileTensor[in_type, Engine=warp_tile.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_tile.linear_idx_type]): Source tile in shared memory containing the B matrix data. - fragments (
TileTensor[in_type, Engine=fragments.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=fragments.linear_idx_type]): The destination scalar tile in local memory, of shape (num_frags, MMA K-by-N registers per thread). - mma_tile_coord_k (
Int): K-dimension coordinate within the warp tile. Defaults to 0. - warp_tile_coord_n (
Int): N-dimension coordinate of the warp tile. Defaults to 0.
def load_b(self, warp_tile: TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type], fragments: TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type], scales: TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], mma_tile_coord_k: Int = Int(0))
Dequantizes repacked INT4 B operands into native bfloat16 registers.
Constraints:
Supports NVIDIA bfloat16 16-by-8-by-16 MMA only. The destination has an even number of fragment rows between two and eight. Packed words need only scalar alignment. This loader consumes the existing INT4 repack format, not FP4 or FP8.
Args:
- warp_tile (
TileTensor[Engine=warp_tile.Engine, address_space=warp_tile.address_space, linear_idx_type=warp_tile.linear_idx_type]): Shared-memory tile of packed int32 or uint32 words. - fragments (
TileTensor[.bfloat16, Engine=fragments.Engine, address_space=fragments.address_space, linear_idx_type=fragments.linear_idx_type]): Local scalar tile with four bfloat16 registers per row. - scales (
TileTensor[.bfloat16, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type]): Local scalar tile with one bfloat16 scale per fragment row. - mma_tile_coord_k (
Int): Coordinate selecting a contiguous 128-word tile.
def load_b(self, warp_tile: LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment], fragments: LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment], scales: LayoutTensor[address_space=scales.address_space, element_layout=scales.element_layout, layout_int_type=scales.layout_int_type, linear_idx_type=scales.linear_idx_type, masked=scales.masked, alignment=scales.alignment], mma_tile_coord_k: Int = Int(0))
Load quantized B matrix fragments from shared memory with dequantization.
This function loads int4 quantized matrix B fragments from shared memory, dequantizes them using the provided scales, and stores the result in register fragments for tensor core operations.
Notes:
- The
warp_tilemust be in shared memory. - The
fragmentsandscalesmust be in local memory. - The unpacker produces bfloat16 pairs from repacked INT4 words.
- The quantized data is stored as int4 values packed into int32 elements.
- Each thread processes multiple fragments by unpacking and dequantizing the int4 values.
Args:
- warp_tile (
LayoutTensor[address_space=warp_tile.address_space, element_layout=warp_tile.element_layout, layout_int_type=warp_tile.layout_int_type, linear_idx_type=warp_tile.linear_idx_type, masked=warp_tile.masked, alignment=warp_tile.alignment]): SourceLayoutTensorin shared memory containing the quantized B matrix data. - fragments (
LayoutTensor[address_space=fragments.address_space, element_layout=fragments.element_layout, layout_int_type=fragments.layout_int_type, linear_idx_type=fragments.linear_idx_type, masked=fragments.masked, alignment=fragments.alignment]): DestinationLayoutTensorto store the dequantized matrix fragments. - scales (
LayoutTensor[address_space=scales.address_space, element_layout=scales.element_layout, layout_int_type=scales.layout_int_type, linear_idx_type=scales.linear_idx_type, masked=scales.masked, alignment=scales.alignment]):LayoutTensorcontaining the scaling factors for dequantization. - mma_tile_coord_k (
Int): K-dimension coordinate within the warp tile. Defaults to 0.
load_c
def load_c(self, c: LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]) -> Self.c_reg_tile_type
Load the C matrix fragments.
Loads matrix C from memory into a LayoutTensor suitable for tensor core operations.
The function handles different hardware architectures and memory access patterns.
Args:
- c (
LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]): The source matrix C data.
Returns:
Self.c_reg_tile_type: The loaded matrix fragments as a LayoutTensor.
def load_c(self, c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type
Loads a scalar matrix tile into the MMA accumulator registers.
Constraints:
The tile must have scalar elements and exactly the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.
Args:
- c (
TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): The complete M-by-N accumulator tile, with arbitrary strides.
Returns:
Self.c_fragment_type: The register fragment accepted by mma_op.
store_d
def store_d(self, d_dst: LayoutTensor[address_space=d_dst.address_space, element_layout=d_dst.element_layout, layout_int_type=d_dst.layout_int_type, linear_idx_type=d_dst.linear_idx_type, masked=d_dst.masked, alignment=d_dst.alignment], d_src: LayoutTensor[address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment])
Store matrix D to destination memory.
Stores the result matrix D from tensor core computation to the destination memory.
Args:
- d_dst (
LayoutTensor[address_space=d_dst.address_space, element_layout=d_dst.element_layout, layout_int_type=d_dst.layout_int_type, linear_idx_type=d_dst.linear_idx_type, masked=d_dst.masked, alignment=d_dst.alignment]): The destination tensor to store the result. - d_src (
LayoutTensor[address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment]): The source tensor containing the computed result.
def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: LayoutTensor[out_type, address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment])
Stores a legacy accumulator fragment into a scalar matrix tile.
Args:
- d_dst (
TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type]): The complete M-by-N output tile, with arbitrary strides. - d_src (
LayoutTensor[out_type, address_space=d_src.address_space, element_layout=d_src.element_layout, layout_int_type=d_src.layout_int_type, linear_idx_type=d_src.linear_idx_type, masked=d_src.masked, alignment=d_src.alignment]): A scalar legacy register fragment of shape (1, registers).
def store_d(self, d_dst: TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type], d_src: TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type])
Stores native MMA accumulator registers into a scalar matrix tile.
Constraints:
The source must have contiguous scalar registers and shape (1, registers). The destination must have scalar elements and the MMA M-by-N shape. NVIDIA supports float32 and float64; AMD supports float32.
Args:
- d_dst (
TileTensor[out_type, Engine=d_dst.Engine, address_space=d_dst.address_space, linear_idx_type=d_dst.linear_idx_type]): The complete M-by-N output tile, with arbitrary strides. - d_src (
TileTensor[out_type, Engine=d_src.Engine, address_space=d_src.address_space, linear_idx_type=d_src.linear_idx_type]): The native register fragment produced bymma_oporload_c.
mma_op
def mma_op(self, a: LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment], b: LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment], c: LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]) -> Self.c_reg_tile_type
Perform matrix multiply-accumulate operation (MMA).
Executes D = A * B + C using tensor cores.
Args:
- a (
LayoutTensor[address_space=a.address_space, element_layout=a.element_layout, layout_int_type=a.layout_int_type, linear_idx_type=a.linear_idx_type, masked=a.masked, alignment=a.alignment]): The A matrix input. - b (
LayoutTensor[address_space=b.address_space, element_layout=b.element_layout, layout_int_type=b.layout_int_type, linear_idx_type=b.linear_idx_type, masked=b.masked, alignment=b.alignment]): The B matrix input. - c (
LayoutTensor[address_space=c.address_space, element_layout=c.element_layout, layout_int_type=c.layout_int_type, linear_idx_type=c.linear_idx_type, masked=c.masked, alignment=c.alignment]): The C matrix input for accumulation.
Returns:
Self.c_reg_tile_type: Self.c_reg_tile_type: The result of the MMA operation.
def mma_op(self, a: TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], c: TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]) -> Self.c_fragment_type
Multiplies native register fragments and adds the accumulator.
Constraints:
Each fragment must contain exactly one contiguous MMA register group.
Args:
- a (
TileTensor[in_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): A scalar fragment of shape (1, A registers). - b (
TileTensor[in_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): A scalar fragment of shape (B registers, 1). - c (
TileTensor[out_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): A scalar fragment of shape (1, C registers).
Returns:
Self.c_fragment_type: A native local accumulator fragment in hardware register order.
mma
def mma(self, a_frag: LayoutTensor[address_space=a_frag.address_space, element_layout=a_frag.element_layout, layout_int_type=a_frag.layout_int_type, linear_idx_type=a_frag.linear_idx_type, masked=a_frag.masked, alignment=a_frag.alignment], b_frag: LayoutTensor[address_space=b_frag.address_space, element_layout=b_frag.element_layout, layout_int_type=b_frag.layout_int_type, linear_idx_type=b_frag.linear_idx_type, masked=b_frag.masked, alignment=b_frag.alignment], c_frag: LayoutTensor[address_space=c_frag.address_space, element_layout=c_frag.element_layout, layout_int_type=c_frag.layout_int_type, linear_idx_type=c_frag.linear_idx_type, masked=c_frag.masked, alignment=c_frag.alignment])
Perform matrix multiply-accumulate operation using tensor cores.
Executes C = A * B + C using tensor cores, where A, B, and C are matrix fragments stored in register memory. This function handles the mapping of fragments to hardware tensor core operations.
Notes:
- All fragments must be properly loaded using the corresponding load functions.
- The function assumes fragments are vectorized layout tensors with dimensions num_vectors x 1.
- The c_frag shape[0] must equal num_m_mmas * num_n_mmas.
- The result is accumulated in-place in c_frag.
Args:
- a_frag (
LayoutTensor[address_space=a_frag.address_space, element_layout=a_frag.element_layout, layout_int_type=a_frag.layout_int_type, linear_idx_type=a_frag.linear_idx_type, masked=a_frag.masked, alignment=a_frag.alignment]): Matrix A fragments as aLayoutTensor. - b_frag (
LayoutTensor[address_space=b_frag.address_space, element_layout=b_frag.element_layout, layout_int_type=b_frag.layout_int_type, linear_idx_type=b_frag.linear_idx_type, masked=b_frag.masked, alignment=b_frag.alignment]): Matrix B fragments as aLayoutTensor. - c_frag (
LayoutTensor[address_space=c_frag.address_space, element_layout=c_frag.element_layout, layout_int_type=c_frag.layout_int_type, linear_idx_type=c_frag.linear_idx_type, masked=c_frag.masked, alignment=c_frag.alignment]): Matrix C fragments as aLayoutTensorfor both input and output.
def mma(self, a_frag: TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type], b_frag: TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type], c_frag: TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type])
Accumulates packed native register groups in hardware order.
Constraints:
Each element must hold exactly one hardware register group. All fragments must reside in local memory. Outer strides may include padding between register groups.
Args:
- a_frag (
TileTensor[in_type, Engine=a_frag.Engine, address_space=a_frag.address_space, linear_idx_type=a_frag.linear_idx_type]): A register groups of shape (M, 1). - b_frag (
TileTensor[in_type, Engine=b_frag.Engine, address_space=b_frag.address_space, linear_idx_type=b_frag.linear_idx_type]): B register groups of shape (N, 1). - c_frag (
TileTensor[Engine=c_frag.Engine, address_space=c_frag.address_space, linear_idx_type=c_frag.linear_idx_type]): Mutable accumulator groups of shape (M * N, 1), with group (m, n) stored at n * M + m.