IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

tensor_core_async

Tensor Core Async Module.

This module provides high-performance abstractions for utilizing NVIDIA's Tensor Cores to perform asynchronous matrix multiplication operations. It implements optimized memory layouts and access patterns for efficient tensor core computations.

Key components:

  • Layout creation functions for K-major and MN-major memory arrangements
  • Swizzling support for improved memory access patterns
  • WGMMA (Warp Group Matrix Multiply-Accumulate) descriptor generation
  • TensorCoreAsync struct with methods for asynchronous matrix multiplication

The module supports various data types, matrix dimensions, and memory configurations, enabling efficient implementation of deep learning primitives and other tensor operations that can leverage hardware acceleration.

Performance features:

  • Asynchronous execution model to overlap computation and memory access
  • Support for different swizzling modes to optimize memory bandwidth
  • Efficient register and shared memory utilization
  • Support for multi-warp group execution

This implementation is specifically optimized for NVIDIA GPUs with Tensor Core support.

comptime values​

st_matrix_m_layout​

comptime st_matrix_m_layout[c_type: DType, WG_BM: Int, num_m_mmas: Int, num_consumer: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())))

Layout for M-major st_matrix in the context of the WGMMA C matrix.

This is for swapAB, where the C matrix is transposed during the write phase, and must be paired with the transposed st_matrix modifier. The modes are the warp group local thread index, decomposed as (8, 2, 2, 4), and (WG_BM // 16, num_m_mmas, num_consumer): the M tile, the MMA tile in N and the consumer warp group. The layout maps a coordinate to an element offset of c_type.

With num_consumer = 2 and WG_BM // 16 = 2, one atom of one warp holds these thread indices, each owning the next 128-bit fragment:

|  0  |  8  |
|  1  |  9  |
| ... | ... |
|  7  | 15  |

| 16  | 24  |
| ... | ... |
| 23  | 31  |

The four warps of a warp group are laid out next to each other.

Parameters​

  • ​c_type (DType): Data type of the C matrix.
  • ​WG_BM (Int): Size of the M dimension of the C matrix in shared memory.
  • ​num_m_mmas (Int): Number of MMA tiles in the N dimension.
  • ​num_consumer (Int): Number of consumer warp groups.

st_matrix_n_layout​

comptime st_matrix_n_layout[c_type: DType, WG_BN: Int, num_m_mmas: Int, num_consumer: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())))

Layout for N-major st_matrix in the context of the WGMMA C matrix.

The modes are the warp group local thread index, decomposed as (16, 2, 4), and (WG_BN // 16, num_m_mmas, num_consumer): the N tile, the MMA tile in M and the consumer warp group. The layout maps a coordinate to an element offset of c_type.

Each 128-bit fragment atom is a (64, 2 * WG_BN // 16) row-major grid of fragments: thread (t0, t1, t2) owns fragment (t0 + 16 * t2, t1). The outer modes step by one fragment column, by one atom (128 fragments) and by WG_BN // 16 atoms respectively.

Parameters​

  • ​c_type (DType): Data type of the C matrix.
  • ​WG_BN (Int): Size of the N dimension of the C matrix in shared memory.
  • ​num_m_mmas (Int): Number of MMA tiles in the M dimension.
  • ​num_consumer (Int): Number of consumer warp groups.

tile_layout_k_major_typed​

comptime tile_layout_k_major_typed[dtype: DType, BM: Int, BK: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, page_dense: Bool = False] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())))

K-major typed Layout for tensor core operations.

Shape ((CM, BM/CM), (sw_K, BK/sw_K)), stride ((sw_K, CM*sw_K), (1, BM*sw_K)) where CM=8 and sw_K = swizzle_mode.bytes() / sizeof(dtype). An outer dimension of extent 1 carries stride 0 (compact), matching tile_layout_k_major().

Parameters​

  • ​dtype (DType): Element data type of the tensor.
  • ​BM (Int): Size of the M dimension in the tile.
  • ​BK (Int): Size of the K dimension in the tile.
  • ​swizzle_mode (TensorMapSwizzle): Memory access pattern swizzling mode (default: SWIZZLE_NONE).
  • ​page_dense (Bool): When True, build the page-dense (chunk-inner, row-major atoms) form that tile_layout_k_major[page_dense=True]() returns instead of the chunk-outer form; the outer strides are the only difference between the two. SWIZZLE_128B only (defaults to False).

tile_layout_mn_major_typed​

comptime tile_layout_mn_major_typed[dtype: DType, mn_dim: Int, k_dim: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, page_dense: Bool = False] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())))

MN-major typed Layout for tensor core operations.

Equivalent to tile_layout_k_major_typed[dtype, k_dim, mn_dim, swizzle_mode].transpose(), except in the page-dense form, which is the native chunk-inner MN-major tile that tile_layout_mn_major[page_dense=True]() returns (not a transpose).

Parameters​

  • ​dtype (DType): Element data type of the tensor.
  • ​mn_dim (Int): Size of the MN dimension.
  • ​k_dim (Int): Size of the K dimension.
  • ​swizzle_mode (TensorMapSwizzle): Memory access pattern swizzling mode (default: SWIZZLE_NONE).
  • ​page_dense (Bool): When True, build the page-dense (chunk-inner, row-major atoms) form that tile_layout_mn_major[page_dense=True]() returns. SWIZZLE_128B only (defaults to False).

tile_sf_layout_k_major_typed​

comptime tile_sf_layout_k_major_typed[BM: Int, BK: Int, SF_SCALE_SIZE: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(Coord(ComptimeInt(), ComptimeInt()), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(Coord(ComptimeInt(), ComptimeInt()), ComptimeInt())))

K-major typed Layout for tensor core scale factors.

Closed form of tile_sf_layout_k_major(): the (32, (4, 4)) scale-factor atom repeated over BM/128 MN atoms and BK/(4*SF_SCALE_SIZE) K atoms, K-atom-minor. An atom count of 1 carries stride 0, matching the compact layout produced by tile_to_shape.

Unlike tile_sf_layout_k_major(), which silently drops a mode when an atom count rounds down to 0, this rejects tile extents that do not cover a whole number of atoms.

Parameters​

  • ​BM (Int): Size of the M dimension in the tile.
  • ​BK (Int): Size of the K dimension in the tile.
  • ​SF_SCALE_SIZE (Int): Number of elements in a scale factor vector.

Structs​

  • ​TensorCoreAsync: High-performance asynchronous tensor core operations for matrix multiplication.

Functions​

Was this page helpful?