For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
tensor_core_async
Tensor Core Async Module.
This module provides high-performance abstractions for utilizing NVIDIA's Tensor Cores to perform asynchronous matrix multiplication operations. It implements optimized memory layouts and access patterns for efficient tensor core computations.
Key components:
- Layout creation functions for K-major and MN-major memory arrangements
- Swizzling support for improved memory access patterns
- WGMMA (Warp Group Matrix Multiply-Accumulate) descriptor generation
- TensorCoreAsync struct with methods for asynchronous matrix multiplication
The module supports various data types, matrix dimensions, and memory configurations, enabling efficient implementation of deep learning primitives and other tensor operations that can leverage hardware acceleration.
Performance features:
- Asynchronous execution model to overlap computation and memory access
- Support for different swizzling modes to optimize memory bandwidth
- Efficient register and shared memory utilization
- Support for multi-warp group execution
This implementation is specifically optimized for NVIDIA GPUs with Tensor Core support.
comptime values
st_matrix_m_layout
comptime st_matrix_m_layout[c_type: DType, WG_BM: Int, num_m_mmas: Int, num_consumer: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())))
Layout for M-major st_matrix in the context of the WGMMA C matrix.
This is for swapAB, where the C matrix is transposed during the write phase,
and must be paired with the transposed st_matrix modifier. The modes are
the warp group local thread index, decomposed as (8, 2, 2, 4), and
(WG_BM // 16, num_m_mmas, num_consumer): the M tile, the MMA tile in N and
the consumer warp group. The layout maps a coordinate to an element offset
of c_type.
With num_consumer = 2 and WG_BM // 16 = 2, one atom of one warp holds
these thread indices, each owning the next 128-bit fragment:
| 0 | 8 |
| 1 | 9 |
| ... | ... |
| 7 | 15 |
| 16 | 24 |
| ... | ... |
| 23 | 31 |The four warps of a warp group are laid out next to each other.
Parameters
st_matrix_n_layout
comptime st_matrix_n_layout[c_type: DType, WG_BN: Int, num_m_mmas: Int, num_consumer: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt(), ComptimeInt())))
Layout for N-major st_matrix in the context of the WGMMA C matrix.
The modes are the warp group local thread index, decomposed as
(16, 2, 4), and (WG_BN // 16, num_m_mmas, num_consumer): the N tile,
the MMA tile in M and the consumer warp group. The layout maps a coordinate
to an element offset of c_type.
Each 128-bit fragment atom is a (64, 2 * WG_BN // 16) row-major grid of
fragments: thread (t0, t1, t2) owns fragment (t0 + 16 * t2, t1). The
outer modes step by one fragment column, by one atom (128 fragments) and by
WG_BN // 16 atoms respectively.
Parameters
tile_layout_k_major_typed
comptime tile_layout_k_major_typed[dtype: DType, BM: Int, BK: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, page_dense: Bool = False] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())))
K-major typed Layout for tensor core operations.
Shape ((CM, BM/CM), (sw_K, BK/sw_K)), stride ((sw_K, CM*sw_K), (1, BM*sw_K))
where CM=8 and sw_K = swizzle_mode.bytes() / sizeof(dtype).
An outer dimension of extent 1 carries stride 0 (compact), matching
tile_layout_k_major().
Parameters
- dtype (
DType): Element data type of the tensor. - BM (
Int): Size of the M dimension in the tile. - BK (
Int): Size of the K dimension in the tile. - swizzle_mode (
TensorMapSwizzle): Memory access pattern swizzling mode (default: SWIZZLE_NONE). - page_dense (
Bool): WhenTrue, build the page-dense (chunk-inner, row-major atoms) form thattile_layout_k_major[page_dense=True]()returns instead of the chunk-outer form; the outer strides are the only difference between the two.SWIZZLE_128Bonly (defaults toFalse).
tile_layout_mn_major_typed
comptime tile_layout_mn_major_typed[dtype: DType, mn_dim: Int, k_dim: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, page_dense: Bool = False] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(ComptimeInt(), ComptimeInt())))
MN-major typed Layout for tensor core operations.
Equivalent to tile_layout_k_major_typed[dtype, k_dim, mn_dim, swizzle_mode].transpose(),
except in the page-dense form, which is the native chunk-inner MN-major tile
that tile_layout_mn_major[page_dense=True]() returns (not a transpose).
Parameters
- dtype (
DType): Element data type of the tensor. - mn_dim (
Int): Size of the MN dimension. - k_dim (
Int): Size of the K dimension. - swizzle_mode (
TensorMapSwizzle): Memory access pattern swizzling mode (default: SWIZZLE_NONE). - page_dense (
Bool): WhenTrue, build the page-dense (chunk-inner, row-major atoms) form thattile_layout_mn_major[page_dense=True]()returns.SWIZZLE_128Bonly (defaults toFalse).
tile_sf_layout_k_major_typed
comptime tile_sf_layout_k_major_typed[BM: Int, BK: Int, SF_SCALE_SIZE: Int] = Layout(Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(Coord(ComptimeInt(), ComptimeInt()), ComptimeInt())), Coord(Coord(ComptimeInt(), ComptimeInt()), Coord(Coord(ComptimeInt(), ComptimeInt()), ComptimeInt())))
K-major typed Layout for tensor core scale factors.
Closed form of tile_sf_layout_k_major(): the (32, (4, 4)) scale-factor
atom repeated over BM/128 MN atoms and BK/(4*SF_SCALE_SIZE) K atoms,
K-atom-minor. An atom count of 1 carries stride 0, matching the compact layout
produced by tile_to_shape.
Unlike tile_sf_layout_k_major(), which silently drops a mode when an atom
count rounds down to 0, this rejects tile extents that do not cover a whole
number of atoms.
Parameters
Structs
-
TensorCoreAsync: High-performance asynchronous tensor core operations for matrix multiplication.
Functions
-
warpgroup_fence: Fences Hopper WGMMA accumulator registers against code motion. -
wgmma_canonical_strides: Returns the canonical descriptor layout's two sub-strides, in elements. -
wgmma_desc_sbo_lbo: Returns the (stride, leading) byte offsets of a WGMMA operand descriptor.