For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mhc_post
def mhc_post[hc_mult: Int, hidden_size: Int, block_size: Int = Int(256)](output: TileTensor[Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], streams: TileTensor[output.dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type], y: TileTensor[output.dtype, Engine=y.Engine, address_space=y.address_space, linear_idx_type=y.linear_idx_type], post: TileTensor[.float32, Engine=post.Engine, address_space=post.address_space, linear_idx_type=post.linear_idx_type], comb: TileTensor[.float32, Engine=comb.Engine, address_space=comb.address_space, linear_idx_type=comb.linear_idx_type], context: DeviceContext, force_non_temporal: Optional[Bool] = None)
Launches the mHC write-back, (num_rows, k_blocks) blocks.
Computes output[t, k, :] = post[t, k] * y[t, :] + sum_j comb[t, j, k] * streams[t, j, :] in fp32 and rounds once. Each thread covers one 16-byte
slice, so k_blocks = hidden_size / (block_size * width) blocks split
each token, and memory accesses turn non-temporal from one token per CU
on AMD (num_rows >= CUs) and past 8 * CUs tokens elsewhere.
Parameters:
- hc_mult (
Int): Number of parallel residual streams. - hidden_size (
Int): Width of one stream. - block_size (
Int): Threads per block.
Args:
- output (
TileTensor[Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Updated streams. Shape:[num_rows, hc_mult, hidden_size]. - streams (
TileTensor[output.dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type]): Incoming streams. Shape:[num_rows, hc_mult, hidden_size]. - y (
TileTensor[output.dtype, Engine=y.Engine, address_space=y.address_space, linear_idx_type=y.linear_idx_type]): Sublayer output. Shape:[num_rows, hidden_size]. - post (
TileTensor[.float32, Engine=post.Engine, address_space=post.address_space, linear_idx_type=post.linear_idx_type]): Weight ofyin each output stream. Shape:[num_rows, hc_mult]. - comb (
TileTensor[.float32, Engine=comb.Engine, address_space=comb.address_space, linear_idx_type=comb.linear_idx_type]): Stream mixer;comb[t, j, k]weighs input streamjin output streamk. Shape:[num_rows, hc_mult, hc_mult]. - context (
DeviceContext): The device context. - force_non_temporal (
Optional[Bool]): Overrides the non-temporal threshold; for benchmarks and tests only.
Raises:
If a shape disagrees with hc_mult, hidden_size or the token
count of streams, or a tensor is not laid out for the kernel's
16-byte vector accesses.