For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
indexer_k_norm_rope_store
def indexer_k_norm_rope_store[fp4: Bool, ratio: Int](keys: TileTensor[.bfloat16, Engine=keys.Engine, address_space=keys.address_space, linear_idx_type=keys.linear_idx_type], positions: TileTensor[.int64, Engine=positions.Engine, address_space=positions.address_space, linear_idx_type=positions.linear_idx_type], frequencies: TileTensor[.float32, Engine=frequencies.Engine, address_space=frequencies.address_space, linear_idx_type=frequencies.linear_idx_type], weight: TileTensor[.bfloat16, Engine=weight.Engine, address_space=weight.address_space, linear_idx_type=weight.linear_idx_type], slots: TileTensor[.int64, Engine=slots.Engine, address_space=slots.address_space, linear_idx_type=slots.linear_idx_type], cache: TileTensor[.uint8, Engine=cache.Engine, address_space=cache.address_space, linear_idx_type=cache.linear_idx_type], page_size: Int, epsilon: Float32, ctx: DeviceContext)
Store 128-channel index keys after RMSNorm and 64-channel GPT-J RoPE.
Constraints:
The FP4 path requires SM100 or newer. Each page contains at least
page_size records of MXFP4_INDEX_RECORD_BYTES for FP4, or of
FP8_INDEX_RECORD_BYTES for FP8. Only group-boundary rows with
nonnegative slots write.
Parameters:
- fp4 (
Bool): Select MXFP4 E2M1/UE8M0; otherwise use E4M3/FP32. - ratio (
Int): Compression group size, either 1 or 2.
Args:
- keys (
TileTensor[.bfloat16, Engine=keys.Engine, address_space=keys.address_space, linear_idx_type=keys.linear_idx_type]): BF16 projected keys. Shape:[count, >= 128], contiguous rows that start at 16-byte boundaries. - positions (
TileTensor[.int64, Engine=positions.Engine, address_space=positions.address_space, linear_idx_type=positions.linear_idx_type]): Nonnegative token positions. Shape:[count]. - frequencies (
TileTensor[.float32, Engine=frequencies.Engine, address_space=frequencies.address_space, linear_idx_type=frequencies.linear_idx_type]): FP32 cosine then sine values, 32 each per position. Shape:[rows, >= 64], contiguous rows that start at 16-byte boundaries. - weight (
TileTensor[.bfloat16, Engine=weight.Engine, address_space=weight.address_space, linear_idx_type=weight.linear_idx_type]): 128 BF16 RMSNorm weights, 16-byte aligned. Shape:[128]. - slots (
TileTensor[.int64, Engine=slots.Engine, address_space=slots.address_space, linear_idx_type=slots.linear_idx_type]): Unique destination slots; negative entries skip the row. Shape:[count]. - cache (
TileTensor[.uint8, Engine=cache.Engine, address_space=cache.address_space, linear_idx_type=cache.linear_idx_type]): Byte cache pages, each page's values preceding its scales. Shape:[pages, page_bytes], contiguous rows that start at 4-byte boundaries. - page_size (
Int): Number of slots in a cache page. - epsilon (
Float32): RMSNorm epsilon. - ctx (
DeviceContext): The device context.
Raises:
If a row of keys, frequencies, or cache is not contiguous and aligned.