IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

indexer_k_quantize_store

def indexer_k_quantize_store[cache_t: KVCacheT, gamma_dtype: DType, beta_dtype: DType, rope_dim: Int, key_col_offset: Int](k_cache: cache_t, k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], gamma: TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], beta: TileTensor[beta_dtype, Engine=beta.Engine, address_space=beta.address_space, linear_idx_type=beta.linear_idx_type], epsilon: Float32, freqs_cis: TileTensor[.bfloat16, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], start_pos: TileTensor[.uint32, Engine=start_pos.Engine, address_space=start_pos.address_space, linear_idx_type=start_pos.linear_idx_type], ctx: DeviceContext)

Normalizes, rotates, FP8-quantizes index keys and stores them and their scales in the cache, in one launch.

Replaces, bit for bit, a biased LayerNorm over each 128-channel key, the interleaved RoPE over the leading rope_dim columns, the per-key FP8 quantization with one E8M0 scale, and the two paged-cache stores. The key lands at its request's next cache position, and its scale, as float32, in the cache's scale page at the same position. Rows past the last offset belong to no request and are skipped.

The key's channels are read out of k in place, starting at key_col_offset, so the merged projection can be handed over whole: the op takes its 128 columns where the GEMM wrote them and no copy of them is launched. A caller with a packed key passes offset 0.

start_pos has to hold an entry per request, or the kernel would read past it; a shorter buffer is refused. The chain this replaces tolerated one when the trailing requests carried no rows, so that pathological case is the one input the fused op refuses and the chain did not.

Parameters:

  • ​cache_t (KVCacheT): The paged key cache type, with float32 scales (inferred).
  • ​gamma_dtype (DType): Dtype of the normalization's scale weight.
  • ​beta_dtype (DType): Dtype of the normalization's bias weight.
  • ​rope_dim (Int): Leading columns the RoPE rotates; 0 rotates none.
  • ​key_col_offset (Int): The column k's key starts at; 0 for a packed key.

Args:

Was this page helpful?