IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_indexer_k_quantize_store_fp8_paged

struct Struct_indexer_k_quantize_store_fp8_paged

Registers indexer.k.quantize_store.fp8.paged.ragged.

Normalizes the 128 channels of each raw index key out of the merged projection's row (they start at key_col_offset) with a biased LayerNorm, rotates their leading rope_dim columns with interleaved RoPE, FP8-quantizes them with one E8M0 scale each, and stores the keys and their float32 scales in the paged indexer cache -- the chain reduce.layer_norm, rope_ragged, quantize_dynamic_scaled_float8, kv_cache.store.paged.ragged and kv_cache.store_k_scales.paged.ragged run, in one launch.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[quantization_granularity: Int, rope_dim: Int, key_col_offset: Int](k: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, input_row_offsets: T, max_prompt_length: T, max_cache_length: T, k_scales_blocks: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, gamma: T, beta: T, epsilon: Float32, freqs_cis: T, start_pos: T, context: DeviceContext)

Was this page helpful?