For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_indexer_k_quantize_store_fp8_paged
struct Struct_indexer_k_quantize_store_fp8_paged
Registers indexer.k.quantize_store.fp8.paged.ragged.
Normalizes the 128 channels of each raw index key out of the merged
projection's row (they start at key_col_offset) with a biased LayerNorm,
rotates their leading rope_dim columns with interleaved RoPE,
FP8-quantizes them with one E8M0 scale each, and stores the keys and their
float32 scales in the paged indexer cache -- the chain
reduce.layer_norm, rope_ragged,
quantize_dynamic_scaled_float8, kv_cache.store.paged.ragged and
kv_cache.store_k_scales.paged.ragged run, in one launch.
Implemented traitsโ
Methodsโ
executeโ
static def execute[quantization_granularity: Int, rope_dim: Int, key_col_offset: Int](k: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, input_row_offsets: T, max_prompt_length: T, max_cache_length: T, k_scales_blocks: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, gamma: T, beta: T, epsilon: Float32, freqs_cis: T, start_pos: T, context: DeviceContext)