For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_kv_cache_store_k_scales_paged
struct Struct_kv_cache_store_k_scales_paged
Registers the kv_cache.store_k_scales.paged.ragged graph op with the graph compiler.
Under decode context parallelism (dcp_size > 1) every rank of the
group receives the same lookup table and global cache_lengths; a row is
stored only on the rank that owns its position
(ContextParallelLayout[dcp_size, dcp_interleave]), at that rank's local
slot. dcp_rank is a host scalar after layer_idx; outside a group
(dcp_size == 1) the caller passes 0 and the layout never reads it.
Implemented traitsโ
Methodsโ
executeโ
static def execute[quantization_granularity: Int, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](input_k_scales: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, input_row_offsets: T, max_prompt_length: T, max_cache_length: T, k_scales_blocks: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, dcp_rank: Int32, context: DeviceContext)