IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_kv_cache_store_paged

struct Struct_kv_cache_store_paged

Registers the kv_cache.store.paged.ragged graph op with the graph compiler.

Under decode context parallelism (dcp_size > 1) every rank of the group receives the same lookup table and global cache_lengths; a row is stored only on the rank that owns its position (ContextParallelLayout[dcp_size, dcp_interleave]), at that rank's local slot. dcp_rank is a host scalar after layer_idx; outside a group (dcp_size == 1) the caller passes 0 and the layout never reads it.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[key_or_value: Int, v_head_dim: Int = Int(-1), dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](inputs: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, input_row_offsets: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, dcp_rank: Int32, context: DeviceContext)

Was this page helpful?