IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo package

kv_cache

Contains implementations for several types of key-value caches.

KV caches are used in transformer models to store key-value tensors output from self-attention layers, allowing previously computed results to be reused across decoding steps.

This package provides the low-level cache types and index-remapping kernels used during paged and sparse attention. These APIs are consumed by the higher-level functions in the nn package and are not typically called directly by model authors.

Modules​

  • ​context_parallel: Token ownership for a KV cache sharded across a context-parallel group.
  • ​hisparse: HiSparse hot-buffer kernels for sparse MLA over a partly host-resident cache.
  • ​mixed_step_split: The mixed-step split record of a paged KV cache input.
  • ​paged_sparse_kv_index_remap: Map logical sequence token indices to MLA sparse physical row encoding.
  • ​types: This module contains the types for the key-value cache APIs.

Was this page helpful?