For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo package
kv_cache
Contains implementations for several types of key-value caches.
KV caches are used in transformer models to store key-value tensors output from self-attention layers, allowing previously computed results to be reused across decoding steps.
This package provides the low-level cache types and index-remapping kernels
used during paged and sparse attention. These APIs are consumed by the
higher-level functions in the nn package and are not typically
called directly by model authors.
Modules
-
context_parallel: Token ownership for a KV cache sharded across a context-parallel group. -
hisparse: HiSparse hot-buffer kernels for sparse MLA over a partly host-resident cache. -
mixed_step_split: The mixed-step split record of a paged KV cache input. -
paged_sparse_kv_index_remap: Map logical sequence token indices to MLA sparsephysical rowencoding. -
types: This module contains the types for the key-value cache APIs.