For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
KVCacheStaticParams
struct KVCacheStaticParams
Compile-time shape parameters shared across all layers of a KV cache.
Groups the attention-head count and per-head size that are fixed for the entire model lifetime, along with the Multi-head Latent Attention flag that changes the KV layout from two caches (K + V) to one (K only).
A split cache (v_head_size > 0) also has one plane, but each row packs a
key of k_head_size() columns and then a value of v_head_size columns.
head_size stays the row pitch, so row arithmetic is unchanged; value
accesses add v_col_base() to the column.
Fields
- num_heads (
Int): - head_size (
Int): - is_mla (
Bool): - fp8_tail_size (
Int): - v_head_size (
Int):
Implemented traits
AnyType,
Copyable,
Deinitable,
Equatable,
ImplicitlyCopyable,
Movable,
RegisterPassable,
TrivialRegisterPassable
Methods
__init__
def __init__(num_heads: Int, head_size: Int, is_mla: Bool = False, fp8_tail_size: Int = Int(0), v_head_size: Int = Int(0)) -> Self
Initialize KVCacheStaticParams. Args: num_heads (Int): Number of attention heads. head_size (Int): Size of each attention head. is_mla (Bool, optional): Whether to use Multi-Linear Attention (MLA) mode. If true, we only store k cache. If False, we store k and v cache. Defaults to False. fp8_tail_size (Int, optional): Trailing bytes of a packed NVFP4 row that hold unscaled FP8 values instead of E2M1 pairs. They have no scales, so they do not count towards the scale row. Defaults to 0. v_head_size (Int, optional): Width of the value packed after the key in each row of a split cache, whose head_size is the key and value widths summed. Defaults to 0, a cache whose key and value each fill a head_size row of their own plane.
is_split
def is_split(self) -> Bool
Whether each row packs a key and a value of different widths.
Returns:
num_planes
def num_planes(self) -> Int
The planes a block holds: two (K and V), or one for an MLA latent or a split cache's packed rows.
Returns:
k_head_size
def k_head_size(self) -> Int
The key's width, which is the whole row unless the cache is split.
Returns:
v_col_base
def v_col_base(self) -> Int
The column the value starts at within a row of its cache view.
Returns:
nvfp4_row_elems
def nvfp4_row_elems(self) -> Int
The logical elements of a packed NVFP4 row: two per packed byte, plus one per FP8 tail byte.
Returns: