IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

KVCacheStaticParams

struct KVCacheStaticParams

Compile-time shape parameters shared across all layers of a KV cache.

Groups the attention-head count and per-head size that are fixed for the entire model lifetime, along with the Multi-head Latent Attention flag that changes the KV layout from two caches (K + V) to one (K only).

A split cache (v_head_size > 0) also has one plane, but each row packs a key of k_head_size() columns and then a value of v_head_size columns. head_size stays the row pitch, so row arithmetic is unchanged; value accesses add v_col_base() to the column.

Fields​

  • ​num_heads (Int):
  • ​head_size (Int):
  • ​is_mla (Bool):
  • ​fp8_tail_size (Int):
  • ​v_head_size (Int):

Implemented traits​

AnyType, Copyable, Deinitable, Equatable, ImplicitlyCopyable, Movable, RegisterPassable, TrivialRegisterPassable

Methods​

__init__​

def __init__(num_heads: Int, head_size: Int, is_mla: Bool = False, fp8_tail_size: Int = Int(0), v_head_size: Int = Int(0)) -> Self

Initialize KVCacheStaticParams. Args: num_heads (Int): Number of attention heads. head_size (Int): Size of each attention head. is_mla (Bool, optional): Whether to use Multi-Linear Attention (MLA) mode. If true, we only store k cache. If False, we store k and v cache. Defaults to False. fp8_tail_size (Int, optional): Trailing bytes of a packed NVFP4 row that hold unscaled FP8 values instead of E2M1 pairs. They have no scales, so they do not count towards the scale row. Defaults to 0. v_head_size (Int, optional): Width of the value packed after the key in each row of a split cache, whose head_size is the key and value widths summed. Defaults to 0, a cache whose key and value each fill a head_size row of their own plane.

is_split​

def is_split(self) -> Bool

Whether each row packs a key and a value of different widths.

Returns:

Bool

num_planes​

def num_planes(self) -> Int

The planes a block holds: two (K and V), or one for an MLA latent or a split cache's packed rows.

Returns:

Int

k_head_size​

def k_head_size(self) -> Int

The key's width, which is the whole row unless the cache is split.

Returns:

Int

v_col_base​

def v_col_base(self) -> Int

The column the value starts at within a row of its cache view.

Returns:

Int

nvfp4_row_elems​

def nvfp4_row_elems(self) -> Int

The logical elements of a packed NVFP4 row: two per packed byte, plus one per FP8 tail byte.

Returns:

Int

Was this page helpful?