IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mixed_step_split

The mixed-step split record of a paged KV cache input.

A mixed step batches decode rows with prefill rows, so an op that picks one route from its longest row sends the decode rows down the prefill route. The KV cache managers fill a host record (mixed_step_split, uint32[MIXED_STEP_SPLIT_SIZE]) for every batch, which cuts one replica batch into a head and a tail view; an op validates it with read_mixed_split, runs each view through its own route, and keeps the unsplit route for any record or operand it cannot use.

comptime values​

MIXED_STEP_DECODE_WIDTH​

comptime MIXED_STEP_DECODE_WIDTH = 8

MIXED_STEP_SPLIT_SIZE​

comptime MIXED_STEP_SPLIT_SIZE = 7

SPLIT_HEAD_LENGTHS_SLOT​

comptime SPLIT_HEAD_LENGTHS_SLOT = 2

SPLIT_PREFILL_CONTEXT_SLOT​

comptime SPLIT_PREFILL_CONTEXT_SLOT = 6

SPLIT_TAIL_LENGTHS_SLOT​

comptime SPLIT_TAIL_LENGTHS_SLOT = 4

Structs​

  • ​MixedSplit: A validated split of one mixed batch into a head and a tail view.
  • ​SplitView: One row range of a split batch and its max lengths.

Functions​

  • ​is_packed: Whether t is packed row-major.
  • ​read_mixed_split: Returns the split the host record describes, or None to keep the unsplit prefill route.
  • ​rebase_row_offsets: Returns the tail view's offsets rebased to start at 0, in one 1-CTA launch: [0, view.rows] query offsets, then the cache offsets when with_cache_offsets.

Was this page helpful?