IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

paged_cache_rows

def paged_cache_rows[v_head_size: Int = Int(-1)](blocks: TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type], split_args: TileTensor[.uint32, Engine=split_args.Engine, linear_idx_type=split_args.linear_idx_type], first_row: Int, num_rows: Int, max_lengths_slot: Int, out result: PagedKVCacheCollection[blocks.dtype, _paged_cache_static_params(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], blocks.LayoutType.static_shape[Int(1)], v_head_size), blocks.LayoutType.static_shape[Int(3)], MutAnyOrigin, ImmutAnyOrigin, ImmutAnyOrigin, MutUntrackedOrigin])

Builds the paged cache over batch rows [first_row, first_row + num_rows), with max lengths from split_args[max_lengths_slot:+2].

Same type as the generic_get_paged_cache collection of the whole batch; only the per-row tables move and the max lengths change. The caller checks that the lookup table and split_args (a kv_cache.mixed_step_split record) are packed. Use it to build a collection from operands; rows_view views an existing collection and keeps its scales.

Returns:

PagedKVCacheCollection[blocks.dtype, _paged_cache_static_params(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], blocks.LayoutType.static_shape[Int(1)], v_head_size), blocks.LayoutType.static_shape[Int(3)], MutAnyOrigin, ImmutAnyOrigin, ImmutAnyOrigin, MutUntrackedOrigin]

Was this page helpful?