IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kv_cache_pack_owned_rows

def kv_cache_pack_owned_rows[cache_t: KVCacheT, //, *, scale_row_elems: Int, col_offset: Int, dcp_size: Int, dcp_interleave: Int](output: TileTensor[.uint8, Engine=output.Engine, linear_idx_type=output.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type], cache: cache_t, dcp_rank: Int, ctx: DeviceContext)

Packs the cached rows this context-parallel rank owns into output.

Request b owns output rows [row_offsets[b], row_offsets[b + 1]), at least max_local_len(cache_length(b)) of them: output row row_offsets[b] + j takes the request's local slot j, the global position global_pos(j) this rank stores (ContextParallelLayout). A slot past the rank's local_len is left as it is. Each row is the cache's raw bytes, the value row then its scale_row_elems scales, at byte column col_offset of output; rows past row_offsets[B] are left as they are.

Every rank of a group packs the same row layout, so after an all-gather the row of position p of request b is owner(p) * N + row_offsets[b] + local_slot(p), with N the rows each rank packed (kv_cache_unpack_prefix_rows).

Parameters:

  • ​cache_t (KVCacheT): The paged cache type (inferred), one head per row.
  • ​scale_row_elems (Int): Scales per row; 0 for an unscaled cache.
  • ​col_offset (Int): First byte column of output the rows land at.
  • ​dcp_size (Int): Ranks of the context-parallel group.
  • ​dcp_interleave (Int): Consecutive positions a rank owns in turn.

Args:

Was this page helpful?