For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kv_cache_pack_owned_rows
def kv_cache_pack_owned_rows[cache_t: KVCacheT, //, *, scale_row_elems: Int, col_offset: Int, dcp_size: Int, dcp_interleave: Int](output: TileTensor[.uint8, Engine=output.Engine, linear_idx_type=output.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type], cache: cache_t, dcp_rank: Int, ctx: DeviceContext)
Packs the cached rows this context-parallel rank owns into output.
Request b owns output rows [row_offsets[b], row_offsets[b + 1]), at
least max_local_len(cache_length(b)) of them: output row
row_offsets[b] + j takes the request's local slot j, the global
position global_pos(j) this rank stores (ContextParallelLayout). A
slot past the rank's local_len is left as it is. Each row is the
cache's raw bytes, the value row then its scale_row_elems scales, at
byte column col_offset of output; rows past row_offsets[B] are
left as they are.
Every rank of a group packs the same row layout, so after an all-gather
the row of position p of request b is
owner(p) * N + row_offsets[b] + local_slot(p), with N the rows each
rank packed (kv_cache_unpack_prefix_rows).
Parameters:
- cache_t (
KVCacheT): The paged cache type (inferred), one head per row. - scale_row_elems (
Int): Scales per row; 0 for an unscaled cache. - col_offset (
Int): First byte column ofoutputthe rows land at. - dcp_size (
Int): Ranks of the context-parallel group. - dcp_interleave (
Int): Consecutive positions a rank owns in turn.
Args:
- output (
TileTensor[.uint8, Engine=output.Engine, linear_idx_type=output.linear_idx_type]):[N, width]bytes, contiguous rows. - row_offsets (
TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type]):[B + 1]output row offsets per request. - cache (
cache_t): One layer of this rank's context-parallel cache, whosecache_lengthis each request's global prefix length. - dcp_rank (
Int): This rank in[0, dcp_size). - ctx (
DeviceContext): Device context used to enqueue the GPU kernel.