For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kv_cache_unpack_prefix_rows
def kv_cache_unpack_prefix_rows[cache_t: KVCacheT, //, *, scale_row_elems: Int, col_offset: Int, dcp_size: Int, dcp_interleave: Int](gathered: TileTensor[.uint8, Engine=gathered.Engine, linear_idx_type=gathered.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type], cache: cache_t, ctx: DeviceContext)
Writes every rank's packed prefix rows into a whole-context cache.
gathered is the group's all-gather of kv_cache_pack_owned_rows,
[dcp_size * N, width], rank-major. Its row s * N + row_offsets[b] + j holds rank s's local slot j of request b, global position
p = global_pos_s(j); it lands at position p of request b in
cache, a cache outside any context-parallel group, if p is below
cache_length(b).
Parameters:
- cache_t (
KVCacheT): The paged cache type (inferred), one head per row. - scale_row_elems (
Int): Scales per row; 0 for an unscaled cache. - col_offset (
Int): First byte column ofgatheredthe rows sit at. - dcp_size (
Int): Ranks of the context-parallel group. - dcp_interleave (
Int): Consecutive positions a rank owns in turn.
Args:
- gathered (
TileTensor[.uint8, Engine=gathered.Engine, linear_idx_type=gathered.linear_idx_type]):[dcp_size * N, width]bytes, contiguous rows. - row_offsets (
TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type]):[B + 1]row offsets of each rank's packed rows. - cache (
cache_t): One layer of the whole-context cache, whosecache_lengthis each request's prefix length. - ctx (
DeviceContext): Device context used to enqueue the GPU kernel.