IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kv_cache_unpack_prefix_rows

def kv_cache_unpack_prefix_rows[cache_t: KVCacheT, //, *, scale_row_elems: Int, col_offset: Int, dcp_size: Int, dcp_interleave: Int](gathered: TileTensor[.uint8, Engine=gathered.Engine, linear_idx_type=gathered.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, linear_idx_type=row_offsets.linear_idx_type], cache: cache_t, ctx: DeviceContext)

Writes every rank's packed prefix rows into a whole-context cache.

gathered is the group's all-gather of kv_cache_pack_owned_rows, [dcp_size * N, width], rank-major. Its row s * N + row_offsets[b] + j holds rank s's local slot j of request b, global position p = global_pos_s(j); it lands at position p of request b in cache, a cache outside any context-parallel group, if p is below cache_length(b).

Parameters:

  • ​cache_t (KVCacheT): The paged cache type (inferred), one head per row.
  • ​scale_row_elems (Int): Scales per row; 0 for an unscaled cache.
  • ​col_offset (Int): First byte column of gathered the rows sit at.
  • ​dcp_size (Int): Ranks of the context-parallel group.
  • ​dcp_interleave (Int): Consecutive positions a rank owns in turn.

Args:

Was this page helpful?