For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
hisparse
HiSparse hot-buffer kernels for sparse MLA over a partly host-resident cache.
A request in mixed residency has latent pages whose lookup table entry is the
null page 0: their only copy lives in a pinned host mirror. The request also
holds a hot region of hot_pages latent pages plus one state page, listed in
its row of the hot table. Sparse MLA reads hot rows like any other cache rows,
because they are rows of the same paged cache.
hisparse_resolve runs once per indexer layer, after the indexer has written
each selection's physical row. It rewrites the selections on null pages to hot
rows. A selection already held in a hot slot reads it, and a miss takes the
least recently used slot that no row of this step needs. Misses are numbered by
their first reference in (row, top-k) order, so the plan is deterministic. They
are queued in the state page as (hot row, host row) pairs, and
hisparse_gather_group copies them for every layer sharing an indexer's
selections in one launch, before the first layer's attention.
Rows on resident pages are left as the indexer wrote them, so the packing that
sparse MLA expects (valid rows first, -1 after) is unchanged. A request with
no hot region (a zero state page) is skipped.
The state page is zero when leased. Everything the kernels keep across steps
is laid out so that zero reads as empty: slot keys are position + 1, slot hit
marks carry a per-call epoch that starts at 1, and a group's LRU order is
initialized on its first call. The hash of one call's selections lives in
shared memory and is cleared at the start of the call, since nothing reads it
afterwards.
comptime values
GATHER_BLOCK
comptime GATHER_BLOCK = 256
Threads per gather block; one warp copies one row.
GATHER_BLOCKS_PER_REQUEST
comptime GATHER_BLOCKS_PER_REQUEST = 16
Gather blocks per request; they stride over its misses.
RESOLVE_BLOCK
comptime RESOLVE_BLOCK = 1024
Threads per request in hisparse_resolve.
Structs
-
HiSparseStateLayout: Byte layout of a request's HiSparse state page.
Functions
-
hisparse_gather_group: Copies the last resolve's misses for consecutive layers in one launch. -
hisparse_resolve: Rewrites selections on host-only pages to hot-buffer rows. -
hisparse_state_bytes: Returns the bytes a request's state page needs.