IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

hisparse

HiSparse hot-buffer kernels for sparse MLA over a partly host-resident cache.

A request in mixed residency has latent pages whose lookup table entry is the null page 0: their only copy lives in a pinned host mirror. The request also holds a hot region of hot_pages latent pages plus one state page, listed in its row of the hot table. Sparse MLA reads hot rows like any other cache rows, because they are rows of the same paged cache.

hisparse_resolve runs once per indexer layer, after the indexer has written each selection's physical row. It rewrites the selections on null pages to hot rows. A selection already held in a hot slot reads it, and a miss takes the least recently used slot that no row of this step needs. Misses are numbered by their first reference in (row, top-k) order, so the plan is deterministic. They are queued in the state page as (hot row, host row) pairs, and hisparse_gather_group copies them for every layer sharing an indexer's selections in one launch, before the first layer's attention.

Rows on resident pages are left as the indexer wrote them, so the packing that sparse MLA expects (valid rows first, -1 after) is unchanged. A request with no hot region (a zero state page) is skipped.

The state page is zero when leased. Everything the kernels keep across steps is laid out so that zero reads as empty: slot keys are position + 1, slot hit marks carry a per-call epoch that starts at 1, and a group's LRU order is initialized on its first call. The hash of one call's selections lives in shared memory and is cleared at the start of the call, since nothing reads it afterwards.

comptime values​

GATHER_BLOCK​

comptime GATHER_BLOCK = 256

Threads per gather block; one warp copies one row.

GATHER_BLOCKS_PER_REQUEST​

comptime GATHER_BLOCKS_PER_REQUEST = 16

Gather blocks per request; they stride over its misses.

RESOLVE_BLOCK​

comptime RESOLVE_BLOCK = 1024

Threads per request in hisparse_resolve.

Structs​

Functions​

Was this page helpful?