For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
topk_filtered
The filtered top-k select for long indexer rows that fill the GPU.
persistent_topk_block_split selects such a row with the full histogram
select, which spends tens of instructions on every column. When K is a few
percent of the row, a strided sample bounds the K-th score from below, one
pass keeps the columns at or above the bound, and an exact select over those
runs in shared memory. A row whose bound does not hold is flagged, and a
second launch runs the full select on the flagged rows only. A filtered row
takes the lowest columns among those tied at the K-th score; a flagged row
keeps the full select's unspecified choice among them.
FilteredTopk is the IndexerTopk that takes this select. Hy4's indexer
runs it at every step, through the
attention.mla.indexer.ragged.float8.paged.remap.filtered op; the filtered
launches run only for score windows whose rows fill the GPU, which in
Hy4's serving config is a prefill chunk. Its Moka
component is Kernels/test/gpu/Moka/kernels/select/filtered_select_plan.mojo,
which runs filtered_topk_block, the same CTA body.
comptime values
FILTER_BLOCKS_PER_SM
comptime FILTER_BLOCKS_PER_SM = Int(3)
CTAs of the filtered select an SM holds. The select is latency-bound per row (barriers and dependent loads, not issue slots), so rows in flight are its throughput, and its registers are capped to fit three CTAs.
FILTER_MIN_RATIO
comptime FILTER_MIN_RATIO = Int(4)
The shortest row the filtered select takes, as a multiple of K.
FILTER_POISONS
comptime FILTER_POISONS = Int(4)
Planted faults, poison 1 through 4, each of which a test must catch: 1 writes only the keys strictly above the threshold key, 2 picks the wrong column from the tie list, 3 takes one too many keys above the bf16 step bin, 4 shifts the plateau's tied columns one slot up.
FILTER_SAMPLES
comptime FILTER_SAMPLES = Int(2048)
Scores sampled per row to bound its K-th score.
FILTER_SMEM_BYTES
comptime FILTER_SMEM_BYTES = (Int((mul (Int(512) // get_gpu_info[CompilationTarget._arch()]()), 1024)) * size_of[Int32]())
Shared memory a CTA of the filtered select takes: its histogram, then its keys. The full select takes the same, so a flagged row fits either kernel.
Structs
-
FilteredTopk: The indexer top-k with the filtered select (filtered_topk_block_split).
Functions
-
filtered_topk_block: Selects one row with the filtered select, or flags it for the full one. -
filtered_topk_block_split:persistent_topk_block_splitwithordered=False, deterministic=False, except that bounded rows which fill the GPU take the filtered select. -
filtered_topk_launch: Runs the filtered select's two launches, with the caller's flags.