IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

topk_filtered

The filtered top-k select for long indexer rows that fill the GPU.

persistent_topk_block_split selects such a row with the full histogram select, which spends tens of instructions on every column. When K is a few percent of the row, a strided sample bounds the K-th score from below, one pass keeps the columns at or above the bound, and an exact select over those runs in shared memory. A row whose bound does not hold is flagged, and a second launch runs the full select on the flagged rows only. A filtered row takes the lowest columns among those tied at the K-th score; a flagged row keeps the full select's unspecified choice among them.

FilteredTopk is the IndexerTopk that takes this select. Hy4's indexer runs it at every step, through the attention.mla.indexer.ragged.float8.paged.remap.filtered op; the filtered launches run only for score windows whose rows fill the GPU, which in Hy4's serving config is a prefill chunk. Its Moka component is Kernels/test/gpu/Moka/kernels/select/filtered_select_plan.mojo, which runs filtered_topk_block, the same CTA body.

comptime values​

FILTER_BLOCKS_PER_SM​

comptime FILTER_BLOCKS_PER_SM = Int(3)

CTAs of the filtered select an SM holds. The select is latency-bound per row (barriers and dependent loads, not issue slots), so rows in flight are its throughput, and its registers are capped to fit three CTAs.

FILTER_MIN_RATIO​

comptime FILTER_MIN_RATIO = Int(4)

The shortest row the filtered select takes, as a multiple of K.

FILTER_POISONS​

comptime FILTER_POISONS = Int(4)

Planted faults, poison 1 through 4, each of which a test must catch: 1 writes only the keys strictly above the threshold key, 2 picks the wrong column from the tie list, 3 takes one too many keys above the bf16 step bin, 4 shifts the plateau's tied columns one slot up.

FILTER_SAMPLES​

comptime FILTER_SAMPLES = Int(2048)

Scores sampled per row to bound its K-th score.

FILTER_SMEM_BYTES​

comptime FILTER_SMEM_BYTES = (Int((mul (Int(512) // get_gpu_info[CompilationTarget._arch()]()), 1024)) * size_of[Int32]())

Shared memory a CTA of the filtered select takes: its histogram, then its keys. The full select takes the same, so a flagged row fits either kernel.

Structs​

  • ​FilteredTopk: The indexer top-k with the filtered select (filtered_topk_block_split).

Functions​

Was this page helpful?