For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
BlockTopKScoreHook
struct BlockTopKScoreHook[block_size: Int]
Writes the ids of each row's top_k best blocks of block_size keys.
A block's value is the maximum score of its keys below the scorer's bound,
-inf if none is. Each row keeps the block of its newest visible key,
((position + 1) // ratio - 1) // block_size, then the best of the
blocks before it; -inf blocks are never kept. Which of the blocks tied
with the k-th best are kept is unspecified. A row's ids come in no
particular order, with -1 in the slots it leaves unused.
The maxima of one score row-window at a time live in a scratch buffer of
max_rows rows, and the selection runs after each window is scored, so
no [tokens, num_blocks] buffer exists.
Parameters
- block_size (
Int): Keys per block, a power of two.
Fields
- block_max (
MutPointer[Float32, MutUntrackedOrigin]): Window scratch,[max_rows, num_blocks]float32. - num_blocks (
Int32): Blocks per row. - max_rows (
Int32): Rows the scratch holds; caps the score window. - live (
MutPointer[Int32, MutUntrackedOrigin]): Window scratch,[max_rows]int32 live block counts. - ids (
MutPointer[Int32, MutUntrackedOrigin]): Kept block ids output,[tokens, top_k]. - top_k (
Int32): Blocks kept per row. - positions (
ImmPointer[Int32, ImmUntrackedOrigin]): Position of each query token,[tokens]. - ratio (
Int32): Tokens per key.
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
IndexerScoreHook,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
device_type
comptime device_type = BlockTopKScoreHook[block_size]
Device-side type; the struct is trivially device-passable.
enabled
comptime enabled = True
Always runs.
Methods
__init__
def __init__(block_max: MutPointer[Float32, MutAnyOrigin], num_blocks: Int, max_rows: Int, live: MutPointer[Int32, MutAnyOrigin], ids: MutPointer[Int32, MutAnyOrigin], top_k: Int, positions: ImmPointer[Int32, ImmutAnyOrigin], ratio: Int) -> Self
Creates the hook.
Args:
- block_max (
MutPointer[Float32, MutAnyOrigin]): Window scratch,[max_rows, num_blocks]float32, 16 B aligned. - num_blocks (
Int): Blocks per row. - max_rows (
Int): Rows the scratch holds, at least 1. - live (
MutPointer[Int32, MutAnyOrigin]): Window scratch,[max_rows]int32. - ids (
MutPointer[Int32, MutAnyOrigin]): Kept block ids output,[tokens, top_k]. - top_k (
Int): Blocks kept per row, at mostPERSISTENT_TOPK_MAX_N. - positions (
ImmPointer[Int32, ImmutAnyOrigin]): Position of each query token,[tokens]. - ratio (
Int): Tokens per key.
window_rows
window
def window(self, ctx: DeviceContext, first_row: Int, rows: Int) -> Self
Sets the window's maxima to -inf and returns its hook.
Args:
- ctx (
DeviceContext): Device context. - first_row (
Int): Query token of the window's row 0. - rows (
Int): Rows in the window, at mostmax_rows.
Returns:
Self: The hook; its scratch row 0 is query token first_row.
after_window
def after_window(self, ctx: DeviceContext, first_row: Int, rows: Int)
Selects the window's kept blocks into rows of ids.
Args:
score
def score[dtype: DType, lane_keys: Int](self, value: Scalar[dtype], row: Int32, key: Int32, key_live: Bool, row_live: Bool) -> Scalar[dtype]
Folds value into its block's maximum and returns it unchanged.
Parameters:
Args:
- value (
Scalar[dtype]): The score. - row (
Int32): Window row of the query token. - key (
Int32): Key column. - key_live (
Bool): Whether the key is below the scorer's bound. - row_live (
Bool): Whether the row is in the window.
Returns:
Scalar[dtype]: value.
get_type_name
static def get_type_name() -> String
Returns the type name for runtime diagnostics.
Returns:
String: Always "BlockTopKScoreHook".