IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BlockTopKScoreHook

struct BlockTopKScoreHook[block_size: Int]

Writes the ids of each row's top_k best blocks of block_size keys.

A block's value is the maximum score of its keys below the scorer's bound, -inf if none is. Each row keeps the block of its newest visible key, ((position + 1) // ratio - 1) // block_size, then the best of the blocks before it; -inf blocks are never kept. Which of the blocks tied with the k-th best are kept is unspecified. A row's ids come in no particular order, with -1 in the slots it leaves unused.

The maxima of one score row-window at a time live in a scratch buffer of max_rows rows, and the selection runs after each window is scored, so no [tokens, num_blocks] buffer exists.

Parameters​

  • ​block_size (Int): Keys per block, a power of two.

Fields​

  • ​block_max (MutPointer[Float32, MutUntrackedOrigin]): Window scratch, [max_rows, num_blocks] float32.
  • ​num_blocks (Int32): Blocks per row.
  • ​max_rows (Int32): Rows the scratch holds; caps the score window.
  • ​live (MutPointer[Int32, MutUntrackedOrigin]): Window scratch, [max_rows] int32 live block counts.
  • ​ids (MutPointer[Int32, MutUntrackedOrigin]): Kept block ids output, [tokens, top_k].
  • ​top_k (Int32): Blocks kept per row.
  • ​positions (ImmPointer[Int32, ImmUntrackedOrigin]): Position of each query token, [tokens].
  • ​ratio (Int32): Tokens per key.

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, IndexerScoreHook, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = BlockTopKScoreHook[block_size]

Device-side type; the struct is trivially device-passable.

enabled​

comptime enabled = True

Always runs.

Methods​

__init__​

def __init__(block_max: MutPointer[Float32, MutAnyOrigin], num_blocks: Int, max_rows: Int, live: MutPointer[Int32, MutAnyOrigin], ids: MutPointer[Int32, MutAnyOrigin], top_k: Int, positions: ImmPointer[Int32, ImmutAnyOrigin], ratio: Int) -> Self

Creates the hook.

Args:

window_rows​

def window_rows(self) -> Int

Returns the rows the scratch holds.

Returns:

Int: max_rows.

window​

def window(self, ctx: DeviceContext, first_row: Int, rows: Int) -> Self

Sets the window's maxima to -inf and returns its hook.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​first_row (Int): Query token of the window's row 0.
  • ​rows (Int): Rows in the window, at most max_rows.

Returns:

Self: The hook; its scratch row 0 is query token first_row.

after_window​

def after_window(self, ctx: DeviceContext, first_row: Int, rows: Int)

Selects the window's kept blocks into rows of ids.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​first_row (Int): Query token of the window's row 0.
  • ​rows (Int): Rows in the window.

score​

def score[dtype: DType, lane_keys: Int](self, value: Scalar[dtype], row: Int32, key: Int32, key_live: Bool, row_live: Bool) -> Scalar[dtype]

Folds value into its block's maximum and returns it unchanged.

Parameters:

  • ​dtype (DType): Score element type.
  • ​lane_keys (Int): Consecutive keys a warp holds.

Args:

  • ​value (Scalar[dtype]): The score.
  • ​row (Int32): Window row of the query token.
  • ​key (Int32): Key column.
  • ​key_live (Bool): Whether the key is below the scorer's bound.
  • ​row_live (Bool): Whether the row is in the window.

Returns:

Scalar[dtype]: value.

get_type_name​

static def get_type_name() -> String

Returns the type name for runtime diagnostics.

Returns:

String: Always "BlockTopKScoreHook".

Was this page helpful?