For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
BlockTopKMaskScoreHook
struct BlockTopKMaskScoreHook[block_size: Int]
Drops every key whose block of block_size keys a row does not keep.
Each row names its kept blocks as top_k ids, -1 for none; a dropped
key scores -inf. Before a score row-window is scored, its rows' ids
expand into a flag scratch of max_rows rows. The top-k still takes every
key below a row's bound when it has at most top_k of them, so each row
must keep at least that many keys for no dropped key to be selected.
Parameters
- block_size (
Int): Keys per block.
Fields
- ids (
ImmPointer[Int32, ImmUntrackedOrigin]): Kept block ids,[tokens, top_k]. - top_k (
Int32): Ids per row. - keep (
MutPointer[UInt8, MutUntrackedOrigin]): Window flag scratch,[max_rows, num_blocks]. - num_blocks (
Int32): Blocks per row; keys past the last block are dropped. - max_rows (
Int32): Rows the scratch holds; caps the score window.
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
IndexerScoreHook,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
device_type
comptime device_type = BlockTopKMaskScoreHook[block_size]
Device-side type; the struct is trivially device-passable.
enabled
comptime enabled = True
Always runs.
Methods
__init__
def __init__(ids: ImmPointer[Int32, ImmutAnyOrigin], top_k: Int, keep: MutPointer[UInt8, MutAnyOrigin], num_blocks: Int, max_rows: Int) -> Self
Creates the hook.
Args:
- ids (
ImmPointer[Int32, ImmutAnyOrigin]): Kept block ids,[tokens, top_k]. - top_k (
Int): Ids per row. - keep (
MutPointer[UInt8, MutAnyOrigin]): Window flag scratch,[max_rows, num_blocks]. - num_blocks (
Int): Blocks per row. - max_rows (
Int): Rows the scratch holds, at least 1.
window_rows
window
def window(self, ctx: DeviceContext, first_row: Int, rows: Int) -> Self
Expands the window's kept ids into flags and returns its hook.
Args:
- ctx (
DeviceContext): Device context. - first_row (
Int): Query token of the window's row 0. - rows (
Int): Rows in the window, at mostmax_rows.
Returns:
Self: The hook; its scratch row 0 is query token first_row.
after_window
def after_window(self, ctx: DeviceContext, first_row: Int, rows: Int)
Does nothing; the flags are rebuilt before each window.
Args:
score
def score[dtype: DType, lane_keys: Int](self, value: Scalar[dtype], row: Int32, key: Int32, key_live: Bool, row_live: Bool) -> Scalar[dtype]
Returns value if its block is kept, else -inf.
Parameters:
Args:
- value (
Scalar[dtype]): The score. - row (
Int32): Window row of the query token. - key (
Int32): Key column. - key_live (
Bool): Whether the key is below the scorer's bound. - row_live (
Bool): Whether the row is in the window.
Returns:
Scalar[dtype]: The score to store.
get_type_name
static def get_type_name() -> String
Returns the type name for runtime diagnostics.
Returns:
String: Always "BlockTopKMaskScoreHook".