IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BlockTopKMaskScoreHook

struct BlockTopKMaskScoreHook[block_size: Int]

Drops every key whose block of block_size keys a row does not keep.

Each row names its kept blocks as top_k ids, -1 for none; a dropped key scores -inf. Before a score row-window is scored, its rows' ids expand into a flag scratch of max_rows rows. The top-k still takes every key below a row's bound when it has at most top_k of them, so each row must keep at least that many keys for no dropped key to be selected.

Parameters​

  • ​block_size (Int): Keys per block.

Fields​

  • ​ids (ImmPointer[Int32, ImmUntrackedOrigin]): Kept block ids, [tokens, top_k].
  • ​top_k (Int32): Ids per row.
  • ​keep (MutPointer[UInt8, MutUntrackedOrigin]): Window flag scratch, [max_rows, num_blocks].
  • ​num_blocks (Int32): Blocks per row; keys past the last block are dropped.
  • ​max_rows (Int32): Rows the scratch holds; caps the score window.

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, IndexerScoreHook, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = BlockTopKMaskScoreHook[block_size]

Device-side type; the struct is trivially device-passable.

enabled​

comptime enabled = True

Always runs.

Methods​

__init__​

def __init__(ids: ImmPointer[Int32, ImmutAnyOrigin], top_k: Int, keep: MutPointer[UInt8, MutAnyOrigin], num_blocks: Int, max_rows: Int) -> Self

Creates the hook.

Args:

window_rows​

def window_rows(self) -> Int

Returns the rows the scratch holds.

Returns:

Int: max_rows.

window​

def window(self, ctx: DeviceContext, first_row: Int, rows: Int) -> Self

Expands the window's kept ids into flags and returns its hook.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​first_row (Int): Query token of the window's row 0.
  • ​rows (Int): Rows in the window, at most max_rows.

Returns:

Self: The hook; its scratch row 0 is query token first_row.

after_window​

def after_window(self, ctx: DeviceContext, first_row: Int, rows: Int)

Does nothing; the flags are rebuilt before each window.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​first_row (Int): Query token of the window's row 0.
  • ​rows (Int): Rows in the window.

score​

def score[dtype: DType, lane_keys: Int](self, value: Scalar[dtype], row: Int32, key: Int32, key_live: Bool, row_live: Bool) -> Scalar[dtype]

Returns value if its block is kept, else -inf.

Parameters:

  • ​dtype (DType): Score element type.
  • ​lane_keys (Int): Consecutive keys a warp holds.

Args:

  • ​value (Scalar[dtype]): The score.
  • ​row (Int32): Window row of the query token.
  • ​key (Int32): Key column.
  • ​key_live (Bool): Whether the key is below the scorer's bound.
  • ​row_live (Bool): Whether the row is in the window.

Returns:

Scalar[dtype]: The score to store.

get_type_name​

static def get_type_name() -> String

Returns the type name for runtime diagnostics.

Returns:

String: Always "BlockTopKMaskScoreHook".

Was this page helpful?