IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

PagedKVScaleRemap

struct PagedKVScaleRemap[page_size: Int, top_k: Int]

Writes each slot's physical row and its row in the scale pool.

For an NVFP4 cache the scales are paged apart from the values: they have their own block ids and page stride, so a consumer cannot derive a selected token's scale row from its value row. Each output row holds 2 * top_k slots: the PagedKVRemap encoding of the slot (the top_k value rows), then the slot's scale row (the next top_k), which is scale_block * scale_rows_per_block + offset, the same row PagedKVCache.scale_row_idx returns for the position. Either is -1 where the position is -1 or its page is unmapped.

Parameters​

  • ​page_size (Int): Tokens per page of both pools.
  • ​top_k (Int): Slots per output row.

Fields​

  • ​physical_out (MutPointer[Int32, MutUntrackedOrigin]): Flat output buffer, [rows, 2 * top_k].
  • ​lut (ImmPointer[UInt32, ImmUntrackedOrigin]): Row-major [lut_rows, lut_cols] value page lookup table.
  • ​scales_lut (ImmPointer[UInt32, ImmUntrackedOrigin]): Row-major [lut_rows, lut_cols] scale page lookup table.
  • ​lut_cols (Int32): Logical pages per batch entry in lut.
  • ​scales_lut_cols (Int32): Logical pages per batch entry in scales_lut.
  • ​lut_rows (Int32): Batch entries in the tables.
  • ​invalid_block_id (UInt32): Value block ids at or above this (the page count) are invalid.
  • ​scales_invalid_block_id (UInt32): Scale block ids at or above this (the scale page count) are invalid.
  • ​scale_rows_per_block (Int32): Rows between consecutive blocks of the scale pool.

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, Movable, RegisterPassable, SparseIndexRemap, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = PagedKVScaleRemap[page_size, top_k]

Device-side type alias; the struct is trivially device-passable.

enabled​

comptime enabled = True

Physical and scale rows are written.

Methods​

__init__​

def __init__(physical_out: MutPointer[Int32], lut: ImmPointer[UInt32], scales_lut: ImmPointer[UInt32], lut_cols: Int, scales_lut_cols: Int, lut_rows: Int, invalid_block_id: UInt32, scales_invalid_block_id: UInt32, scale_rows_per_block: Int) -> Self

Describes the two pools to remap into and where to write.

Args:

  • ​physical_out (MutPointer[Int32]): Flat output buffer, [rows, 2 * top_k].
  • ​lut (ImmPointer[UInt32]): Row-major [lut_rows, lut_cols] value page lookup table.
  • ​scales_lut (ImmPointer[UInt32]): Row-major [lut_rows, lut_cols] scale page lookup table.
  • ​lut_cols (Int): Logical pages per batch entry in lut.
  • ​scales_lut_cols (Int): Logical pages per batch entry in scales_lut.
  • ​lut_rows (Int): Batch entries in the tables.
  • ​invalid_block_id (UInt32): Value block ids at or above this are invalid; the value pool's page count.
  • ​scales_invalid_block_id (UInt32): Scale block ids at or above this are invalid; the scale pool's page count.
  • ​scale_rows_per_block (Int): Rows between consecutive scale blocks.

offset​

def offset(self, slots: Int) -> Self

Returns this remap with physical_out advanced past slots slots.

Args:

  • ​slots (Int): Slots of the logical output to advance by, a whole number of rows.

Returns:

Self: The advanced remap.

store​

def store(self, idx: Int, batch: Int, logical: Int32)

Writes both rows of slot idx for logical position logical.

Args:

  • ​idx (Int): Flat slot of the logical output, row * top_k + column.
  • ​batch (Int): Batch entry the slot's row belongs to.
  • ​logical (Int32): The slot's logical token position, or -1.

get_type_name​

static def get_type_name() -> String

Returns the type name for runtime diagnostics.

Returns:

String: Always "PagedKVScaleRemap".

Was this page helpful?