For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
PagedKVScaleRemap
struct PagedKVScaleRemap[page_size: Int, top_k: Int]
Writes each slot's physical row and its row in the scale pool.
For an NVFP4 cache the scales are paged apart from the values: they have
their own block ids and page stride, so a consumer cannot derive a
selected token's scale row from its value row. Each output row holds
2 * top_k slots: the PagedKVRemap encoding of the slot (the
top_k value rows), then the slot's scale row (the next top_k), which is
scale_block * scale_rows_per_block + offset, the same row
PagedKVCache.scale_row_idx returns for the position. Either is -1 where
the position is -1 or its page is unmapped.
Parameters
Fields
- physical_out (
MutPointer[Int32, MutUntrackedOrigin]): Flat output buffer,[rows, 2 * top_k]. - lut (
ImmPointer[UInt32, ImmUntrackedOrigin]): Row-major[lut_rows, lut_cols]value page lookup table. - scales_lut (
ImmPointer[UInt32, ImmUntrackedOrigin]): Row-major[lut_rows, lut_cols]scale page lookup table. - lut_cols (
Int32): Logical pages per batch entry inlut. - scales_lut_cols (
Int32): Logical pages per batch entry inscales_lut. - lut_rows (
Int32): Batch entries in the tables. - invalid_block_id (
UInt32): Value block ids at or above this (the page count) are invalid. - scales_invalid_block_id (
UInt32): Scale block ids at or above this (the scale page count) are invalid. - scale_rows_per_block (
Int32): Rows between consecutive blocks of the scale pool.
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
Movable,
RegisterPassable,
SparseIndexRemap,
TrivialRegisterPassable
comptime members
device_type
comptime device_type = PagedKVScaleRemap[page_size, top_k]
Device-side type alias; the struct is trivially device-passable.
enabled
comptime enabled = True
Physical and scale rows are written.
Methods
__init__
def __init__(physical_out: MutPointer[Int32], lut: ImmPointer[UInt32], scales_lut: ImmPointer[UInt32], lut_cols: Int, scales_lut_cols: Int, lut_rows: Int, invalid_block_id: UInt32, scales_invalid_block_id: UInt32, scale_rows_per_block: Int) -> Self
Describes the two pools to remap into and where to write.
Args:
- physical_out (
MutPointer[Int32]): Flat output buffer,[rows, 2 * top_k]. - lut (
ImmPointer[UInt32]): Row-major[lut_rows, lut_cols]value page lookup table. - scales_lut (
ImmPointer[UInt32]): Row-major[lut_rows, lut_cols]scale page lookup table. - lut_cols (
Int): Logical pages per batch entry inlut. - scales_lut_cols (
Int): Logical pages per batch entry inscales_lut. - lut_rows (
Int): Batch entries in the tables. - invalid_block_id (
UInt32): Value block ids at or above this are invalid; the value pool's page count. - scales_invalid_block_id (
UInt32): Scale block ids at or above this are invalid; the scale pool's page count. - scale_rows_per_block (
Int): Rows between consecutive scale blocks.
offset
def offset(self, slots: Int) -> Self
Returns this remap with physical_out advanced past slots slots.
Args:
- slots (
Int): Slots of the logical output to advance by, a whole number of rows.
Returns:
Self: The advanced remap.
store
def store(self, idx: Int, batch: Int, logical: Int32)
Writes both rows of slot idx for logical position logical.
Args:
get_type_name
static def get_type_name() -> String
Returns the type name for runtime diagnostics.
Returns:
String: Always "PagedKVScaleRemap".