For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
RelativeLogitsMask
struct RelativeLogitsMask[window_size: Int = Int(0), Bias: Loadable & TrivialRegisterPassable = NoView, CacheLengths: Loadable & TrivialRegisterPassable = NoView, RowOffsets: Loadable & TrivialRegisterPassable = NoView]
Causal (or sliding-window causal when window_size > 0) visibility, plus an additive bias by relative distance.
The bias is gathered by rel_dist = q - k from a (tokens, heads, extent)
table and added on every visible position; distances outside
[0, extent) carry no bias. Masked lanes come out as exactly
MASK_VALUE.
Operands: the bias, the uint32 cache lengths [batch] (tokens cached
before this call's), and the uint32 input row offsets [batch + 1].
Fields
- bias (
Bias): - cache_lengths (
CacheLengths): - row_offsets (
RowOffsets):
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
MHAMask,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
apply_log2e_after_mask
comptime apply_log2e_after_mask = True
check_mask_during_decoding
comptime check_mask_during_decoding = True
device_type
comptime device_type = RelativeLogitsMask[window_size, Bias, CacheLengths, RowOffsets]
has_row_bias
comptime has_row_bias = True
mask_out_of_bound
comptime mask_out_of_bound = is_nvidia_gpu()
mask_safe_out_of_bounds
comptime mask_safe_out_of_bounds = True
MaskType
comptime MaskType[*Ts: Fused & Input & LoadAccess] = RelativeLogitsMask[window_size, values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), values[2](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]
Parameters
- *Ts (
Fused&Input&LoadAccess):
supports_decode_token_fold
comptime supports_decode_token_fold = False
Whether AMD decode may fold several query tokens into one tile under this mask, and start its split-K key range at start_column: both status and start_column stay valid when asked about the span of query rows a sequence's decode step carries.
Methods
make
static def make[*Ts: Fused & Input & LoadAccess](operands: TensorTuple[*Ts.values]) -> RelativeLogitsMask[window_size, Ts.values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[2](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]
Builds the mask from the bias, cache lengths and row offsets.
Parameters:
- *Ts (
Fused&Input&LoadAccess): The operand types: the bias, the cache lengths and the row offsets.
Args:
- operands (
TensorTuple[*Ts.values]): The bias, the cache lengths and the row offsets.
Returns:
get_type_name
name
visibility_mask
def visibility_mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]
Returns:
mask
def mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]
Returns:
add_row_bias
def add_row_bias[BN: Int, //](self, mut srow: Array[Float32, BN], seq_id: UInt32, head: UInt32, q_idx: UInt32, k_start: Int32)
status
def status(self, seq_id: UInt32, tile_offset: Coord[Int, Int], tile_size: Coord[Int, Int]) -> TileMaskStatus
Returns:
start_column
def start_column[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32) -> UInt32
Returns:
start_column_alignment
total_iters
def total_iters[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32
Returns:
count_nonfull_sets
last_masked_set_end
def last_masked_set_end[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32
Returns:
masked_set_ends
def masked_set_ends[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> StaticTuple[UInt32, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
Returns:
StaticTuple[UInt32, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
nonfull_sets
static def nonfull_sets[BM: Int, BN: Int]() -> StaticTuple[TileMaskStatus, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
Returns:
StaticTuple[TileMaskStatus, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
mask_strategies
static def mask_strategies[BM: Int, BN: Int]() -> StaticTuple[MaskStrategy, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
Returns:
StaticTuple[MaskStrategy, RelativeLogitsMask.count_nonfull_sets(BM, BN)]
mask_bits
def mask_bits(self, seq_id: UInt32, score_row: Int32, col_start: Int32, num_keys: Int32) -> UInt32
Returns: