IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

RelativeLogitsMask

struct RelativeLogitsMask[window_size: Int = Int(0), Bias: Loadable & TrivialRegisterPassable = NoView, CacheLengths: Loadable & TrivialRegisterPassable = NoView, RowOffsets: Loadable & TrivialRegisterPassable = NoView]

Causal (or sliding-window causal when window_size > 0) visibility, plus an additive bias by relative distance.

The bias is gathered by rel_dist = q - k from a (tokens, heads, extent) table and added on every visible position; distances outside [0, extent) carry no bias. Masked lanes come out as exactly MASK_VALUE.

Operands: the bias, the uint32 cache lengths [batch] (tokens cached before this call's), and the uint32 input row offsets [batch + 1].

Fields​

  • ​bias (Bias):
  • ​cache_lengths (CacheLengths):
  • ​row_offsets (RowOffsets):

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, MHAMask, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

apply_log2e_after_mask​

comptime apply_log2e_after_mask = True

check_mask_during_decoding​

comptime check_mask_during_decoding = True

device_type​

comptime device_type = RelativeLogitsMask[window_size, Bias, CacheLengths, RowOffsets]

has_row_bias​

comptime has_row_bias = True

mask_out_of_bound​

comptime mask_out_of_bound = is_nvidia_gpu()

mask_safe_out_of_bounds​

comptime mask_safe_out_of_bounds = True

MaskType​

comptime MaskType[*Ts: Fused & Input & LoadAccess] = RelativeLogitsMask[window_size, values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), values[2](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]

Parameters​

supports_decode_token_fold​

comptime supports_decode_token_fold = False

Whether AMD decode may fold several query tokens into one tile under this mask, and start its split-K key range at start_column: both status and start_column stay valid when asked about the span of query rows a sequence's decode step carries.

Methods​

make​

static def make[*Ts: Fused & Input & LoadAccess](operands: TensorTuple[*Ts.values]) -> RelativeLogitsMask[window_size, Ts.values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[2](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]

Builds the mask from the bias, cache lengths and row offsets.

Parameters:

  • ​*Ts (Fused & Input & LoadAccess): The operand types: the bias, the cache lengths and the row offsets.

Args:

Returns:

RelativeLogitsMask[window_size, Ts.values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[2](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]: The mask.

get_type_name​

static def get_type_name() -> String

Returns:

String

name​

static def name() -> String

Returns:

String

visibility_mask​

def visibility_mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]

Returns:

SIMD[dtype, width]

mask​

def mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]

Returns:

SIMD[dtype, width]

add_row_bias​

def add_row_bias[BN: Int, //](self, mut srow: Array[Float32, BN], seq_id: UInt32, head: UInt32, q_idx: UInt32, k_start: Int32)

status​

def status(self, seq_id: UInt32, tile_offset: Coord[Int, Int], tile_size: Coord[Int, Int]) -> TileMaskStatus

Returns:

TileMaskStatus

start_column​

def start_column[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32) -> UInt32

Returns:

UInt32

start_column_alignment​

static def start_column_alignment[BM: Int, BN: Int, page_size: Int]() -> Int

Returns:

Int

total_iters​

def total_iters[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32

Returns:

UInt32

count_nonfull_sets​

static def count_nonfull_sets(BM: Int, BN: Int) -> Int

Returns:

Int

last_masked_set_end​

def last_masked_set_end[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32

Returns:

UInt32

masked_set_ends​

def masked_set_ends[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> StaticTuple[UInt32, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[UInt32, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

nonfull_sets​

static def nonfull_sets[BM: Int, BN: Int]() -> StaticTuple[TileMaskStatus, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[TileMaskStatus, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

mask_strategies​

static def mask_strategies[BM: Int, BN: Int]() -> StaticTuple[MaskStrategy, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[MaskStrategy, RelativeLogitsMask.count_nonfull_sets(BM, BN)]

mask_bits​

def mask_bits(self, seq_id: UInt32, score_row: Int32, col_start: Int32, num_keys: Int32) -> UInt32

Returns:

UInt32

sliding_window_size​

static def sliding_window_size() -> Int

Returns:

Int

Was this page helpful?