IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

SegmentMask

struct SegmentMask[Ids: Loadable & TrivialRegisterPassable = NoView, Offsets: Loadable & TrivialRegisterPassable = NoView]

Segment mask over ragged sequences: a query sees a key iff either token's segment id is -1 or the two ids are equal.

Tokens with id -1 see, and are seen by, every token of their sequence; every other segment sees only itself and the -1 tokens. Within each sequence the ids must be nondecreasing, with -1 the only negative id, so a tile's status follows from the ids at its corners, and the keys a query tile sees lie in one range of columns.

Operands: one int32 segment id per token of the ragged batch, and the batch's [batch + 1] uint32 row offsets.

Fields​

  • ​segment_ids (Ids):
  • ​row_offsets (Offsets):

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, MHAMask, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

apply_log2e_after_mask​

comptime apply_log2e_after_mask = False

check_mask_during_decoding​

comptime check_mask_during_decoding = True

device_type​

comptime device_type = SegmentMask[Ids, Offsets]

has_row_bias​

comptime has_row_bias = False

Whether add_row_bias adds this mask's additive bias, leaving visibility_mask the rest of mask.

mask_out_of_bound​

comptime mask_out_of_bound = True

mask_safe_out_of_bounds​

comptime mask_safe_out_of_bounds = True

MaskType​

comptime MaskType[*Ts: Fused & Input & LoadAccess] = SegmentMask[values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]

Parameters​

supports_decode_token_fold​

comptime supports_decode_token_fold = False

Whether AMD decode may fold several query tokens into one tile under this mask, and start its split-K key range at start_column: both status and start_column stay valid when asked about the span of query rows a sequence's decode step carries.

Methods​

make​

static def make[*Ts: Fused & Input & LoadAccess](operands: TensorTuple[*Ts.values]) -> SegmentMask[Ts.values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]

Builds the mask from the segment ids and row offsets.

Parameters:

Args:

Returns:

SegmentMask[Ts.values[0](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable), Ts.values[1](Input & LoadAccess & UnsafeFusedView).FusedViewType(Loadable & TrivialRegisterPassable)]: The mask.

get_type_name​

static def get_type_name() -> String

Returns:

String

name​

static def name() -> String

Returns:

String

mask​

def mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]

Returns:

SIMD[dtype, width]

status​

def status(self, seq_id: UInt32, tile_offset: Coord[Int, Int], tile_size: Coord[Int, Int]) -> TileMaskStatus

Returns:

TileMaskStatus

start_column​

def start_column[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32) -> UInt32

Returns:

UInt32

start_column_alignment​

static def start_column_alignment[BM: Int, BN: Int, page_size: Int]() -> Int

Returns:

Int

total_iters​

def total_iters[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32

Returns:

UInt32

count_nonfull_sets​

static def count_nonfull_sets(BM: Int, BN: Int) -> Int

Returns:

Int

last_masked_set_end​

def last_masked_set_end[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32

Returns:

UInt32

masked_set_ends​

def masked_set_ends[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> StaticTuple[UInt32, SegmentMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[UInt32, SegmentMask.count_nonfull_sets(BM, BN)]

nonfull_sets​

static def nonfull_sets[BM: Int, BN: Int]() -> StaticTuple[TileMaskStatus, SegmentMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[TileMaskStatus, SegmentMask.count_nonfull_sets(BM, BN)]

mask_strategies​

static def mask_strategies[BM: Int, BN: Int]() -> StaticTuple[MaskStrategy, SegmentMask.count_nonfull_sets(BM, BN)]

Returns:

StaticTuple[MaskStrategy, SegmentMask.count_nonfull_sets(BM, BN)]

mask_bits​

def mask_bits(self, seq_id: UInt32, score_row: Int32, col_start: Int32, num_keys: Int32) -> UInt32

Returns:

UInt32

sliding_window_size​

static def sliding_window_size() -> Int

Returns:

Int

Was this page helpful?