For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
AndMask
struct AndMask[T: MHAMask, S: MHAMask, //, lhs: T, rhs: S]
Mask that's the AND of two masks. If both masks mask off an element, the element is masked off.
Implemented traits
AnyType,
Copyable,
Defaultable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
MHAMask,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
apply_log2e_after_mask
comptime apply_log2e_after_mask = T.apply_log2e_after_mask or S.apply_log2e_after_mask
check_mask_during_decoding
comptime check_mask_during_decoding = T.check_mask_during_decoding and S.check_mask_during_decoding
device_type
comptime device_type = AndMask[lhs, rhs]
has_row_bias
comptime has_row_bias = False
Whether add_row_bias adds this mask's additive bias, leaving visibility_mask the rest of mask.
mask_out_of_bound
comptime mask_out_of_bound = T.mask_out_of_bound or S.mask_out_of_bound
mask_safe_out_of_bounds
comptime mask_safe_out_of_bounds = T.mask_safe_out_of_bounds and S.mask_safe_out_of_bounds
MaskType
comptime MaskType[*Ts: Fused & Input & LoadAccess] = AndMask[lhs, rhs]
Parameters
- *Ts (
Fused&Input&LoadAccess):
supports_decode_token_fold
comptime supports_decode_token_fold = False
Whether AMD decode may fold several query tokens into one tile under this mask, and start its split-K key range at start_column: both status and start_column stay valid when asked about the span of query rows a sequence's decode step carries.
Methods
get_type_name
name
mask
def mask[dtype: DType, width: SIMDLength, //](self, coord: Coord[Int, Int, Int, Int], score_vec: SIMD[dtype, width]) -> SIMD[dtype, width]
Returns:
status
def status(self, seq_id: UInt32, tile_offset: Coord[Int, Int], tile_size: Coord[Int, Int]) -> TileMaskStatus
Returns:
start_column
def start_column[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32) -> UInt32
Returns:
start_column_alignment
total_iters
def total_iters[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32
Returns:
count_nonfull_sets
last_masked_set_end
def last_masked_set_end[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> UInt32
Returns:
masked_set_ends
def masked_set_ends[BM: Int, BN: Int, page_size: Int](self, seq_id: UInt32, row: UInt32, num_cols: UInt32) -> StaticTuple[UInt32, AndMask.count_nonfull_sets(BM, BN)]
Returns:
nonfull_sets
static def nonfull_sets[BM: Int, BN: Int]() -> StaticTuple[TileMaskStatus, AndMask.count_nonfull_sets(BM, BN)]
Returns:
StaticTuple[TileMaskStatus, AndMask.count_nonfull_sets(BM, BN)]
mask_strategies
static def mask_strategies[BM: Int, BN: Int]() -> StaticTuple[MaskStrategy, AndMask.count_nonfull_sets(BM, BN)]
Returns:
StaticTuple[MaskStrategy, AndMask.count_nonfull_sets(BM, BN)]
mask_bits
def mask_bits(self, seq_id: UInt32, score_row: Int32, col_start: Int32, num_keys: Int32) -> UInt32
Returns: