For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mha_gpu_naive
def mha_gpu_naive[output_type: DType, k_t: MHAOperand, v_t: MHAOperand, mask_t: MHAMask, //, ragged: Bool = False, sink: Bool = False, _use_valid_length: Bool = False, _is_cache_length_accurate: Bool = False](q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, mask_functor: mask_t, output: TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q.dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Launch the naive (two-pass BMM) GPU attention implementation.
Computes attention as two separate batched matrix multiplications using
temporary GMEM storage for the P = softmax(Q·Kᵀ / scale) intermediate.
This is slower than flash attention but supports any head depth and serves
as a correctness reference. Dispatches three sequential kernels:
_bmm0_bs (Q·Kᵀ + mask), softmax normalisation, and _bmm1_bs (P·V).
Parameters:
- output_type (
DType): Element type of the attention output. - k_t (
MHAOperand): Key operand type (dense or KV-cache). - v_t (
MHAOperand): Value operand type (dense or KV-cache). - mask_t (
MHAMask): Attention mask type. - ragged (
Bool):Truefor ragged-batch inputs. - sink (
Bool):Trueto enable attention-sink mode. - _use_valid_length (
Bool):Trueto read per-sequence valid lengths. - _is_cache_length_accurate (
Bool):Truewhen cache length is exact.
Args:
- q (
TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. - k (
k_t): Key operand. - v (
v_t): Value operand. - mask_functor (
mask_t): Mask instance. - output (
TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable output tensor. - valid_length (
OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Per-sequence valid lengths;Nonefor dense inputs. - scale (
Float32): Softmax temperature scale. - batch_size (
Int): Number of sequences in the batch. - max_prompt_len (
Int): Maximum query sequence length. - max_cache_size (
Int): Maximum key/value sequence length. - num_heads (
Int): Number of query heads. - depth (
Int): Attention head depth. - group (
Int): GQA group size. - ctx (
DeviceContextBase): GPU device context. - sink_weights (
OptionalReg[TileTensor[q.dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Sink-token weights for attention-sink mode.
def mha_gpu_naive[q_type: DType, k_type: DType, v_type: DType, output_type: DType, mask_type: DType, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, mask_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: TileTensor[mask_type, mask_tt_layout, Engine=mask.Engine, linear_idx_type=mask.linear_idx_type], output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, batch_size: Int, seq_len: Int, num_keys: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes reference GPU attention with a materialized mask.
Parameters:
- q_type (
DType): Element type of the query tensor. - k_type (
DType): Element type of the key tensor. - v_type (
DType): Element type of the value tensor. - output_type (
DType): Element type of the output tensor. - mask_type (
DType): Element type of the dense attention mask tensor. - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor. - k_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the key tensor. - v_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the value tensor. - mask_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the mask tensor. - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor. - sink (
Bool):Trueto enable attention-sink mode (defaults toFalse).
Args:
- q (
TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensorwith BSHD layout. - k (
TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type]): KeyTileTensor. - v (
TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type]): ValueTileTensor. - mask (
TileTensor[mask_type, mask_tt_layout, Engine=mask.Engine, linear_idx_type=mask.linear_idx_type]): Dense attention maskTileTensor. - output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable outputTileTensor. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - batch_size (
Int): Number of sequences in the batch. - seq_len (
Int): Maximum query sequence length in the batch. - num_keys (
Int): Maximum key/value sequence length. - num_heads (
Int): Number of query heads. - depth (
Int): Attention head depth (key/value dimension per head). - group (
Int): GQA group size (query heads per key/value head). - ctx (
DeviceContextBase): GPU device context for kernel dispatch. - sink_weights (
OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.
def mha_gpu_naive[q_type: DType, k_type: DType, v_type: DType, output_type: DType, MaskType: MHAMask, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: MaskType, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, batch_size: Int, seq_len: Int, num_keys: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes reference GPU attention with an MHAMask functor.
Parameters:
- q_type (
DType): Element type of the query tensor. - k_type (
DType): Element type of the key tensor. - v_type (
DType): Element type of the value tensor. - output_type (
DType): Element type of the output tensor. - MaskType (
MHAMask): Attention mask type implementingMHAMask. - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor. - k_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the key tensor. - v_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the value tensor. - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor. - sink (
Bool):Trueto enable attention-sink mode (defaults toFalse).
Args:
- q (
TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensorwith BSHD layout. - k (
TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type]): KeyTileTensor. - v (
TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type]): ValueTileTensor. - mask (
MaskType): Mask instance used to apply the attention mask. - output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable outputTileTensor. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - batch_size (
Int): Number of sequences in the batch. - seq_len (
Int): Maximum query sequence length in the batch. - num_keys (
Int): Maximum key/value sequence length. - num_heads (
Int): Number of query heads. - depth (
Int): Attention head depth (key/value dimension per head). - group (
Int): GQA group size (query heads per key/value head). - ctx (
DeviceContextBase): GPU device context for kernel dispatch. - sink_weights (
OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.
def mha_gpu_naive[q_type: DType, output_type: DType, k_t: MHAOperand, v_t: MHAOperand, mask_t: MHAMask, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, ragged: Bool = False, sink: Bool = False, _use_valid_length: Bool = False, _is_cache_length_accurate: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, mask_functor: mask_t, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes reference GPU attention with MHAOperand key/value.
Parameters:
- q_type (
DType): Element type of the query tensor. - output_type (
DType): Element type of the attention output. - k_t (
MHAOperand): Key operand type (dense or KV-cache). - v_t (
MHAOperand): Value operand type (dense or KV-cache). - mask_t (
MHAMask): Attention mask type. - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor. - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor. - valid_length_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the valid-length tensor. - ragged (
Bool):Truefor ragged-batch inputs. - sink (
Bool):Trueto enable attention-sink mode. - _use_valid_length (
Bool):Trueto honour per-sequence valid lengths. - _is_cache_length_accurate (
Bool):Truewhen the cache length already excludes the current prompt.
Args:
- q (
TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensorwith BSHD layout. - k (
k_t): Key operand. - v (
v_t): Value operand. - mask_functor (
mask_t): Mask instance used to apply the attention mask. - output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable outputTileTensor. - valid_length (
TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-sequence valid lengths as aTileTensor. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - batch_size (
Int): Number of sequences in the batch. - max_prompt_len (
Int): Maximum query sequence length in the batch. - max_cache_size (
Int): Maximum key/value sequence length. - num_heads (
Int): Number of query heads. - depth (
Int): Attention head depth (key/value dimension per head). - group (
Int): GQA group size (query heads per key/value head). - ctx (
DeviceContextBase): GPU device context for kernel dispatch. - sink_weights (
OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.
def mha_gpu_naive[q_type: DType, output_type: DType, cache_t: KVCacheT, mask_t: MHAMask, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, ragged: Bool = False, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask_functor: mask_t, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes reference GPU attention with KV-cache key/value.
Wraps the cache operands and delegates to the native MHAOperand overload.
Parameters:
- q_type (
DType): Element type of the query tensor. - output_type (
DType): Element type of the attention output. - cache_t (
KVCacheT): KV-cache type backing the key and value tensors. - mask_t (
MHAMask): Attention mask type. - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor. - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor. - valid_length_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the valid-length tensor. - ragged (
Bool):Truefor ragged-batch inputs. - sink (
Bool):Trueto enable attention-sink mode.
Args:
- q (
TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensorwith BSHD layout. - k (
cache_t): Key operand backed by a KV cache. - v (
cache_t): Value operand backed by a KV cache. - mask_functor (
mask_t): Mask instance used to apply the attention mask. - output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable outputTileTensor. - valid_length (
TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-sequence valid lengths as aTileTensor. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - batch_size (
Int): Number of sequences in the batch. - max_prompt_len (
Int): Maximum query sequence length in the batch. - max_cache_size (
Int): Maximum key/value sequence length. - num_heads (
Int): Number of query heads. - depth (
Int): Attention head depth (key/value dimension per head). - group (
Int): GQA group size (query heads per key/value head). - ctx (
DeviceContextBase): GPU device context for kernel dispatch. - sink_weights (
OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.