IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mha_gpu_naive

def mha_gpu_naive[output_type: DType, k_t: MHAOperand, v_t: MHAOperand, mask_t: MHAMask, //, ragged: Bool = False, sink: Bool = False, _use_valid_length: Bool = False, _is_cache_length_accurate: Bool = False](q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, mask_functor: mask_t, output: TileTensor[output_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q.dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Launch the naive (two-pass BMM) GPU attention implementation.

Computes attention as two separate batched matrix multiplications using temporary GMEM storage for the P = softmax(Q·Kᵀ / scale) intermediate. This is slower than flash attention but supports any head depth and serves as a correctness reference. Dispatches three sequential kernels: _bmm0_bs (Q·Kᵀ + mask), softmax normalisation, and _bmm1_bs (P·V).

Parameters:

  • ​output_type (DType): Element type of the attention output.
  • ​k_t (MHAOperand): Key operand type (dense or KV-cache).
  • ​v_t (MHAOperand): Value operand type (dense or KV-cache).
  • ​mask_t (MHAMask): Attention mask type.
  • ​ragged (Bool): True for ragged-batch inputs.
  • ​sink (Bool): True to enable attention-sink mode.
  • ​_use_valid_length (Bool): True to read per-sequence valid lengths.
  • ​_is_cache_length_accurate (Bool): True when cache length is exact.

Args:

def mha_gpu_naive[q_type: DType, k_type: DType, v_type: DType, output_type: DType, mask_type: DType, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, mask_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: TileTensor[mask_type, mask_tt_layout, Engine=mask.Engine, linear_idx_type=mask.linear_idx_type], output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, batch_size: Int, seq_len: Int, num_keys: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Computes reference GPU attention with a materialized mask.

Parameters:

  • ​q_type (DType): Element type of the query tensor.
  • ​k_type (DType): Element type of the key tensor.
  • ​v_type (DType): Element type of the value tensor.
  • ​output_type (DType): Element type of the output tensor.
  • ​mask_type (DType): Element type of the dense attention mask tensor.
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor.
  • ​k_tt_layout (TensorLayout): Compile-time TensorLayout of the key tensor.
  • ​v_tt_layout (TensorLayout): Compile-time TensorLayout of the value tensor.
  • ​mask_tt_layout (TensorLayout): Compile-time TensorLayout of the mask tensor.
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor.
  • ​sink (Bool): True to enable attention-sink mode (defaults to False).

Args:

def mha_gpu_naive[q_type: DType, k_type: DType, v_type: DType, output_type: DType, MaskType: MHAMask, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: MaskType, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, batch_size: Int, seq_len: Int, num_keys: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Computes reference GPU attention with an MHAMask functor.

Parameters:

  • ​q_type (DType): Element type of the query tensor.
  • ​k_type (DType): Element type of the key tensor.
  • ​v_type (DType): Element type of the value tensor.
  • ​output_type (DType): Element type of the output tensor.
  • ​MaskType (MHAMask): Attention mask type implementing MHAMask.
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor.
  • ​k_tt_layout (TensorLayout): Compile-time TensorLayout of the key tensor.
  • ​v_tt_layout (TensorLayout): Compile-time TensorLayout of the value tensor.
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor.
  • ​sink (Bool): True to enable attention-sink mode (defaults to False).

Args:

def mha_gpu_naive[q_type: DType, output_type: DType, k_t: MHAOperand, v_t: MHAOperand, mask_t: MHAMask, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, ragged: Bool = False, sink: Bool = False, _use_valid_length: Bool = False, _is_cache_length_accurate: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, v: v_t, mask_functor: mask_t, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Computes reference GPU attention with MHAOperand key/value.

Parameters:

  • ​q_type (DType): Element type of the query tensor.
  • ​output_type (DType): Element type of the attention output.
  • ​k_t (MHAOperand): Key operand type (dense or KV-cache).
  • ​v_t (MHAOperand): Value operand type (dense or KV-cache).
  • ​mask_t (MHAMask): Attention mask type.
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor.
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor.
  • ​valid_length_tt_layout (TensorLayout): Compile-time TensorLayout of the valid-length tensor.
  • ​ragged (Bool): True for ragged-batch inputs.
  • ​sink (Bool): True to enable attention-sink mode.
  • ​_use_valid_length (Bool): True to honour per-sequence valid lengths.
  • ​_is_cache_length_accurate (Bool): True when the cache length already excludes the current prompt.

Args:

def mha_gpu_naive[q_type: DType, output_type: DType, cache_t: KVCacheT, mask_t: MHAMask, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, ragged: Bool = False, sink: Bool = False](q: TileTensor[q_type, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask_functor: mask_t, output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, batch_size: Int, max_prompt_len: Int, max_cache_size: Int, num_heads: Int, depth: Int, group: Int, ctx: DeviceContextBase, sink_weights: OptionalReg[TileTensor[q_type, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Computes reference GPU attention with KV-cache key/value.

Wraps the cache operands and delegates to the native MHAOperand overload.

Parameters:

  • ​q_type (DType): Element type of the query tensor.
  • ​output_type (DType): Element type of the attention output.
  • ​cache_t (KVCacheT): KV-cache type backing the key and value tensors.
  • ​mask_t (MHAMask): Attention mask type.
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor.
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor.
  • ​valid_length_tt_layout (TensorLayout): Compile-time TensorLayout of the valid-length tensor.
  • ​ragged (Bool): True for ragged-batch inputs.
  • ​sink (Bool): True to enable attention-sink mode.

Args:

Was this page helpful?