For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flash_attention_split_kv
def flash_attention_split_kv[dtype: DType, rank: Int, mask_rank: Int, InputKFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputVFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputKCacheFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputVCacheFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputMaskFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], //](input_k_fn: InputKFnType, input_v_fn: InputVFnType, input_k_cache_fn: InputKCacheFnType, input_v_cache_fn: InputVCacheFnType, input_mask_fn: InputMaskFnType, q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k_shape: IndexList[rank], v_shape: IndexList[rank], k_cache_shape: IndexList[(rank + Int(1))], v_cache_shape: IndexList[(rank + Int(1))], mask_shape: IndexList[mask_rank], output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, ctx: Optional[DeviceContext[CompilationTarget()]] = None)
Variant of flash attention that takes the previous KV cache input_{k,v}_cache_fn and the current KV tensors input_k_fn and input_v_fn as separate arguments.
This works around the fact that fusion can't currently look through concat.
So this kernel does an in-place concat fusion by changing the input lambdas
input_{k,v}_cache_fn_wrapper to take previous sequence KV elements from
the KV cache, and current KV elements from tensors k and v.
Parameters:
- dtype (
DType): The element type of the query, key, value, and output tensors (inferred). - rank (
Int): The number of dimensions in the query, key, value, and output tensors, either 3 or 4 (inferred). - mask_rank (
Int): The number of dimensions in the attention mask tensor (inferred). - InputKFnType (
def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type ofinput_k_fn(inferred). - InputVFnType (
def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type ofinput_v_fn(inferred). - InputKCacheFnType (
def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type ofinput_k_cache_fn(inferred). - InputVCacheFnType (
def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type ofinput_v_cache_fn(inferred). - InputMaskFnType (
def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type ofinput_mask_fn(inferred).
Args:
- input_k_fn (
InputKFnType): Function loading aSIMDvector of current key elements at a givenCoordindex. - input_v_fn (
InputVFnType): Function loading aSIMDvector of current value elements at a givenCoordindex. - input_k_cache_fn (
InputKCacheFnType): Function loading aSIMDvector of cached key elements at a givenCoordindex. - input_v_cache_fn (
InputVCacheFnType): Function loading aSIMDvector of cached value elements at a givenCoordindex. - input_mask_fn (
InputMaskFnType): Function loading aSIMDvector of additive mask values at a givenCoordindex. - q (
TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor in BSHD layout. - k_shape (
IndexList[rank]): Shape of the current key tensor in BSHD layout. - v_shape (
IndexList[rank]): Shape of the current value tensor in BSHD layout. - k_cache_shape (
IndexList[(rank + Int(1))]): Shape of the cached key tensor with one extra leading dimension. - v_cache_shape (
IndexList[(rank + Int(1))]): Shape of the cached value tensor with one extra leading dimension. - mask_shape (
IndexList[mask_rank]): Shape of the attention mask tensor. - output (
TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor to write the attention results into. - scale (
Float32): Scaling factor applied to the query-key dot products. - ctx (
Optional[DeviceContext[CompilationTarget()]]): Optional device context for controlling parallelism (defaults toNone).