IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

SplitKPartition

struct SplitKPartition[dtype: DType, OPartialType: OptionalPointer = NullPointer[.float32]]

A multi-partition split-K scheme for MHA decoding over long sequences.

Divides the key sequence across num_partitions CTAs. Each CTA writes its partial softmax numerator/denominator to the buffer pointed to by ptr, and a separate reduction kernel merges the results. Over-launches up to max_num_partitions CTAs so the grid shape is stable across varying key lengths (enabling CUDA graph capture).

Parameters​

  • ​dtype (DType): The accumulator element type used for the partial statistics buffer and the final output.
  • ​OPartialType (OptionalPointer): An f32 buffer for each partition's O, for a kernel that stores its partials unrounded; null when the kernel stores them in its output type.

Fields​

  • ​ptr (MutPointer[Scalar[SplitKPartition[dtype, OPartialType].accum_dtype], MutAnyOrigin]):
  • ​num_partitions_value (UInt32):
  • ​max_num_partitions_value (UInt32):
  • ​o_partial (OPartialType):

Implemented traits​

AnyType, Copyable, Deinitable, ImplicitlyCopyable, MHAPartitionScheme, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

accum_dtype​

comptime accum_dtype = dtype

do_partition​

comptime do_partition = True

LSEPointerType​

comptime LSEPointerType = NonNullPointer[SplitKPartition[dtype, OPartialType].accum_dtype]

OPartialPointerType​

comptime OPartialPointerType = OPartialType

Methods​

__init__​

def __init__(ptr: MutPointer[Scalar[Self.accum_dtype], MutAnyOrigin], num_partitions_value: UInt32, max_num_partitions_value: UInt32, o_partial: OPartialType = null_pointer[OPartialType]()) -> Self

num_partitions​

def num_partitions(self) -> UInt32

Returns:

UInt32

max_num_partitions​

def max_num_partitions(self) -> UInt32

Returns:

UInt32

lse_pointer​

def lse_pointer(self) -> Self.LSEPointerType

Returns:

Self.LSEPointerType

o_partial_pointer​

def o_partial_pointer(self) -> Self.OPartialPointerType

Returns:

Self.OPartialPointerType

Was this page helpful?