For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
SplitKPartition
struct SplitKPartition[dtype: DType, OPartialType: OptionalPointer = NullPointer[.float32]]
A multi-partition split-K scheme for MHA decoding over long sequences.
Divides the key sequence across num_partitions CTAs. Each CTA writes
its partial softmax numerator/denominator to the buffer pointed to by
ptr, and a separate reduction kernel merges the results. Over-launches
up to max_num_partitions CTAs so the grid shape is stable across
varying key lengths (enabling CUDA graph capture).
Parameters
- dtype (
DType): The accumulator element type used for the partial statistics buffer and the final output. - OPartialType (
OptionalPointer): An f32 buffer for each partition's O, for a kernel that stores its partials unrounded; null when the kernel stores them in its output type.
Fields
- ptr (
MutPointer[Scalar[SplitKPartition[dtype, OPartialType].accum_dtype], MutAnyOrigin]): - num_partitions_value (
UInt32): - max_num_partitions_value (
UInt32): - o_partial (
OPartialType):
Implemented traits
AnyType,
Copyable,
Deinitable,
ImplicitlyCopyable,
MHAPartitionScheme,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
accum_dtype
comptime accum_dtype = dtype
do_partition
comptime do_partition = True
LSEPointerType
comptime LSEPointerType = NonNullPointer[SplitKPartition[dtype, OPartialType].accum_dtype]
OPartialPointerType
comptime OPartialPointerType = OPartialType
Methods
__init__
def __init__(ptr: MutPointer[Scalar[Self.accum_dtype], MutAnyOrigin], num_partitions_value: UInt32, max_num_partitions_value: UInt32, o_partial: OPartialType = null_pointer[OPartialType]()) -> Self
num_partitions
max_num_partitions
lse_pointer
def lse_pointer(self) -> Self.LSEPointerType
Returns:
Self.LSEPointerType
o_partial_pointer
def o_partial_pointer(self) -> Self.OPartialPointerType
Returns:
Self.OPartialPointerType