For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
fa4_splitk_combine
FA4 SM100 traditional (workspace / unfused) split-K combine kernel.
Merges the P per-partition partial outputs the FA4 1Q attention kernel wrote
to a global workspace into the final attention output, using fused Log-Sum-Exp
(LSE) weights for numerical stability.
Workspace layout (produced by the attention kernel's do_partition egress):
o_partial:[P, num_rows_q, num_q_heads, ov_depth](intermediate dtype), each partition holds its LOCALLY-normalizedO_p / l_p.lse_partial:[P, num_rows_q, num_q_heads](f32), fused per-row LSE in log2 domain (lse_p = log2(l_p) + m_p).
Output layout (matches the non-split ragged store): [num_rows_q, num_q_heads, ov_depth].
Per output row-head (token, q_head) (flattened to rh = token*num_q_heads + q_head), reducing over partitions p:
m* = max_p lse_pglobal_lse = log2(Sum_p exp2(lse_p - m*)) + m*scale_p = exp2(lse_p - global_lse)(folds the1 / denomnormalization in:exp2(lse - log2(denom) - m) == exp2(lse - m) / denom)O[d] = Sum_p scale_p * o_partial[p, rh, d]
Everything stays in the log2 (base-2) domain to match the FA4 softmax
(exp2/log2); crossing into natural-exp would silently corrupt the result.
Perf structure (adapted from mla_decode_combine.mojo's
mla_combine_kernel / mla_combine_kernel_split_parallel)
_COMBINE_WARPS warps per output row-head, each folding every
_COMBINE_WARPS-th partition, summed through shared memory at the end.
Within a warp ov_depth is split across lanes via a vectorized vec_size /
elems_per_thread partition (128-bit loads where ov_depth and the
intermediate dtype allow it). The LSE reduction (row max, log-sum-exp,
per-partition scale_p) is computed ONCE into per-lane registers -- not the
3x redundant global-memory re-read (once for the max, once for the
denominator, once per depth-element inside the accumulation loop) the original
correctness-first version did. The weighted accumulation loop broadcasts each
partition's scale from its owning lane via warp.shuffle_idx and prefetches
the warp's NEXT partition's o_partial vector into registers while the
CURRENT partition's contribution is folded into the running sum (software
pipelining), so the load latency of the next partition overlaps the FMA work
of the current one.
Two kernel variants share this body, selected by the comptime P_STATIC
parameter:
P_STATIC > 0:num_partitionslanded on a rung of the sharedsplitk_p_ladderthatdispatch.mojo's_bucket_wssnaps productionPonto. Both the LSE and accumulation loops arecomptime for-unrolled over the exactP_STATIC, which also keeps the per-lanelocal_lsearray register-indexed (divmod(p, WARP_SIZE)folds at comptime).P_STATIC == 0: an off-rungP. Reachable two ways -- a test force-knob, and_bucket_wscapping its bucketed value atws_p_ceiling, which is not itself a rung (B200: 37 through the sweep, then decaying). The same vectorized- prefetched accumulation shape runs over a RUNTIME
num_partitionsbound, paying a dynamically-indexedlocal_lse. The array is sized at a fixed comptime ceiling (P_MAX, covering every current-generationsm_count) so this fallback compiles exactly once, independent of thePit is handed.
- prefetched accumulation shape runs over a RUNTIME
Target hardware family: NVIDIA SM100 (B200 / B300).
PDL: this kernel is a Programmatic Dependent Launch consumer of the
attention producer and a producer for the o-proj matmul behind it. It calls
wait_on_dependent_grids() (all threads of its block, before the rh >= RH
guard) and launches with pdl_launch_attributes(MHA_PDL_LEVEL). It also
releases its own dependents on entry, before that wait.
Releasing early is safe because griddepcontrol.wait blocks until the whole
prerequisite grid has COMPLETED and its memory is visible, not until it has
triggered (PTX ISA, griddepcontrol). A trigger only decides when the
successor is scheduled, so the o-proj GEMM may become resident, and prefetch
its weights, while this kernel is still running, yet cannot read output
before this grid has finished writing it.
The attention producer does NOT release this kernel from its prologue for the
do_partition config (kernel.mojo), so it only triggers implicitly, when each
of its CTAs exits. The combine therefore cannot start before the last attention
CTA has exited, and none of attention's compute overlaps it. That is a measured
choice, not a correctness requirement: a prologue release is as safe as the one
in this kernel, but it made small-batch decode slower (see kernel.mojo). The
PROGRAMMATIC_STREAM_SERIALIZATION attribute is still needed: it lets the
combine's launch and prologue overlap the attention grid's completion drain
after its last CTA exits, instead of starting only after that drain (an earlier
measurement, not repeated by this change, gave 1.06-1.13x on P=148
combine-heavy shapes). With -D MHA_PDL=false all of this is comptime-pruned
back to plain stream ordering.
comptime values
P_MAX
comptime P_MAX = get_defined_int[StringSpan("FA4_COMBINE_P_MAX"), Int(160)]()
Functions
-
fa4_splitk_combine: Launches the workspace split-K combine overnum_rows_q * num_q_headsoutput row-heads,_COMBINE_WARPSwarps each.