For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
workspace
The scratch workspace a fused op's host code stages its own launches through.
A fused graph op sometimes has to hand one kernel's output to a second kernel it
runs itself, with no graph tensor in between: the chain arm of
mo.composite.distributed.allreduce_rms_norm_exit_concat writes the reduced
sum and then normalizes it, and that sum is not a value the graph asked for. The signal buffer's per-rank scratch is not available for that: the
collectives stage through it themselves, so a region a kernel writes as its
output would alias one they read.
The workspace is one buffer per device, created on first use, grown geometrically past the largest request and then kept for the process's life. It is published through the compiler runtime's global cache, so a pointer a captured graph baked in stays valid: a device graph capture records the launches, and nothing here moves or frees what they were handed.
The largest caller today bounds the workspace: the exit collective's chain
arm stages a [rows, cols] sum whose row count is capped by the
context-encoding batch budget (max_batch_input_tokens, 8192 by default),
which at Kimi K3's 3584-wide float32 latent is 8192 x 3584 x 4 bytes
= 112 MiB per device. A workspace that had to grow leaves its old buffer
mapped (see staging_workspace), so a device can hold about twice that
counting the live workspace and the old mappings growth left behind.
A caller owns the workspace for one of its own launch sequences, on one device's stream at a time. That is what makes one region enough: the two launches that share it are enqueued back to back on the device's stream by the same invocation, so on the device the writer has retired before the reader starts, and no other caller of this module can be between them, because a second invocation enqueues its own pair after this one's on that same stream. A caller that ran two sequences concurrently on two streams of the same device, or that lets its reader outlive the pair, would alias another's rows and needs a region of its own: nothing here detects that.
Structsβ
- β
StagingSlot: What the global cache holds for one device: its workspace and its size.
Functionsβ
- β
staging_workspace: Returns this device's staging workspace, holding at leastelements.