IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

workspace

The scratch workspace a fused op's host code stages its own launches through.

A fused graph op sometimes has to hand one kernel's output to a second kernel it runs itself, with no graph tensor in between: the chain arm of mo.composite.distributed.allreduce_rms_norm_exit_concat writes the reduced sum and then normalizes it, and that sum is not a value the graph asked for. The signal buffer's per-rank scratch is not available for that: the collectives stage through it themselves, so a region a kernel writes as its output would alias one they read.

The workspace is one buffer per device, created on first use, grown geometrically past the largest request and then kept for the process's life. It is published through the compiler runtime's global cache, so a pointer a captured graph baked in stays valid: a device graph capture records the launches, and nothing here moves or frees what they were handed.

The largest caller today bounds the workspace: the exit collective's chain arm stages a [rows, cols] sum whose row count is capped by the context-encoding batch budget (max_batch_input_tokens, 8192 by default), which at Kimi K3's 3584-wide float32 latent is 8192 x 3584 x 4 bytes = 112 MiB per device. A workspace that had to grow leaves its old buffer mapped (see staging_workspace), so a device can hold about twice that counting the live workspace and the old mappings growth left behind.

A caller owns the workspace for one of its own launch sequences, on one device's stream at a time. That is what makes one region enough: the two launches that share it are enqueued back to back on the device's stream by the same invocation, so on the device the writer has retired before the reader starts, and no other caller of this module can be between them, because a second invocation enqueues its own pair after this one's on that same stream. A caller that ran two sequences concurrently on two streams of the same device, or that lets its reader outlive the pair, would alias another's rows and needs a region of its own: nothing here detects that.

Structs​

  • ​StagingSlot: What the global cache holds for one device: its workspace and its size.

Functions​

Was this page helpful?