IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_gumbel_argmax_from_residual

struct Struct_gumbel_argmax_from_residual

Registers the sampler.gumbel_argmax_from_residual graph op.

Draws one token per row proportionally to max(p - q, 0), the rejection sampler's residual, by Gumbel-max over its ln. Both the subtract and the clamp happen in registers inside the sampling pass, so the residual is never written to or read from HBM: the alternative materializes the proposal mass and the residual, two [rows, vocab] round-trips.

fallback carries one drafted token id per row; a non-negative entry marks a row with no proposal distribution, and that row subtracts a one-hot at the token instead of q. The value is bit-identical to subtracting a materialized one-hot row, and it is what the caller already has on hand, so passing the flag costs nothing.

Arity is fixed, so this is a separate registration rather than a mode on sampler.gumbel_argmax_from_probs.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[_trace_name: StaticString](out_tokens: T, seed: T, probs: T, sub: T, fallback: T, ctx: DeviceContext)

Was this page helpful?