For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
accept_pair
def accept_pair(probs: TileTensor[.float32, Engine=probs.Engine, address_space=probs.address_space, linear_idx_type=probs.linear_idx_type], dist: TileTensor[.float32, Engine=dist.Engine, address_space=dist.address_space, linear_idx_type=dist.linear_idx_type], tokens: TileTensor[.int64, Engine=tokens.Engine, address_space=tokens.address_space, linear_idx_type=tokens.linear_idx_type], coins: TileTensor[.float32, Engine=coins.Engine, address_space=coins.address_space, linear_idx_type=coins.linear_idx_type], rejected: TileTensor[.bool, Engine=rejected.Engine, address_space=rejected.address_space, linear_idx_type=rejected.linear_idx_type], fallback: TileTensor[.int64, Engine=fallback.Engine, address_space=fallback.address_space, linear_idx_type=fallback.linear_idx_type], ctx: DeviceContext)
Writes the sampled-draft verdict's per-position pair, in one launch.
For each (request, draft position) the accept test needs two numbers
at one vocabulary index -- the target's masked probability there and the
proposal's probability there -- and the residual needs to know whether the
row carried a proposal at all. Reaching them with graph ops costs a
[rows, steps, 3] index tensor to gather through, then a compare, a
select, a multiply and a compare: one launch per member over a mask of a
few dozen entries.
q == 0 means the scheduler had no distribution for the row, so the
proposal mass is taken as one (the test degrades to typical acceptance)
and the residual subtracts a one-hot at the drafted token instead of the
proposal. Both the substitution and that per-row choice happen here, so
the caller reads only the two decisions.
Args:
- probs (
TileTensor[.float32, Engine=probs.Engine, address_space=probs.address_space, linear_idx_type=probs.linear_idx_type]): The masked target distribution,[rows, steps, vocab]. - dist (
TileTensor[.float32, Engine=dist.Engine, address_space=dist.address_space, linear_idx_type=dist.linear_idx_type]): The proposal distribution,[rows, steps, vocab]. - tokens (
TileTensor[.int64, Engine=tokens.Engine, address_space=tokens.address_space, linear_idx_type=tokens.linear_idx_type]): The drafted token per position,[rows, steps]. - coins (
TileTensor[.float32, Engine=coins.Engine, address_space=coins.address_space, linear_idx_type=coins.linear_idx_type]): The accept coin per position,[rows, steps]. - rejected (
TileTensor[.bool, Engine=rejected.Engine, address_space=rejected.address_space, linear_idx_type=rejected.linear_idx_type]): Out, the rejection mask[rows, steps]. - fallback (
TileTensor[.int64, Engine=fallback.Engine, address_space=fallback.address_space, linear_idx_type=fallback.linear_idx_type]): Out,-1where the row had a proposal, else its token. - ctx (
DeviceContext): The device context.
Raises:
Error: If a rank or a vocabulary size disagrees with probs.