For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
TextContext
TextContext
class max.pipelines.context.TextContext(*, max_length, tokens, request_id=<factory>, eos_tracker=<factory>, vocab_size=None, log_probabilities=0, log_probabilities_echo=False, ignore_eos=False, json_schema=None, grammar=None, grammar_state=<factory>, sampling_params=<factory>, model_name='', _matcher=None, status=GenerationStatus.ACTIVE, _log_probabilities_data=<factory>, _is_initial_prompt=True, _is_padding_ctx=False, _draft_offset=0, _spec_decoding_state=None, in_reasoning_phase=False, target_endpoint=None, dkv_cache_hint=None, cache_salt=None, cached_prefix_length=None, cached_prefix_external_length=0, _cache_metrics_emitted=False, trace_carrier=None)
Bases: object
A base class for model context, specifically for Text model variants.
This class manages the state and processing of text generation, including token management, caching, and generation parameters.
-
Parameters:
-
- max_length (int) – Maximum allowed length of the generated sequence
- tokens (TokenBuffer) – NumPy array containing the token IDs
- request_id (RequestID) – A unique identifier for this sequence.
- eos_tracker (EOSTracker) – holds EOS config and performs checks for EOS conditions
- vocab_size (int | None) – Optional vocabulary size for validating generated token IDs
- log_probabilities (int) – Whether to return token log probabilities
- log_probabilities_echo (bool) – Whether to return log probabilities for prompt tokens
- ignore_eos (bool) – Whether to ignore end of sequence tokens and continue generating
- json_schema (str | None) – Optional JSON schema for structured output
- grammar (str | None)
- grammar_state (GrammarEnforcementState)
- sampling_params (SamplingParams) – Parameters controlling the token sampling strategy
- model_name (str)
- _matcher (Any | None)
- status (GenerationStatus)
- _log_probabilities_data (dict[int, LogProbabilities]) – Token log probabilities data
- _is_initial_prompt (bool) – Whether this is the initial prompt encoding
- _is_padding_ctx (bool) – Whether this context is a DP batch padding context
- _draft_offset (int) – Offset for draft decoding
- _spec_decoding_state (SpecDecodingState | None) – Optional per-request speculative decoding state
- in_reasoning_phase (bool)
- target_endpoint (str | None) – Optional target endpoint identifier for routing requests
- dkv_cache_hint (bytes | None)
- cache_salt (str | None)
- cached_prefix_length (int | None)
- cached_prefix_external_length (int)
- _cache_metrics_emitted (bool)
- trace_carrier (dict[str, str] | None)
advance_fsm()
advance_fsm(token)
Advance the FSM matcher state by one token.
This method:
- Updates enforcement state based on tool call boundaries (if conditional)
- Advances the FSM if grammar is currently enforced
It does NOT modify the token buffer. Use advance_token_buffer()
separately if token buffer advancement is needed, or use update()
for the common case of advancing both together.
Matcher rejection is not expected at this point (assuming the bitmask was applied correctly). But if the matcher does reject a token, enforcement is disabled for the rest of the request. Continuing to enforce against a desynced matcher would produce schema-shaped nonsense (every downstream bitmask would be filtered against a stale grammar position with no relation to what was emitted). Instead we let the request finish unconstrained.
-
Parameters:
-
token (int) – The token to consume in the FSM.
-
Returns:
-
True if the token was handled — either consumed by the FSM, recognized as a state-transition delimiter (e.g. a thinking-end token), or skipped because enforcement is inactive. False only when no matcher is present.
-
Return type:
advance_token_buffer()
advance_token_buffer(new_token, log_probabilities=None)
Advance the token buffer without touching FSM state.
This method handles token buffer mutations including:
- Chunked prefill advancement
- Log probability storage
- Token buffer advancement
- EOS/max-length status updates
It does NOT advance the FSM matcher. Use advance_fsm() separately
if FSM advancement is needed, or use update() for the common case
of advancing both together.
-
Parameters:
-
- new_token (int) – The token to append to the buffer.
- log_probabilities (LogProbabilities | None) – Optional log probabilities for this token.
-
Return type:
-
None
apply_processing_offset()
apply_processing_offset(offset)
Applies a processing offset to the token buffer.
-
Parameters:
-
offset (int)
-
Return type:
-
None
cache_salt
Optional per-request salt that isolates this prompt’s prefix-cache entries from other requests sharing the same tokens.
Combined with kv_cache_hash_seed via XOR to seed the block hash.
Works under any kv_cache_hash_algo: a cryptographic guarantee
under sha256/sha256_64, best-effort under ahash64. Capped
at 512 chars at the OpenAI schema layer.
cached_prefix_external_length
cached_prefix_external_length: int = 0
How many of cached_prefix_length tokens the KV connector served.
Set alongside cached_prefix_length on first admission. The remainder
came from the on-device prefix cache, which is what lets the scheduler tag
its hit counter per tier without a second lookup. Always 0 when no
connector is configured.
Not split into the connector’s own tiers: KVConnector.load() reports
only a loaded-block count, so the host/disk (dKV G1/G2) breakdown does not
cross that boundary and these tokens are reported as external rather
than guessed at.
cached_prefix_length
Number of prompt tokens served from the KV prefix cache.
Set by the block manager when a request is admitted to a CE batch
(0 if no matching prefix). BatchMetrics.create
consumes the value to emit a per-request cache hit rate observation, and
uses _cache_metrics_emitted to guard against re-emitting on
chunked-prefill follow-up calls.
compute_num_available_steps()
compute_num_available_steps(max_seq_len)
Computes the maximum number of steps without exceeding max_seq_len.
Takes the current context length into account.
dkv_cache_hint
The Orchestrator’s dkv_cache_hint for this request, as JSON bytes.
Opaque here. The serving layer only carries it across the API-server to
model-worker boundary and hands it to the KV connector’s load; the dKV
connector parses it in Rust to route each block to the peer that holds it.
None when the request carried no hint.
eos_tracker
eos_tracker: EOSTracker
get_min_token_logit_mask()
get_min_token_logit_mask(num_steps)
Returns per-step masks for logits that should be masked (e.g. EOS during min_tokens).
This is primarily used for the min_tokens setting, where we mask
EOS tokens in the logits to avoid generating them before we reach
min_tokens.
grammar
Grammar for constrained decoding (e.g., regex grammar).
When set, this takes precedence over json_schema. Used for model-specific
constrained decoding formats like Kimi’s tool call grammar.
grammar_enforced
property grammar_enforced: bool
Whether grammar is currently being enforced.
grammar_state
grammar_state: GrammarEnforcementState
Grammar enforcement state for constrained decoding.
ignore_eos
ignore_eos: bool = False
in_reasoning_phase
in_reasoning_phase: bool = False
Whether the latest committed tokens are inside a <think>...</think>
block. Toggled host-side after each commit when a reasoning parser is
configured.
is_done
property is_done: bool
Whether text generation has finished.
is_initial_prompt
property is_initial_prompt: bool
Returns true if the context has not been updated with tokens.
json_schema
last_realized_token
property last_realized_token: int
The most recent realized (non-placeholder) token in the buffer.
Overlap decode appends a single FUTURE_TOKEN placeholder while a
forward is in flight, so tokens[-1] may be unrealized. Readers that
need a real token must use this instead.
log_probabilities
log_probabilities: int = 0
log_probabilities_echo
log_probabilities_echo: bool = False
matcher
property matcher: GrammarMatcher | None
The optional grammar matcher for constrained decoding.
max_length
max_length: int
min_tokens
property min_tokens: int
The minimum number of new tokens to generate.
model_name
model_name: str = ''
new_padding_context()
classmethod new_padding_context(*, max_length, model_name)
Creates a single-token dummy context for DP batch padding.
DP batch padding must construct dummies of the architecture’s
concrete context type: for VLMs the overlap pipeline narrows every
context in an executed batch to TextAndVisionContext, so a
plain TextContext dummy would fail that check. Subclasses with
required constructor fields supply empty defaults via
_padding_context_required_fields().
realize_future_token()
realize_future_token(new_token, log_probabilities=None)
Overwrite the placeholder future token with the actual token.
This is primarily used for overlap scheduling.
-
Parameters:
-
- new_token (int)
- log_probabilities (LogProbabilities | None)
-
Return type:
-
None
request_id
request_id: RequestID
requires_structured_output_flag
property requires_structured_output_flag: bool
Whether this request requires –enable-structured-output.
reset()
reset()
Resets the context’s state by combining all tokens into a new prompt.
-
Return type:
-
None
restore_grammar_state()
restore_grammar_state(snapshot)
Forwards to GrammarEnforcementState.restore.
-
Parameters:
-
snapshot (GrammarEnforcementSnapshot)
-
Return type:
-
None
sampling_params
sampling_params: SamplingParams
set_matcher()
set_matcher(matcher)
Sets the grammar matcher for constrained decoding.
-
Parameters:
-
matcher (GrammarMatcher)
-
Return type:
-
None
set_thinking_region()
set_thinking_region(start_token_ids, end_token_ids)
Configure thinking region for conditional grammar enforcement.
When a thinking region is configured and _in_thinking_region is True,
grammar enforcement is suspended until the end token sequence is detected.
This enables reasoning output during constrained decoding.
set_tool_region()
set_tool_region(start_token_ids, end_token_ids)
Set token sequences for conditional tool call enforcement.
snapshot_grammar_state()
snapshot_grammar_state()
Forwards to GrammarEnforcementState.snapshot.
-
Return type:
spec_decoding_state
property spec_decoding_state: SpecDecodingState
Gets or creates the per-request speculative decoding state.
status
status: GenerationStatus = 'active'
target_endpoint
to_generation_output()
to_generation_output()
Get completion tokens that are ready to be returned to the user.
This method retrieves tokens that have been generated but not yet delivered to the user, along with their associated log probability data.
-
Returns:
-
The completion tokens and their associated log probabilities, if available.
-
Return type:
tokens
tokens: TokenBuffer
tools_forced
property tools_forced: bool
Whether tool calling was forced.
trace_carrier
Serialized W3C trace context (via opentelemetry.propagate.inject)
captured from the inbound request’s OTel context. Threaded onto this
context because it crosses into the model-worker process by value, so the
scheduler can re-extract it and parent its phase spans under the
caller’s trace instead of starting new root spans.
update()
update(new_token, log_probabilities=None)
Advance both token buffer and FSM state.
This is the standard single-step update that most callers should use.
It combines advance_token_buffer() and advance_fsm() for the
common case where both need to be advanced together.
For multi-step execution where FSM is advanced separately (e.g., to compute bitmasks between steps), use the individual methods directly.
-
Parameters:
-
- new_token (int) – The token to append and consume.
- log_probabilities (LogProbabilities | None) – Optional log probabilities for this token.
-
Return type:
-
None
update_enforcement_state()
update_enforcement_state(token)
Advance the grammar-enforcement state machine by one token.
Forwards to GrammarEnforcementState.update_enforcement_state().
update_with_future_token()
update_with_future_token()
Append a placeholder future token to the generated tokens.
This is primarily used for overlap scheduling. For structured output contexts (those with a matcher), only the token buffer is advanced. The FSM will be advanced later when the future token is realized with the actual generated token.
-
Return type:
-
None