For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

RequestStateStore

RequestStateStore

class max.pipelines.lib.request_state.RequestStateStore

source

Bases: ABC

Holds the state a registered decoder keeps for each request in flight.

It owns every request’s state: it assigns each new request a slot, assembles a batch’s state for the graph, stores what the graph returns, and frees a finished request’s slot.

input_types()

abstract input_types()

source

Returns the graph inputs the state adds, in graph order.

Return type:

tuple[TensorType | BufferType, …]

max_dims()

max_dims()

source

Returns the largest size of each symbolic dim of the state inputs.

A serving engine allocates each per-step input once at its largest size, so every symbolic dim of a TensorType input needs a bound. A BufferType input is allocated once by the store and needs none. The default bounds nothing, which suits a store whose tensor inputs have static shapes.

Return type:

Mapping[str, int]

prepare()

abstract prepare(batch)

source

Returns the step’s state inputs for a batch.

Gives each request it hasn’t seen a fresh slot first; a request it holds keeps its state, as across chunked prefill.

Parameters:

batch (Sequence[TextContext]) – The step’s contexts, in batch order.

Returns:

One buffer per input_types() entry, in its order.

Return type:

tuple[Buffer, …]

release()

abstract release(request_id)

source

Frees the state a finished request held.

Parameters:

request_id (RequestID)

Return type:

None

release_warmup()

release_warmup(request_ids)

source

Frees the state a graph-capture warmup probe claimed.

Parameters:

request_ids (Sequence[RequestID])

Return type:

None

update()

update(request_ids, outputs)

source

Stores the state a step returned into its requests’ slots.

The default is for a store whose graph updates its buffers in place, so the decoder returns no state.

Parameters:

• request_ids (Sequence[RequestID]) – The step’s requests, in batch order.
• outputs (Sequence[Buffer]) – The decoder’s state outputs, in their order.

Raises:

ValueError – If the decoder returned state.

Return type:

None