For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

ModuleRuntime

ModuleRuntime

class max.pipelines.lib.registration_runtime.ModuleRuntime(mesh, data_parallel_degree, max_batch_input_tokens, max_batch_total_tokens, ep_size=1, ep_use_allreduce=False, ep=None, max_batch_size=None)

source

Bases: object

The deployment a registered module’s config is completed with.

Parameters:

• mesh (DeviceMesh)
• data_parallel_degree (int)
• max_batch_input_tokens (int)
• max_batch_total_tokens (int | None)
• ep_size (int)
• ep_use_allreduce (bool)
• ep (EPConfig | None)
• max_batch_size (int | None)

data_parallel_degree

data_parallel_degree: int

source

The number of data-parallel replicas.

ep

ep: EPConfig | None = None

source

The initialized expert-parallel config, under expert parallelism.

ep_size

ep_size: int = 1

source

The number of devices the experts are spread over; one without expert parallelism.

ep_use_allreduce

ep_use_allreduce: bool = False

source

Whether expert parallelism combines through an allreduce.

max_batch_input_tokens

max_batch_input_tokens: int

source

The most tokens one batch feeds the decoder.

max_batch_size

max_batch_size: int | None = None

source

The most requests a step holds, as the memory planner sized it, or None when the caller doesn’t size batches.

max_batch_total_tokens

max_batch_total_tokens: int | None

source

The KV context one batch can hold, as the memory planner sized it, or None when the planner set no bound.

mesh

mesh: DeviceMesh

source

The devices the decoder runs on, as the framework constructs it.