IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.muse_glimmer

Muse Glimmer vision-language architecture (ModuleV3 API).

MuseGlimmerConfig​

class max.pipelines.architectures.muse_glimmer.MuseGlimmerConfig(*, devices, dtype, kv_params, image_token_id, video_token_id, text_config, vision_config, quantization_encoding=None)

source

Bases: ArchConfigWithKVCache, ArchConfigWithVisionCache

Top-level Muse Glimmer configuration composing text and vision.

Parameters:

DEFAULT_ENCODING​

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS​

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'bfloat16'}

source

calculate_max_seq_len()​

classmethod calculate_max_seq_len(huggingface_config, model_config)

source

Bounds max_length by the text decoder’s context.

Parameters:

Return type:

int

construct_kv_params()​

static construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

Constructs KV cache parameters from the top-level HuggingFace config.

Parameters:

Return type:

MultiKVCacheParams

devices​

devices: list[DeviceRef]

source

dtype​

dtype: DType

source

estimate_vision_cache_entry_bytes()​

classmethod estimate_vision_cache_entry_bytes(huggingface_config)

source

Bytes of one max-resolution image’s embeddings, or 0 for a checkpoint without a vision tower.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

finalize()​

finalize(huggingface_config, state_dict, return_logits)

source

Sets the fields that depend on the loaded weights and pipeline.

Parameters:

  • huggingface_config (AutoConfig) – HuggingFace model configuration.
  • state_dict (dict[str, WeightData]) – Model weights dictionary.
  • return_logits (ReturnLogits) – Return logits configuration.

Return type:

None

get_kv_params()​

get_kv_params()

source

Returns the KV cache parameters.

Return type:

MultiKVCacheParams

get_max_seq_len()​

get_max_seq_len()

source

Returns the maximum sequence length of the text decoder.

Return type:

int

get_vision_cache_row_spec()​

classmethod get_vision_cache_row_spec(huggingface_config)

source

One bfloat16 row of the text hidden size per merged image token.

Parameters:

huggingface_config (AutoConfig)

Return type:

tuple[int, DType] | None

image_token_id​

image_token_id: int

source

initialize()​

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initializes from pipeline configuration.

Parameters:

  • pipeline_config (PipelineConfig) – The MAX Engine pipeline configuration.
  • model_config (MAXModelConfig | None) – Optional model config override.
  • max_seq_len (int) – The effective maximum sequence length.

Returns:

An initialized config instance.

Return type:

Self

initialize_from_config()​

classmethod initialize_from_config(pipeline_config, huggingface_config, *, max_seq_len)

source

Initializes from pipeline and HuggingFace configs.

Parameters:

  • pipeline_config (PipelineConfig) – The MAX Engine pipeline configuration.
  • huggingface_config (AutoConfig) – Top-level HuggingFace model configuration.
  • max_seq_len (int) – The effective maximum sequence length.

Returns:

A config instance ready for finalization.

Return type:

Self

kv_params​

kv_params: MultiKVCacheParams

source

quantization_encoding​

quantization_encoding: SupportedEncoding | None = None

source

text_config​

text_config: MuseGlimmerTextConfig

source

video_token_id​

video_token_id: int

source

vision_config​

vision_config: MuseGlimmerVisionConfig | None

source

None when the checkpoint has no vision_config.

MuseGlimmerInputs​

class max.pipelines.architectures.muse_glimmer.MuseGlimmerInputs(tokens, input_row_offsets, return_n_logits, *, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: ModelInputs

Inputs for the Muse Glimmer language tower.

Image embeddings arrive on the base vision_embeddings / vision_scatter_indices fields, populated by the pipeline-owned VisionEncoderCache.

Parameters:

buffers​

property buffers: tuple[Buffer, ...]

source

Flat language-graph input tuple in compile ABI order.

input_row_offsets​

input_row_offsets: Buffer

source

Tensor containing the offsets for each row in the ragged input sequence.

return_n_logits​

return_n_logits: Buffer

source

Number of logits to return.

tokens​

tokens: Buffer

source

Tensor containing the input token IDs.

MuseGlimmerModel​

class max.pipelines.architectures.muse_glimmer.MuseGlimmerModel(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, max_batch_size=1)

source

Bases: LogProbabilitiesMixin, ModuleV3MultiGraphPipelineModelWithKVCache[TextAndVisionContext, Any]

The Muse Glimmer pipeline model (ModuleV3).

The vision and language towers are compiled as two graphs. The pipeline’s VisionEncoderCache drives the SupportsVisionEncoding hooks and scatters the image embeddings into the <|patch|> positions.

Parameters:

batch_processor_cls​

batch_processor_cls

source

alias of MuseGlimmerBatchProcessor

config​

config: MuseGlimmerConfig

source

empty_vision_embeddings()​

empty_vision_embeddings(devices)

source

Per-device zero-row image embeddings for text-only batches.

Cached: this is hit on every step, so it must not allocate per call.

Parameters:

devices (list[Device])

Return type:

list[Buffer]

execute()​

execute(model_inputs)

source

Executes the language tower with the prepared inputs.

Parameters:

model_inputs (ModelInputs)

Return type:

ModelOutputs

get_num_layers()​

classmethod get_num_layers(huggingface_config)

source

Parameters:

huggingface_config (AutoConfig)

Return type:

int

language_model​

language_model: CompiledCallable[Any, Any]

source

The compiled language tower.

model​

property model: CompiledCallable[Any, Any]

source

The language tower, for device graph capture and replay.

Vision runs at prefill through vision_execute(), so only the language tower is captured.

model_config_cls​

model_config_cls

source

alias of MuseGlimmerConfig

pack_vision_inputs()​

pack_vision_inputs(selection, devices)

source

Packs the pipeline-selected uncached images to device.

Parameters:

Return type:

list[Buffer] | None

vision_execute()​

vision_execute(selection, devices, packed)

source

Runs the vision tower on the images pack_vision_inputs() packed.

Parameters:

Return type:

VisionEncodeResult

vision_model​

vision_model: Callable[..., Any] | None

source

The compiled vision tower; None for a checkpoint without one.

MuseGlimmerTextConfig​

class max.pipelines.architectures.muse_glimmer.MuseGlimmerTextConfig(*, dtype, devices, hidden_size, num_attention_heads, num_key_value_heads, head_dim, num_hidden_layers, intermediate_size, vocab_size, rms_norm_eps, post_norm_eps, layer_types, layer_rope_theta, sliding_window, rope_theta, max_position_embeddings, max_seq_len, qk_scale_factor, output_multiplier, final_logit_softcapping, tie_word_embeddings=False, kv_params, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, target_layer_ids=None)

source

Bases: ArchConfigWithBoundedMaxSeqLen

Text decoder configuration for Muse Glimmer.

Parameters:

construct_kv_params()​

classmethod construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

Builds the sliding_attention + full_attention KV tree.

Parameters:

Return type:

MultiKVCacheParams

devices​

devices: list[DeviceRef]

source

dtype​

dtype: DType

source

final_logit_softcapping​

final_logit_softcapping: float

source

head_dim​

head_dim: int

source

hidden_size​

hidden_size: int

source

initialize_from_config()​

classmethod initialize_from_config(pipeline_config, huggingface_config, *, max_seq_len, dtype, cache_dtype)

source

Initializes from the HuggingFace text_config.

Parameters:

  • pipeline_config (PipelineConfig) – The MAX Engine pipeline configuration.
  • huggingface_config (AutoConfig) – The HuggingFace text configuration.
  • max_seq_len (int) – The effective maximum sequence length.
  • dtype (DType) – The weight dtype.
  • cache_dtype (DType) – The KV cache dtype.

Returns:

An initialized MuseGlimmerTextConfig.

Return type:

Self

intermediate_size​

intermediate_size: int

source

kv_params​

kv_params: MultiKVCacheParams

source

layer_rope_theta​

layer_rope_theta: list[float]

source

Per-layer RoPE theta; 0 marks a NoPE layer.

layer_types​

layer_types: list[str]

source

max_position_embeddings​

max_position_embeddings: int

source

max_seq_len​

max_seq_len: int

source

num_attention_heads​

num_attention_heads: int

source

num_hidden_layers​

num_hidden_layers: int

source

num_key_value_heads​

num_key_value_heads: int

source

output_multiplier​

output_multiplier: float

source

Scales the final hidden state before lm_head.

post_norm_eps​

post_norm_eps: float

source

Epsilon of the post-attention and post-FFN norms.

qk_scale_factor​

qk_scale_factor: float

source

return_hidden_states​

return_hidden_states: ReturnHiddenStates = 'none'

source

return_logits​

return_logits: ReturnLogits = 'last_token'

source

rms_norm_eps​

rms_norm_eps: float

source

Epsilon of the input, pre-FFN, embedding, qk and final norms.

rope_theta​

rope_theta: float

source

sliding_window​

sliding_window: int

source

target_layer_ids​

target_layer_ids: list[int] | None = None

source

tie_word_embeddings​

tie_word_embeddings: bool = False

source

use_rope​

use_rope: list[bool]

source

vocab_size​

vocab_size: int

source

MuseGlimmerVisionConfig​

class max.pipelines.architectures.muse_glimmer.MuseGlimmerVisionConfig(*, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, hidden_act, layer_norm_eps, layer_types, patch_size, patch_temporal, merge_size, pos_emb_height, pos_emb_width, rope_theta, projector_hidden_size, window_size_patches=32)

source

Bases: object

Vision tower and adapter configuration for Muse Glimmer.

Parameters:

  • hidden_size (int)
  • intermediate_size (int)
  • num_hidden_layers (int)
  • num_attention_heads (int)
  • hidden_act (str)
  • layer_norm_eps (float)
  • layer_types (list[str])
  • patch_size (int)
  • patch_temporal (int)
  • merge_size (int)
  • pos_emb_height (int)
  • pos_emb_width (int)
  • rope_theta (float)
  • projector_hidden_size (int)
  • window_size_patches (int)

head_dim​

property head_dim: int

source

hidden_act​

hidden_act: str

source

hidden_size​

hidden_size: int

source

initialize_from_config()​

classmethod initialize_from_config(huggingface_config)

source

Initializes from the top-level HuggingFace config.

Parameters:

huggingface_config (AutoConfig) – The top-level HuggingFace configuration, which holds the projector widths beside vision_config.

Returns:

An initialized MuseGlimmerVisionConfig.

Return type:

MuseGlimmerVisionConfig

intermediate_size​

intermediate_size: int

source

layer_norm_eps​

layer_norm_eps: float

source

layer_types​

layer_types: list[str]

source

full_attention or window_attention per layer.

merge_size​

merge_size: int

source

num_attention_heads​

num_attention_heads: int

source

num_hidden_layers​

num_hidden_layers: int

source

out_hidden_size​

property out_hidden_size: int

source

Width of one merged token after the pixel shuffle.

patch_dim​

property patch_dim: int

source

Width of one flattened temporal x RGB x patch x patch input.

patch_size​

patch_size: int

source

patch_temporal​

patch_temporal: int

source

pos_emb_height​

pos_emb_height: int

source

pos_emb_width​

pos_emb_width: int

source

projector_hidden_size​

projector_hidden_size: int

source

rope_theta​

rope_theta: float

source

window_size_patches​

window_size_patches: int = 32

source

Side of the square patch window the window_attention layers use.

Models

Cookbook

Puzzles

Blog

Community