For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.eagle_llama3

EAGLE speculative decoding draft model for Llama 3.

EagleLlama3ArchConfig

class max.pipelines.architectures.eagle_llama3.EagleLlama3ArchConfig(*, hidden_size, num_attention_heads, num_key_value_heads, num_hidden_layers, rope_theta, rope_scaling_params, max_seq_len, intermediate_size, interleaved_rope_weights, vocab_size, dtype, model_quantization_encoding, quantization_config, kv_params, return_logits=ReturnLogits.LAST_TOKEN, norm_method='rms_norm', norm_dtype=None, attention_bias=False, rms_norm_eps=None, tie_word_embeddings=False, stacked_mlp=False, stacked_qkv=False, attention_multiplier, embedding_multiplier, residual_multiplier, devices, clip_qkv, quant_config=None, longrope_scaling_params=None, logits_scaling=1.0, return_hidden_states=ReturnHiddenStates.NONE, target_layer_ids=None, use_subgraphs=True, data_parallel_degree=1, sliding_window=None, quantization_encoding=None)

source

Bases: Llama3Config

The EAGLE draft’s Llama 3 config, which builds the draft’s module.

The draft runs only beside a target, so it has no pipeline model.

Parameters:

• hidden_size (int)
• num_attention_heads (int)
• num_key_value_heads (int)
• num_hidden_layers (int)
• rope_theta (float)
• rope_scaling_params (Llama3RopeScalingParams | None)
• max_seq_len (int)
• intermediate_size (int)
• interleaved_rope_weights (bool)
• vocab_size (int)
• dtype (DType)
• model_quantization_encoding (QuantizationEncoding | None)
• quantization_config (QuantizationConfig | None)
• kv_params (KVCacheParams)
• return_logits (ReturnLogits)
• norm_method (Literal['rms_norm', 'layer_norm'])
• norm_dtype (DType | None)
• attention_bias (bool)
• rms_norm_eps (float | None)
• tie_word_embeddings (bool)
• stacked_mlp (bool)
• stacked_qkv (bool)
• attention_multiplier (float)
• embedding_multiplier (float)
• residual_multiplier (float)
• devices (list[DeviceRef])
• clip_qkv (float | None)
• quant_config (QuantConfig | None)
• longrope_scaling_params (LongRoPEScalingParams | None)
• logits_scaling (float)
• return_hidden_states (ReturnHiddenStates)
• target_layer_ids (list[int] | None)
• use_subgraphs (bool)
• data_parallel_degree (int)
• sliding_window (int | None)
• quantization_encoding (SupportedEncoding | None)

build_module()

classmethod build_module(context)

source

Builds the EAGLE draft from the draft checkpoint context names.

The draft computes in its checkpoint’s own precision. It embeds and projects through its target’s embedding and head, so spec decoding shares those onto it.

Raises:

ValueError – If the method isn’t eagle.

Parameters:

context (ModelLoadContext)

Return type:

tuple[EagleLlama3, Llama3Config]