For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_grouped_matmul_block_scaled_amd_swiglu_quant
struct Struct_grouped_matmul_block_scaled_amd_swiglu_quant[lane_bytes: Int = Int(32), scales_max_padded_m: Int = Int(0)]
MOGG wrapper for grouped MXFP8 matmul with a fused SwiGLU epilogue.
Runs gemm1 of an MoE FFN and folds the gated activation plus the MXFP8 requantize into its epilogue, so the activation leaves the kernel already quantized and no standalone activation kernel is needed. AMD preshuffled-B only.
Parameters
- lane_bytes (
Int): Operand bytes per lane. MXFP8 only, so always 32. - scales_max_padded_m (
Int): Per-expert slot stride for the emitted scales. Nonzero writes the 4D slot layout that gemm2 reads without a preshuffle; 0 writes plain row-major.
Implemented traits
Methods
execute
static def execute[a_type: DType, b_type: DType, //, target: StringSpan[ImmStaticOrigin], activation: StringSpan[ImmStaticOrigin] = StringSpan("silu"), swiglu_alpha: Float32 = 0, swiglu_limit: Float32 = 0](c: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c.static_spec], c_scales: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c_scales.static_spec], a: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a.static_spec], b: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b.static_spec], a_scales: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a_scales.static_spec], b_scales: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b_scales.static_spec], expert_start_indices: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_start_indices.static_spec], expert_ids: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_ids.static_spec], max_num_tokens_per_expert: UInt32, num_active_experts: UInt32, estimated_total_m: UInt32, decode_grid_m_cap: UInt32, decode_grid_m_rows: UInt32, context: DeviceContextBase)
Executes gemm1 with the SwiGLU and MXFP8 requantize fused in.
Parameters:
- a_type (
DType): The activation dtype (one byte wide). - b_type (
DType): The weight dtype (one byte wide). - target (
StringSpan[ImmStaticOrigin]): The target GPU device. - activation (
StringSpan[ImmStaticOrigin]): The gated activation the epilogue applies, built at compile time withswiglu_alphaandswiglu_limit. - swiglu_alpha (
Float32): The activation's alpha. - swiglu_limit (
Float32): The activation's clamp limit.
Args:
- c (
ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c.static_spec]): The MXFP8 activation, shape (total_tokens, N / 2). - c_scales (
ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c_scales.static_spec]): The activation's E8M0 block scales. - a (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a.static_spec]): The input tensor of shape (total_tokens, K). - b (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b.static_spec]): Sigma-permuted weights, (num_experts, N, K). - a_scales (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a_scales.static_spec]): The A scale factors in 2D layout. - b_scales (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b_scales.static_spec]): Sigma-permuted B scale factors in 3D layout. - expert_start_indices (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_start_indices.static_spec]): The starting token index for each expert. - expert_ids (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_ids.static_spec]): The expert ID for each group. - max_num_tokens_per_expert (
UInt32): The maximum token count for any expert. - num_active_experts (
UInt32): The number of active experts. - estimated_total_m (
UInt32): Estimated total received tokens for this GPU, used to pick the persistent vs direct kernel path. - decode_grid_m_cap (
UInt32): Decode-band gate; 0 disables. - decode_grid_m_rows (
UInt32): Rows grid.y must cover per expert at decode. - context (
DeviceContextBase): The device context pointer.