IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_grouped_matmul_block_scaled_amd_swiglu_quant

struct Struct_grouped_matmul_block_scaled_amd_swiglu_quant[lane_bytes: Int = Int(32), scales_max_padded_m: Int = Int(0)]

MOGG wrapper for grouped MXFP8 matmul with a fused SwiGLU epilogue.

Runs gemm1 of an MoE FFN and folds the gated activation plus the MXFP8 requantize into its epilogue, so the activation leaves the kernel already quantized and no standalone activation kernel is needed. AMD preshuffled-B only.

Parameters​

  • ​lane_bytes (Int): Operand bytes per lane. MXFP8 only, so always 32.
  • ​scales_max_padded_m (Int): Per-expert slot stride for the emitted scales. Nonzero writes the 4D slot layout that gemm2 reads without a preshuffle; 0 writes plain row-major.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[a_type: DType, b_type: DType, //, target: StringSpan[ImmStaticOrigin], activation: StringSpan[ImmStaticOrigin] = StringSpan("silu"), swiglu_alpha: Float32 = 0, swiglu_limit: Float32 = 0](c: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c.static_spec], c_scales: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=c_scales.static_spec], a: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a.static_spec], b: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b.static_spec], a_scales: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=a_scales.static_spec], b_scales: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=b_scales.static_spec], expert_start_indices: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_start_indices.static_spec], expert_ids: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=expert_ids.static_spec], max_num_tokens_per_expert: UInt32, num_active_experts: UInt32, estimated_total_m: UInt32, decode_grid_m_cap: UInt32, decode_grid_m_rows: UInt32, context: DeviceContextBase)

Executes gemm1 with the SwiGLU and MXFP8 requantize fused in.

Parameters:

  • ​a_type (DType): The activation dtype (one byte wide).
  • ​b_type (DType): The weight dtype (one byte wide).
  • ​target (StringSpan[ImmStaticOrigin]): The target GPU device.
  • ​activation (StringSpan[ImmStaticOrigin]): The gated activation the epilogue applies, built at compile time with swiglu_alpha and swiglu_limit.
  • ​swiglu_alpha (Float32): The activation's alpha.
  • ​swiglu_limit (Float32): The activation's clamp limit.

Args:

Was this page helpful?