IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_rms_norm_quantize_mxfp8_q_kv

struct Struct_rms_norm_quantize_mxfp8_q_kv

Registers the quantization.rms_norm.quantize.mxfp8.q_kv graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute(output: T, scales: T, kv_output: T, input: T, q_gamma: T, kv_gamma: T, epsilon: Float32, context: DeviceContext)

RMS-normalizes the q and kv parts of each row, quantizing q to MXFP8.

Args:

  • ​output (T): The quantized q of shape [M, q_cols].
  • ​scales (T): The block scales of q in the 5D TCGEN interleaved layout.
  • ​kv_output (T): The normalized kv of shape [M, kv_cols].
  • ​input (T): The input of shape [M, q_cols + kv_cols], q then kv.
  • ​q_gamma (T): The q weight.
  • ​kv_gamma (T): The kv weight.
  • ​epsilon (Float32): The RMS norm epsilon.
  • ​context (DeviceContext): The device context.

Was this page helpful?