IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

HeadRMSNormSigmoidGate

struct HeadRMSNormSigmoidGate

Registers the norm.head_rms_norm_sigmoid_gate graph op.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[n_heads: Int](output: T, x: T, qkvg: T, epsilon: Float32, ctx: DeviceContext)

Writes rms_norm(x) * sigmoid(gate) per 64-channel head, with the gate the last n_heads * 64 columns of qkvg.

Parameters:

  • ​n_heads (Int): Heads in x (this device's).

Args:

  • ​output (T): [tokens, n_heads * 64].
  • ​x (T): The attention output [tokens, n_heads * 64].
  • ​qkvg (T): The [q | k | v | gate] projection.
  • ​epsilon (Float32): The norm's epsilon.
  • ​ctx (DeviceContext): Device context.

Raises:

Error: If the launch fails.

Was this page helpful?