IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

LogProbabilitiesRagged

struct LogProbabilitiesRagged

Registers the compute_log_probabilities_ragged graph op computing per-token log probabilities over ragged batches.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[levels: Int](lp_logits: T, lp_tokens: T, logits: T, tokens: T, sampled_tokens: T, logit_row_offsets: T, token_row_offsets: T, lp_output_offsets: T, lp_output_offsets_host: T, ctx: DeviceContext)

Executes the ragged log-probabilities computation across all batches.

Parameters:

  • ​levels (Int): Number of heap levels controlling the top-k width; the output stores 2**levels entries per token.

Args:

  • ​lp_logits (T): Output tensor of shape [num_output_tokens, 2**levels] receiving the log probabilities.
  • ​lp_tokens (T): Output tensor of shape [num_output_tokens, 2**levels] receiving the token ids paired with lp_logits.
  • ​logits (T): Input logits ragged by batch, shape [total_rows, vocab_size].
  • ​tokens (T): Previously generated tokens across all batches, ragged.
  • ​sampled_tokens (T): Most recently sampled token for each batch.
  • ​logit_row_offsets (T): Per-batch start offsets into the first axis of logits.
  • ​token_row_offsets (T): Per-batch start offsets into tokens.
  • ​lp_output_offsets (T): Per-batch start offsets into the output row axis.
  • ​lp_output_offsets_host (T): Host-resident copy of lp_output_offsets whose last element gives the total number of output tokens.
  • ​ctx (DeviceContext): Device context used to enqueue the computation.

Raises:

Error: If lp_logits and lp_tokens disagree on axis 0, or if either output's axis 1 does not match 2**levels.

Was this page helpful?