For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
LogProbabilitiesRagged
struct LogProbabilitiesRagged
Registers the compute_log_probabilities_ragged graph op computing per-token log probabilities over ragged batches.
Implemented traits
Methods
execute
static def execute[levels: Int](lp_logits: T, lp_tokens: T, logits: T, tokens: T, sampled_tokens: T, logit_row_offsets: T, token_row_offsets: T, lp_output_offsets: T, lp_output_offsets_host: T, ctx: DeviceContext)
Executes the ragged log-probabilities computation across all batches.
Parameters:
- levels (
Int): Number of heap levels controlling the top-k width; the output stores2**levelsentries per token.
Args:
- lp_logits (
T): Output tensor of shape[num_output_tokens, 2**levels]receiving the log probabilities. - lp_tokens (
T): Output tensor of shape[num_output_tokens, 2**levels]receiving the token ids paired withlp_logits. - logits (
T): Input logits ragged by batch, shape[total_rows, vocab_size]. - tokens (
T): Previously generated tokens across all batches, ragged. - sampled_tokens (
T): Most recently sampled token for each batch. - logit_row_offsets (
T): Per-batch start offsets into the first axis oflogits. - token_row_offsets (
T): Per-batch start offsets intotokens. - lp_output_offsets (
T): Per-batch start offsets into the output row axis. - lp_output_offsets_host (
T): Host-resident copy oflp_output_offsetswhose last element gives the total number of output tokens. - ctx (
DeviceContext): Device context used to enqueue the computation.
Raises:
Error: If lp_logits and lp_tokens disagree on axis 0, or if
either output's axis 1 does not match 2**levels.