1. Notation: what the model actually outputs#

Let $x_{1:t}$ be the tokens generated so far. A causal language model (e.g., GPT-2-like) outputs logits $z \in \mathbb{R}^{|V|}$ for the next token:

These logits become a probability distribution by softmax:

Important detail:

• model.generate(...) returns token ids (the chosen tokens), not logits.
• To get logits/scores during generation in HuggingFace, you must request them (see code below).

2. A vivid example (correctly extracting logits)#

We feed GPT-2 with the prompt "The cat ran after a" and generate 4 tokens. We will print:

• the chosen next token ids (what decoding decided)
• the logits distribution for each step (what the model predicted)

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = GPT2LMHeadModel.from_pretrained("gpt2").to(device)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model.eval()

prompt = "The cat ran after a"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

with torch.no_grad():
    out = model.generate(
        input_ids,
        max_new_tokens=4,
        do_sample=False,  # greedy by default
        return_dict_in_generate=True,
        output_scores=True,
    )

sequences = out.sequences[0]                 # token ids (prompt + generated)
new_token_ids = sequences[-4:]               # only the 4 new tokens

print("New token ids:", new_token_ids.tolist())
print("Generated text:", tokenizer.decode(sequences, skip_special_tokens=True))

# out.scores is a list of length = max_new_tokens
# each element is logits for that step: [batch, vocab]
for step, logits in enumerate(out.scores, start=1):
    topk = torch.topk(logits[0], k=5)
    ids = topk.indices.tolist()
    vals = topk.values.tolist()
    toks = [tokenizer.decode([i]) for i in ids]
    print(f"Step {step} top-5 tokens:")
    for tok, logit in zip(toks, vals):
        print(f"  {tok!r:>10}  logit={logit:8.3f}")

Conceptually:

• The model computes logits $z$.
• The decoder turns logits into the next token (greedy, beam, sampling, …).

3. Temperature: sharpening or flattening the distribution#

Before choosing a token, it is common to apply temperature scaling $T>0$:

Effects:

• $T \to 0^+$: distribution becomes very peaked (approaches greedy / argmax)
• $T = 1$: original distribution
• $T > 1$: flatter distribution (more randomness)

In practice, temperature is one of the most important knobs for “creativity vs. correctness”.

4. Greedy decoding (deterministic)#

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = GPT2LMHeadModel.from_pretrained("gpt2").to(device)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model.eval()

prompt = "The cat ran after a"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

max_new_tokens = 4
with torch.no_grad():
    for _ in range(max_new_tokens):
        logits = model(input_ids).logits  # [batch, seq_len, vocab]
        next_logits = logits[:, -1, :]    # [batch, vocab]
        next_id = torch.argmax(next_logits, dim=-1, keepdim=True)
        input_ids = torch.cat([input_ids, next_id], dim=1)

print(tokenizer.decode(input_ids[0], skip_special_tokens=True))

5. Beam search (deterministic, “global-ish”)#

Greedy decoding is locally optimal. Beam search keeps multiple hypotheses.

out = model.generate(
    input_ids,
    max_new_tokens=80,
    num_beams=4,
    do_sample=False,
    early_stopping=True,
    length_penalty=1.0,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

6. Sampling (stochastic)#

Sampling draws the next token from the probability distribution instead of always taking the max:

Sampling is the main tool for diversity.

7. Top-$k$ sampling#

Top-$k$ sampling keeps only the $k$ most probable tokens and renormalizes:

1. Let $S_k$ be the set of the $k$ tokens with highest probability.
2. Define $p'(v) \propto p(v)$ for $v\in S_k$, and $p'(v)=0$ otherwise.
3. Sample $x_{t+1} \sim \text{Categorical}(p')$.

Intuition:

• Small $k$ behaves more like greedy.
• Large $k$ approaches naive sampling.

HuggingFace:

out = model.generate(
    input_ids,
    max_new_tokens=120,
    do_sample=True,
    top_k=50,
    temperature=0.8,
)

8. Top-$p$ (nucleus) sampling#

Top-$p$ chooses the smallest token set whose cumulative probability is at least $p$.

Let tokens be sorted by probability: $p_1 \ge p_2 \ge \cdots$. Choose the smallest $m$ such that:

Then sample from those $m$ tokens after renormalization.

Why this is often better than top-$k$:

• The “right” number of plausible tokens changes with context.
• Top-$p$ adapts the candidate set size automatically.

HuggingFace:

out = model.generate(
    input_ids,
    max_new_tokens=120,
    do_sample=True,
    top_p=0.9,
    temperature=0.8,
)

Rule of thumb: start with top_p=0.9 and tune temperature.

9. A single-step implementation: temperature + top-$k$ + top-$p$#

The following helper illustrates the core mechanics.

import torch

def sample_next_token(
    logits: torch.Tensor,
    temperature: float = 1.0,
    top_k: int | None = None,
    top_p: float | None = None,
):
    # logits: [vocab]
    if temperature <= 0:
        raise ValueError("temperature must be > 0")

    logits = logits / temperature

    # Top-k truncation
    if top_k is not None and top_k > 0:
        values, _ = torch.topk(logits, k=min(top_k, logits.numel()))
        cutoff = values[-1]
        logits = torch.where(logits < cutoff, torch.tensor(float("-inf"), device=logits.device), logits)

    # Convert to probs for top-p
    probs = torch.softmax(logits, dim=-1)

    if top_p is not None and 0 < top_p < 1:
        sorted_probs, sorted_idx = torch.sort(probs, descending=True)
        cdf = torch.cumsum(sorted_probs, dim=-1)
        keep = cdf <= top_p
        # ensure at least 1 token
        keep[..., 0] = True

        filtered = torch.zeros_like(probs)
        filtered[sorted_idx[keep]] = probs[sorted_idx[keep]]
        probs = filtered / filtered.sum()

    next_id = torch.multinomial(probs, num_samples=1)
    return next_id

This is essentially what “sampling” decoders do, with a few extra details for speed and numerical stability.

10. Preventing repetition and degeneration#

Many decoding failures come from degeneration: loops, repeated phrases, or bland continuations.

out = model.generate(
    input_ids,
    max_new_tokens=160,
    do_sample=True,
    top_p=0.9,
    temperature=0.8,
    repetition_penalty=1.1,
)

out = model.generate(
    input_ids,
    max_new_tokens=160,
    do_sample=True,
    top_p=0.9,
    no_repeat_ngram_size=3,
)

11. Choosing a strategy: practical guidance#

There is no universally best decoder; choose based on task.

12. Two modern add-ons (optional but useful)#

out = model.generate(
    input_ids,
    max_new_tokens=160,
    penalty_alpha=0.6,
    top_k=4,
)

13. Summary#

• The model outputs logits; decoding decides what tokens to emit.
• Greedy: fastest and stable, but can be dull.
• Beam search: better for structured tasks; can be dull for open-ended text.
• Sampling (top-$k$/top-$p$ + temperature): best for diversity; needs controls to avoid incoherence.
• Repetition controls and stopping rules are part of decoding in practice.

If you want, I can also add a small section comparing these methods on the same prompt with side-by-side outputs (and plots of entropy / top-$p$ set size) to make the differences visually obvious.