Learning path · Level 2 · Tutorial 4

How Language Models Generate Text

Explain how language models produce text and use that mechanism to account for context limits, varied answers, and confident factual errors.

  • Level: Beginner
  • Duration: 17 min
  • 8 steps
Start tutorial Back to the path
How Language Models Generate Text

What you will go through

  1. Text Becomes Tokens Understand tokens as text units and see why a token count can differ from a word count. 2 min
  2. Learning to Predict What Comes Next See how token sequences become training examples for next-token prediction, and why learned weights are patterns rather than a searchable store of ready-made responses. 2 min
  3. Generating One Token at a Time Trace how a language model repeatedly predicts, selects, and appends one token at a time. 3 min
  4. Why Answers Can Vary See how sampling among plausible next tokens can produce different continuations from the same starting sequence. 2 min
  5. Learned Patterns and Current Context Distinguish patterns learned during training from information supplied in a current context, including the finite context window. 2 min
  6. When Earlier Information Is Missing See how a context boundary can remove earlier information and why that differs from sampling-driven variation. 2 min
  7. Why Fluent Answers Can Be Wrong 2 min
  8. Explain a Model Answer Apply tokenization, repeated next-token generation, sampling, context limits, and source comparison to explain a compact model-answer case. 2 min

What you will learn

  • Interpret an illustrative token split and explain why token counts need not equal word counts.
  • Trace a short worked example of repeated next-token selection.
  • Distinguish information learned during training from information supplied in the current context.
  • Explain how limited context and different token selections can contribute to inconsistent answers.
  • Explain why a fluent continuation can contain invented details even when it sounds confident.

Before you start

  • How Models Learn from Data
  • How Neural Networks Learn Layered Patterns

Step 1 of 8

Text Becomes Tokens

Understand tokens as text units and see why a token count can differ from a word count.

See the boundaries

Tokens are text units

Before a language model can work with text, it represents the text as tokens. A token is a unit of text. It may be a whole word, a word fragment, punctuation, or text that includes a space.

The brackets in this lesson show token boundaries. They are an illustration of how one sentence can be divided; the brackets are not part of the sentence itself.

One sentence, several token units

A diagram divides the sentence Reading is fun. into blocks for Read, ing, space-plus-is, space-plus-fun, and the period.

Illustrative token split; actual splits depend on the tokenizer

Words and tokens are different counts

Example

Read the illustrated split

Here is the illustrated split:

[Read][ing][ is][ fun][.]

The visible word “Reading” is divided into [Read] and [ing], so a token boundary does not have to match a word boundary. [ fun] is a whole-word token in this illustration, and its leading space is part of that token. [.] is a punctuation token.

The sentence has 3 words: Reading, is, fun. It has 5 tokens in this illustrated split. Counting words therefore does not necessarily give the token count.

Why the count can differ

This is not a universal counting rule. Actual token splits depend on the tokenizer: a system that divides text into tokens. Another tokenizer may divide the same sentence differently.

The useful idea is not to memorize this particular count. Instead, notice that tokens can cross the boundaries we normally use when counting words.

Tip

Look closely at spaces

When you see brackets such as [ is], check whether a space appears inside the brackets. In an illustrative split, that space belongs to the token.

Quick check

Classify the token units

Match each token from the illustrated split to its type.

Apply the idea

Explain the token count

Refer to the illustrative split from the earlier screen. Identify one whole-word token, one word-fragment token, and the punctuation token. Then explain why this sentence has 3 words but 5 tokens in the illustration.

Write at least 60 characters (0/60).

Step 2 of 8

Learning to Predict What Comes Next

See how token sequences become training examples for next-token prediction, and why learned weights are patterns rather than a searchable store of ready-made responses.

A target comes from the sequence

From a sequence to a training task

Training uses sequences of tokens to create many small learning tasks.

Consider this illustrative split:

Reading is fun.
[Read][ing][ is][ fun][.]

Illustrative token split; actual splits depend on the tokenizer

For a task called next-token prediction, the model receives a preceding sequence and the following token is the target. For example, after [Read][ing][ is], the target is [ fun].

During training, the model compares its prediction with that target and adjusts its weights to become more useful at predicting what follows.

A shifted training example

A row of token blocks where a preceding group points to the next highlighted token block.

A preceding sequence is the input; the following token provides the target for next-token prediction.

What training leaves behind

Example

Many small prediction examples

One token sequence can create overlapping training examples by shifting the target one position at a time:

  • Preceding sequence: [Read] → target: [ing]
  • Preceding sequence: [Read][ing] → target: [ is]
  • Preceding sequence: [Read][ing][ is] → target: [ fun]

Across many examples, training develops patterns about wording, grammar, and associations across text. The goal is not to memorize one fixed reply for each sequence, but to make predictions that fit patterns seen across many sequences.

Warning

Patterns are not stored replies

The learned patterns are encoded in the model's weights—the adjustable settings changed during training. Weights are not a dependable searchable archive of the exact training examples.

A model may produce wording that resembles a pattern it learned, but that does not mean it can reliably look up, quote, or prove a particular training sentence.

Practice: read a training example

Identify the training task

Consider this illustrative training sequence: [The][ door][ opens].

Which option correctly identifies one training case and what the model learns from many such cases?

Step 3 of 8

Generating One Token at a Time

Trace how a language model repeatedly predicts, selects, and appends one token at a time.

From sequence to next-token prediction

Predict → Select → Append → Repeat

Generation begins with the tokens available so far. For the starting sequence Reading is, the model considers possible next tokens and assigns each an illustrative probability. It does not need to retrieve a complete, finished sentence first.

The generation cycle is: Predict → Select → Append → Repeat. A selected token becomes part of the available sequence before the next prediction.

One generation cycle

A diagram showing the available sequence [Read][ing][ is], possible next-token candidates, one selected token, and an extended sequence.

Predict → Select → Append → Repeat: the selected token is appended to [Read][ing][ is] and becomes available for the next prediction.

Possible next tokens after `Reading is`

Illustrative probabilities, not measured model output

This complete illustrative distribution totals 100%.
Candidate next tokenProbability
[ fun]50%
[ a]30%
[ not]20%

Extend the sequence, then predict again

Example

Two iterations of generation

In this worked example, [ fun] is selected first and appended. The model then makes a new prediction using the longer sequence. Next, [.] is selected and appended.

Each new prediction uses the sequence built so far. Later possibilities and probabilities can therefore differ from earlier ones.

Sequence extension walkthrough

The result appended in one iteration is the available sequence for the next iteration.

Each selection extends the sequence before the next prediction.
IterationPredict usingSelectAppend to form
1[Read][ing][ is][ fun][Read][ing][ is][ fun]
2[Read][ing][ is][ fun][.][Read][ing][ is][ fun][.]

Repeat until stopping

This walkthrough stops after the period. More generally, repeated cycles continue until a stopping condition is reached. The output is constructed token by token; it is not necessarily a finished answer retrieved all at once.

Practice the generation cycle

Reconstruct the two iterations

Put the operations from the worked example in their correct order. Include the updated sequence after each selected token.

  1. Predict using [Read][ing][ is][ fun]
  2. Predict using [Read][ing][ is]
  3. Append it: [Read][ing][ is][ fun]
  4. Select [.]
  5. Append it: [Read][ing][ is][ fun][.]
  6. Select [ fun]

Step 4 of 8

Why Answers Can Vary

See how sampling among plausible next tokens can produce different continuations from the same starting sequence.

More than one next token may fit

Several continuations can fit

For the same available sequence, several next tokens may be plausible. The model can assign a probability to each candidate rather than finding one required continuation.

A higher probability means a candidate is more likely under the model’s learned patterns. It does not mean that every other candidate is impossible.

A branching next-token choice

A token sequence branches into three possible next-token paths, with one path wider than the others.

One available sequence can lead to several candidate next tokens. Branch width represents relative likelihood, not a guaranteed choice.

Sampling can choose a non-top candidate

Probabilities describe candidates

Consider the starting sequence: I would like.

Illustrative probabilities, not measured model output

The probabilities below form a complete illustrative distribution: together, they total 100%.

Illustrative candidate tokens

Possible next tokens after I would like

A complete illustrative distribution for one next-token prediction.
Candidate tokenProbability
`[ tea]`50%
`[ coffee]`30%
`[ water]`20%

Example

Sampling is not a fixed rule

Sampling selects among candidate tokens using their probabilities. It can select [ coffee] or [ water] even though [ tea] has the largest probability.

So, “highest probability” does not mean “must be chosen.”

One branch can lead to another

Example

An early selection can change what follows

Suppose two runs begin with exactly the same starting sequence, I would like.

Run A: [ tea] is selected, then later [ with lemon], then [.].

Run B: [ coffee] is selected, then later [.].

After the first selection, each run has a different sequence. The next-token prediction is therefore based on a different sequence, so later continuations can diverge further.

Different runs may also match

Variation is possible, not guaranteed. With the same full starting sequence, separate runs can select different plausible tokens through sampling. They can also select the same token and produce the same result.

Apply the branching idea

Check your understanding

Two runs begin with exactly the same sequence but continue differently. Which explanation is justified?

Step 5 of 8

Learned Patterns and Current Context

Distinguish patterns learned during training from information supplied in a current context, including the finite context window.

Two places information can come from

Two information sources

Two sources of information

A language model can draw on two different sources while generating text:

  • Learned during training: broad patterns encoded in its model weights, such as patterns of wording, grammar, and associations.
  • Supplied in the current context: information included for this generation, such as instructions, supplied text, conversation messages, and tokens generated so far.

The two sources work together, but they are not the same thing. Training happened before the current request; context is available for the current generation.

A two-panel diagram contrasts broad patterns encoded in model weights during training with instructions, source text, messages, and generated tokens available in the current context.

Information learned during training and information supplied in the current context are distinct inputs to generation.

Use information supplied now

A supplied note

Adding a note does not retrain the model

Suppose a request includes a short note about Maple Library. That note is Supplied in the current context, so the model can use its details while answering the request.

Sending the note does not, by itself, change the model weights. In other words, it does not immediately add a new learned fact to the model through training.

Example

Source note

The workshop starts at noon. Entry is free.

The note is available for this answer because it was supplied with the request.

The model answer can use the details present in the current source note.
Source noteModel answer
The workshop starts at noon. Entry is free.The Maple Library workshop starts at noon, and entry is free.

Tip

Key distinction

A current message can make information available for one generation without creating an immediate weight update.

Context has a limit

A bounded working span

The context window is finite

The context window is the finite span of tokens available for a generation. It can contain instructions, supplied text, conversation messages, and generated tokens so far.

Both the input and the developing output occupy space in this window. Its exact capacity depends on the model and product, so there is no universal token limit to memorize.

A token timeline shows a limited current context between earlier material outside the context and generated text that continues to occupy the available span.

The current context is a finite token span: supplied input and generated text share its available space.

Check your understanding

Classify the information source

Match each description to its correct classification.

Tap an item and then its matching pair.

Step 6 of 8

When Earlier Information Is Missing

See how a context boundary can remove earlier information and why that differs from sampling-driven variation.

What fits in the current context?

A boundary around available information

A long conversation can contain more text than the model receives for one response. Its context window is the finite token span available at that moment—not necessarily the complete conversation history.

In this example, an earlier message claims that Maple Library has 20 seats. That message is now Outside the current context. The available note in the Current context says: The workshop starts at noon. Entry is free.

The earlier seat claim is not directly available as current-context information, even though the model might still produce related wording from learned patterns or an inference.

Context-boundary diagram

A conversation timeline: an old Maple Library seat claim is faded outside the active context boundary, while a note with a noon clock and free-entry ticket is inside the current context; generated text follows the active window.

The faded area is Outside the current context. The outlined area is the Current context, containing The workshop starts at noon. Entry is free. The speech bubble is Generated text.

Do not infer one cause too quickly

Two different sources of inconsistency

Products can handle a long history in different ways. They may omit earlier material, replace it with a summary, or require shorter input. Do not assume every product keeps or removes messages in the same way.

When an earlier detail is not supplied in the current context, it cannot directly guide the next-token prediction for that response. This is different from sampling: sampling can produce different continuations even when the available starting context is unchanged.

Missing context versus sampling

An output by itself does not reveal which mechanism caused a difference.

Both mechanisms can contribute to inconsistent answers, but they describe different situations.
QuestionUnavailable contextSampling
What changed?An earlier detail is no longer available.A next token can be selected differently.
Starting contextMay be missing relevant earlier information.Can be unchanged across runs.
What can output alone prove?It cannot prove this was the cause.It cannot prove this was the cause.

Quick check

Check the causal distinction

Two answers in a long conversation disagree about a library detail. What is the most justified conclusion from the answers alone?

Apply the distinction

Explain the diagram

Refer to the context-boundary diagram from the first screen. In 2–4 sentences, identify which Maple Library details remain in the Current context, identify the earlier detail that is outside it, and explain how that absence differs from variation caused by sampling.

Write at least 100 characters (0/100).

Step 7 of 8

Why Fluent Answers Can Be Wrong

Fluent wording can add details

Plausible is not verified

Source note

The workshop starts at noon. Entry is free.

Model answer

“The workshop starts at noon. Entry is free. The room has 20 seats.”

A model generates a plausible continuation from learned patterns and the current context. Plausible wording is not automatically a verified factual statement.

Here, the time and admission details are supported by the note. The seat count is not supplied by it. A hallucination is a generated detail that is incorrect or unsupported. It describes an output failure, not an intention to deceive.

A comparison diagram shows that a generated answer can repeat two details supported by a source while adding an unconnected claim about 20 seats.

The extra seat detail can sound natural even though the supplied note does not establish it.

Three evidence labels

Compare claims with the source

Use the source itself to distinguish three kinds of details. A claim that is Not established is not automatically false: the source simply does not provide enough evidence for it.

Confidence, polished phrasing, or a high probability for a token do not change these labels. They describe how likely a continuation may be, not whether a source proves it.

Evidence labels

Classify a detail by its relationship to the Source note, not by how believable it sounds.

A source can support some details while failing to establish others.
LabelRelationship to the Source noteExample detail
SupportedThe note establishes the detail.Entry is free.
ContradictedThe note establishes the opposite.The workshop does not start at noon.
Not establishedThe note gives no basis for the detail.20 seats are available.

Tip

Key point

A fluent answer may mix all three kinds of claims. Even when the relevant note remains in the current context, next-token prediction does not by itself check every generated statement against that note.

Practice: classify a claim

Classify the details

Source note

The workshop starts at noon. Entry is free.

Match each Model answer detail to its evidence label.

Tap an item and then its matching pair.

Explain why confidence is not evidence

Apply the comparison

Source note

The workshop starts at noon. Entry is free.

Model answer

“The workshop starts at noon. Entry is not free, and 20 seats are available.”

In 3–5 sentences, classify each distinct detail as Supported, Contradicted, or Not established. Then explain why the answer’s assertive wording does not resolve any evidence gap.

Write at least 120 characters (0/120).

Step 8 of 8

Explain a Model Answer

Apply tokenization, repeated next-token generation, sampling, context limits, and source comparison to explain a compact model-answer case.

Connect mechanism and evidence

A compact generation case

One compact case, four checks

Use this case to connect the ideas from the tutorial:

  1. Token boundaries: tokens are not necessarily words.
  2. Generation: each selected token extends the sequence for the next prediction.
  3. Available information: distinguish learned patterns from material supplied now.
  4. Factual support: compare each answer detail with the source, not with how fluent it sounds.

Illustrative token split; actual splits depend on the tokenizer

[Read][ing][ is][ fun][.]

The sentence Reading is fun. has 3 words but 5 tokens in this illustration. The leading spaces in [ is] and [ fun] belong to those illustrated tokens.

Starting sequence: Reading is

Generation cycle: Predict → Select → Append → Repeat

In iteration 1, [ fun] is selected and appended. In iteration 2, the longer sequence is used and [.] is selected.

How the pieces connect

A diagram connects five token blocks, a repeated next-token generation cycle, and a finite context window holding a library note and generated text.

A generated answer is built token by token from the sequence available in the current context.

Example

Compare details with the source

Maple Library comparison

Source note

The workshop starts at noon. Entry is free.

Model answer A

“The Maple Library workshop starts at noon. Entry is free. It has 20 seats.”

  • noon — Supported
  • free — Supported
  • 20 seats — Not established

Model answer B

“The workshop does not start at noon. Entry is free.”

  • “does not start at noon” — Contradicted
  • free — Supported

A source can support some details in an answer while failing to establish another. A detail that is Not established is not automatically proven false; the source simply does not support it.

Reason without overclaiming

Make careful claims

Match each observation to the most careful conclusion.

Tap an item and then its matching pair.

Summary

Synthesis checklist

  • A token split can divide a word, so token count and word count can differ.
  • Next-token prediction repeatedly selects and appends tokens; sampling can lead to different continuations.
  • Learned during training and Supplied in the current context are different information sources.
  • The context window is finite. Missing material and different token selections are distinct possible contributors to inconsistency.
  • Compare claims with source text: fluent, confident wording is not factual evidence.

Explain the full case

Final application: explain the answers

Write a 4–6 sentence explanation of this self-contained case. Address every requested point.

Tokenization

Reading is fun. has this split:

[Read][ing][ is][ fun][.]

Generation trace

Starting sequence: Reading is

Illustrative probabilities, not measured model output

Iteration 1: [ fun] 50%, [ great] 30%, [ useful] 20%. The selected token is [ fun].

Iteration 2, after appending the selected token: [.] 70%, [!] 30%. The selected token is [.].

Information available for this run

  • Learned during training: language patterns learned from many token sequences.
  • Supplied in the current context: the Maple Library source note below.
  • Outside the current context: earlier conversation material not supplied here.
  • Current context: the source note and this task, within the finite context window.
  • Generated text: the two model answers below.

Source note

The workshop starts at noon. Entry is free.

Model answer A

“The Maple Library workshop starts at noon. Entry is free. It has 20 seats.”

Model answer B

“The workshop does not start at noon. Entry is free.”

In your explanation: (1) state what the split shows about tokens versus words; (2) trace the two selections using Predict → Select → Append → Repeat; (3) distinguish learned information from supplied information; (4) give a careful, plausible explanation for varied answers without claiming that the output proves one cause; and (5) classify 20 seats and “does not start at noon” using Supported, Contradicted, or Not established, explaining why confidence is not evidence.

Write at least 300 characters (0/300).

Step complete

Congratulations! You completed: How Language Models Generate Text

You can now explain a model answer as token selections shaped by learned patterns and available context—not as a guarantee of factual truth.

Download the App now to have access to + 5000 free courses, exercises, certificates and lots of content without paying anything!

  • 100% free online courses from start to finish

    Thousands of online courses in video, ebooks and audiobooks.

  • More than 60 thousand free exercises

    To test your knowledge during online courses

  • Valid free Digital Certificate with QR Code

    Generated directly from your cell phone's photo gallery and sent to your email

Cursa app on the ebook screen, the video course screen and the course exercises screen, plus the course completion certificate