
Step 1 of 8
Text Becomes Tokens
Understand tokens as text units and see why a token count can differ from a word count.
Learning path · Level 2 · Tutorial 4
Explain how language models produce text and use that mechanism to account for context limits, varied answers, and confident factual errors.
Text Becomes Tokens
Understand tokens as text units and see why a token count can differ from a word count. 2 min
Learning to Predict What Comes Next
See how token sequences become training examples for next-token prediction, and why learned weights are patterns rather than a searchable store of ready-made responses. 2 min
Generating One Token at a Time
Trace how a language model repeatedly predicts, selects, and appends one token at a time. 3 min
Why Answers Can Vary
See how sampling among plausible next tokens can produce different continuations from the same starting sequence. 2 min
Learned Patterns and Current Context
Distinguish patterns learned during training from information supplied in a current context, including the finite context window. 2 min
When Earlier Information Is Missing
See how a context boundary can remove earlier information and why that differs from sampling-driven variation. 2 min
Why Fluent Answers Can Be Wrong
2 min
Explain a Model Answer
Apply tokenization, repeated next-token generation, sampling, context limits, and source comparison to explain a compact model-answer case. 2 min

Step 1 of 8
Understand tokens as text units and see why a token count can differ from a word count.
Before a language model can work with text, it represents the text as tokens. A token is a unit of text. It may be a whole word, a word fragment, punctuation, or text that includes a space.
The brackets in this lesson show token boundaries. They are an illustration of how one sentence can be divided; the brackets are not part of the sentence itself.

Illustrative token split; actual splits depend on the tokenizer
Example
Here is the illustrated split:
[Read][ing][ is][ fun][.]
The visible word “Reading” is divided into [Read] and [ing], so a token boundary does not have to match a word boundary. [ fun] is a whole-word token in this illustration, and its leading space is part of that token. [.] is a punctuation token.
The sentence has 3 words: Reading, is, fun. It has 5 tokens in this illustrated split. Counting words therefore does not necessarily give the token count.
This is not a universal counting rule. Actual token splits depend on the tokenizer: a system that divides text into tokens. Another tokenizer may divide the same sentence differently.
The useful idea is not to memorize this particular count. Instead, notice that tokens can cross the boundaries we normally use when counting words.
Tip
When you see brackets such as [ is], check whether a space appears inside the brackets. In an illustrative split, that space belongs to the token.
Match each token from the illustrated split to its type.
Refer to the illustrative split from the earlier screen. Identify one whole-word token, one word-fragment token, and the punctuation token. Then explain why this sentence has 3 words but 5 tokens in the illustration.
Write at least 60 characters (0/60).

Step 2 of 8
See how token sequences become training examples for next-token prediction, and why learned weights are patterns rather than a searchable store of ready-made responses.
Training uses sequences of tokens to create many small learning tasks.
Consider this illustrative split:
Reading is fun.[Read][ing][ is][ fun][.]
Illustrative token split; actual splits depend on the tokenizer
For a task called next-token prediction, the model receives a preceding sequence and the following token is the target. For example, after [Read][ing][ is], the target is [ fun].
During training, the model compares its prediction with that target and adjusts its weights to become more useful at predicting what follows.

A preceding sequence is the input; the following token provides the target for next-token prediction.
Example
One token sequence can create overlapping training examples by shifting the target one position at a time:
[Read] → target: [ing][Read][ing] → target: [ is][Read][ing][ is] → target: [ fun]Across many examples, training develops patterns about wording, grammar, and associations across text. The goal is not to memorize one fixed reply for each sequence, but to make predictions that fit patterns seen across many sequences.
Warning
The learned patterns are encoded in the model's weights—the adjustable settings changed during training. Weights are not a dependable searchable archive of the exact training examples.
A model may produce wording that resembles a pattern it learned, but that does not mean it can reliably look up, quote, or prove a particular training sentence.
Consider this illustrative training sequence: [The][ door][ opens].
Which option correctly identifies one training case and what the model learns from many such cases?

Step 3 of 8
Trace how a language model repeatedly predicts, selects, and appends one token at a time.
Generation begins with the tokens available so far. For the starting sequence Reading is, the model considers possible next tokens and assigns each an illustrative probability. It does not need to retrieve a complete, finished sentence first.
The generation cycle is: Predict → Select → Append → Repeat. A selected token becomes part of the available sequence before the next prediction.
![A diagram showing the available sequence [Read][ing][ is], possible next-token candidates, one selected token, and an extended sequence.](https://cursa-json-routes.s3.amazonaws.com/learning-paths/en/artificial-intelligence-from-zero-to-advanced-use/0009c8dc3836/level-02/tutorial-04/step-03/screen-01/item-02.webp)
Predict → Select → Append → Repeat: the selected token is appended to [Read][ing][ is] and becomes available for the next prediction.
Illustrative probabilities, not measured model output
| Candidate next token | Probability |
|---|---|
| [ fun] | 50% |
| [ a] | 30% |
| [ not] | 20% |
Example
In this worked example, [ fun] is selected first and appended. The model then makes a new prediction using the longer sequence. Next, [.] is selected and appended.
Each new prediction uses the sequence built so far. Later possibilities and probabilities can therefore differ from earlier ones.
The result appended in one iteration is the available sequence for the next iteration.
| Iteration | Predict using | Select | Append to form |
|---|---|---|---|
| 1 | [Read][ing][ is] | [ fun] | [Read][ing][ is][ fun] |
| 2 | [Read][ing][ is][ fun] | [.] | [Read][ing][ is][ fun][.] |
This walkthrough stops after the period. More generally, repeated cycles continue until a stopping condition is reached. The output is constructed token by token; it is not necessarily a finished answer retrieved all at once.
Put the operations from the worked example in their correct order. Include the updated sequence after each selected token.

Step 4 of 8
See how sampling among plausible next tokens can produce different continuations from the same starting sequence.
For the same available sequence, several next tokens may be plausible. The model can assign a probability to each candidate rather than finding one required continuation.
A higher probability means a candidate is more likely under the model’s learned patterns. It does not mean that every other candidate is impossible.

One available sequence can lead to several candidate next tokens. Branch width represents relative likelihood, not a guaranteed choice.
Consider the starting sequence: I would like.
Illustrative probabilities, not measured model output
The probabilities below form a complete illustrative distribution: together, they total 100%.
Possible next tokens after I would like
| Candidate token | Probability |
|---|---|
| `[ tea]` | 50% |
| `[ coffee]` | 30% |
| `[ water]` | 20% |
Example
Sampling selects among candidate tokens using their probabilities. It can select [ coffee] or [ water] even though [ tea] has the largest probability.
So, “highest probability” does not mean “must be chosen.”
Example
Suppose two runs begin with exactly the same starting sequence, I would like.
Run A: [ tea] is selected, then later [ with lemon], then [.].
Run B: [ coffee] is selected, then later [.].
After the first selection, each run has a different sequence. The next-token prediction is therefore based on a different sequence, so later continuations can diverge further.
Variation is possible, not guaranteed. With the same full starting sequence, separate runs can select different plausible tokens through sampling. They can also select the same token and produce the same result.
Two runs begin with exactly the same sequence but continue differently. Which explanation is justified?

Step 5 of 8
Distinguish patterns learned during training from information supplied in a current context, including the finite context window.
A language model can draw on two different sources while generating text:
The two sources work together, but they are not the same thing. Training happened before the current request; context is available for the current generation.

Information learned during training and information supplied in the current context are distinct inputs to generation.
Suppose a request includes a short note about Maple Library. That note is Supplied in the current context, so the model can use its details while answering the request.
Sending the note does not, by itself, change the model weights. In other words, it does not immediately add a new learned fact to the model through training.
Example
The workshop starts at noon. Entry is free.
The note is available for this answer because it was supplied with the request.
| Source note | Model answer |
|---|---|
| The workshop starts at noon. Entry is free. | The Maple Library workshop starts at noon, and entry is free. |
Tip
A current message can make information available for one generation without creating an immediate weight update.
The context window is the finite span of tokens available for a generation. It can contain instructions, supplied text, conversation messages, and generated tokens so far.
Both the input and the developing output occupy space in this window. Its exact capacity depends on the model and product, so there is no universal token limit to memorize.

The current context is a finite token span: supplied input and generated text share its available space.
Match each description to its correct classification.
Tap an item and then its matching pair.

Step 6 of 8
See how a context boundary can remove earlier information and why that differs from sampling-driven variation.
A long conversation can contain more text than the model receives for one response. Its context window is the finite token span available at that moment—not necessarily the complete conversation history.
In this example, an earlier message claims that Maple Library has 20 seats. That message is now Outside the current context. The available note in the Current context says: The workshop starts at noon. Entry is free.
The earlier seat claim is not directly available as current-context information, even though the model might still produce related wording from learned patterns or an inference.

The faded area is Outside the current context. The outlined area is the Current context, containing The workshop starts at noon. Entry is free. The speech bubble is Generated text.
Products can handle a long history in different ways. They may omit earlier material, replace it with a summary, or require shorter input. Do not assume every product keeps or removes messages in the same way.
When an earlier detail is not supplied in the current context, it cannot directly guide the next-token prediction for that response. This is different from sampling: sampling can produce different continuations even when the available starting context is unchanged.
An output by itself does not reveal which mechanism caused a difference.
| Question | Unavailable context | Sampling |
|---|---|---|
| What changed? | An earlier detail is no longer available. | A next token can be selected differently. |
| Starting context | May be missing relevant earlier information. | Can be unchanged across runs. |
| What can output alone prove? | It cannot prove this was the cause. | It cannot prove this was the cause. |
Two answers in a long conversation disagree about a library detail. What is the most justified conclusion from the answers alone?
Refer to the context-boundary diagram from the first screen. In 2–4 sentences, identify which Maple Library details remain in the Current context, identify the earlier detail that is outside it, and explain how that absence differs from variation caused by sampling.
Write at least 100 characters (0/100).

Step 7 of 8
The workshop starts at noon. Entry is free.
“The workshop starts at noon. Entry is free. The room has 20 seats.”
A model generates a plausible continuation from learned patterns and the current context. Plausible wording is not automatically a verified factual statement.
Here, the time and admission details are supported by the note. The seat count is not supplied by it. A hallucination is a generated detail that is incorrect or unsupported. It describes an output failure, not an intention to deceive.

The extra seat detail can sound natural even though the supplied note does not establish it.
Use the source itself to distinguish three kinds of details. A claim that is Not established is not automatically false: the source simply does not provide enough evidence for it.
Confidence, polished phrasing, or a high probability for a token do not change these labels. They describe how likely a continuation may be, not whether a source proves it.
Classify a detail by its relationship to the Source note, not by how believable it sounds.
| Label | Relationship to the Source note | Example detail |
|---|---|---|
| Supported | The note establishes the detail. | Entry is free. |
| Contradicted | The note establishes the opposite. | The workshop does not start at noon. |
| Not established | The note gives no basis for the detail. | 20 seats are available. |
Tip
A fluent answer may mix all three kinds of claims. Even when the relevant note remains in the current context, next-token prediction does not by itself check every generated statement against that note.
The workshop starts at noon. Entry is free.
Match each Model answer detail to its evidence label.
Tap an item and then its matching pair.
The workshop starts at noon. Entry is free.
“The workshop starts at noon. Entry is not free, and 20 seats are available.”
In 3–5 sentences, classify each distinct detail as Supported, Contradicted, or Not established. Then explain why the answer’s assertive wording does not resolve any evidence gap.
Write at least 120 characters (0/120).

Step 8 of 8
Apply tokenization, repeated next-token generation, sampling, context limits, and source comparison to explain a compact model-answer case.
Use this case to connect the ideas from the tutorial:
Illustrative token split; actual splits depend on the tokenizer
[Read][ing][ is][ fun][.]
The sentence Reading is fun. has 3 words but 5 tokens in this illustration. The leading spaces in [ is] and [ fun] belong to those illustrated tokens.
Starting sequence: Reading is
Generation cycle: Predict → Select → Append → Repeat
In iteration 1, [ fun] is selected and appended. In iteration 2, the longer sequence is used and [.] is selected.

A generated answer is built token by token from the sequence available in the current context.
Example
Source note
The workshop starts at noon. Entry is free.
Model answer A
“The Maple Library workshop starts at noon. Entry is free. It has 20 seats.”
noon — Supportedfree — Supported20 seats — Not establishedModel answer B
“The workshop does not start at noon. Entry is free.”
free — SupportedA source can support some details in an answer while failing to establish another. A detail that is Not established is not automatically proven false; the source simply does not support it.
Match each observation to the most careful conclusion.
Tap an item and then its matching pair.
Summary
Write a 4–6 sentence explanation of this self-contained case. Address every requested point.
Reading is fun. has this split:
[Read][ing][ is][ fun][.]
Starting sequence: Reading is
Illustrative probabilities, not measured model output
Iteration 1: [ fun] 50%, [ great] 30%, [ useful] 20%. The selected token is [ fun].
Iteration 2, after appending the selected token: [.] 70%, [!] 30%. The selected token is [.].
The workshop starts at noon. Entry is free.
“The Maple Library workshop starts at noon. Entry is free. It has 20 seats.”
“The workshop does not start at noon. Entry is free.”
In your explanation: (1) state what the split shows about tokens versus words; (2) trace the two selections using Predict → Select → Append → Repeat; (3) distinguish learned information from supplied information; (4) give a careful, plausible explanation for varied answers without claiming that the output proves one cause; and (5) classify 20 seats and “does not start at noon” using Supported, Contradicted, or Not established, explaining why confidence is not evidence.
Write at least 300 characters (0/300).
Congratulations! You completed: How Language Models Generate Text
Thousands of online courses in video, ebooks and audiobooks.
To test your knowledge during online courses
Generated directly from your cell phone's photo gallery and sent to your email
Download our app via QR Code or the links below:.
+ 10 million
students
Free and Valid
Certificate
60 thousand free
exercises
4.8/5 rating in
app stores
Free courses in
video and ebooks