
Step 1 of 8
Give Each Dataset a Job
Assign training, validation, and test data to their distinct roles in the Northstar Support model-development example.
Learning path · Level 2 · Tutorial 2
Read small model-performance tables and judge whether apparently strong results provide credible evidence of useful performance on new cases.
Give Each Dataset a Job
Assign training, validation, and test data to their distinct roles in the Northstar Support model-development example. 2 min
Read What a Score Actually Says
Interpret an accuracy result, identify the evaluated dataset behind it, and make a cautious claim about performance on new cases. 2 min
Recognize an Overfitting Warning
Compare training and validation results to recognize a warning pattern consistent with overfitting, while keeping the conclusion appropriately cautious. 2 min
Spot Information That Should Not Be There
Recognize data leakage from overlapping records, exposed labels, and information unavailable at prediction time. 3 min
Keep the Final Check Independent
Learn how repeated test-guided decisions weaken the credibility of a final model score. 2 min
Look Behind High Accuracy
Look past a high overall accuracy by comparing category counts with a simple baseline on the same evaluation messages. 2 min
Ask Which Mistakes Matter
Compare missed urgent messages and false alarms, then make a conditional judgment based on Northstar Support's stated priority. 2 min
Make a Cautious Evaluation Judgment
Integrate dataset roles, independence, category counts, baselines, and error consequences into a cautious evaluation judgment. 3 min

Step 1 of 8
Assign training, validation, and test data to their distinct roles in the Northstar Support model-development example.
Northstar Support is building a model that classifies each incoming customer message as Urgent or Routine. Each reviewed message has:
Message text at arrival: an input available when the message arrivesConfirmed priority: the label established by reviewThe team separates its examples into three datasets because each dataset has a different job. This separation helps distinguish learning from checking how well a chosen model may work on cases it has not used during development.

Use training data to learn, validation data to make development choices, and test data for the final independent check.
At Northstar Support, the worked split contains:
1,000 training messages100 validation messages100 test messagesTraining data supplies the examples used to learn patterns. The model is fitted using the training messages and their Confirmed priority labels.
Validation data is held out of that learning process. It supports development choices, such as choosing which model version to keep.
Test data is also held out of learning. It is saved for the final independent check, after the development choices are settled.
Example
A simple development sequence:
Validation data and test data are both held-out data: they are kept out of the examples used to fit the model. But they are not interchangeable. Validation data helps guide development; test data checks the finished choice independently.
Tip
“Final” means final for this development cycle: do not use the test data as another source of choices while deciding which version to build. The next steps will examine why this distinction matters.
Match each Northstar Support activity to the dataset with the correct job.
Tap an item and then its matching pair.

Step 2 of 8
Interpret an accuracy result, identify the evaluated dataset behind it, and make a cautious claim about performance on new cases.
Accuracy is the share of predictions that match the known labels.
For Northstar Support, compare each predicted priority with its Confirmed priority. If Model B makes 96 correct predictions among 100 validation messages, its accuracy is:
96 correct out of 100 = 96%
Accuracy summarizes agreement on the examples that were evaluated. It does not describe every message the model could receive.

Accuracy compares correct predictions with all evaluated messages.
Read a result table in context, not as a percentage alone. First identify the dataset, then the number of examples, then the number of correct predictions.
This is a development snapshot. Model B was evaluated on 100 validation messages. Its 96% validation accuracy means 96 correct out of 100 = 96%.
Its 97% training accuracy describes performance on the 1,000 training messages used while learning. It is not, by itself, evidence of performance on new messages.
Model B results during development. The test result is intentionally not read until development choices are settled.
| Model | Training accuracy | Validation accuracy | Test accuracy |
|---|---|---|---|
| Model B | 97% | 96% | Not yet evaluated |
Tip
When reading an accuracy claim, ask three quick questions:
A percentage without these details has limited meaning.
Generalization means useful performance on relevant new examples that were not used to develop the model.
A validation result gives development feedback on held-out messages. After choices are settled, an untouched and relevant test set provides a more independent final check.
Training accuracy is about familiar examples. Validation and test results are more useful evidence about new cases, but a score still describes only the evaluated examples and conditions. It does not guarantee the same result for every future message.

Held-out results offer evidence about relevant unseen messages; a final test check remains separate from development.
Example
Suppose next month's messages use new wording, describe different issues, or come from a different customer mix. A 96% validation result does not promise that the same percentage will appear then.
A careful claim is: “Model B achieved 96% validation accuracy on these 100 validation messages.”
Northstar Support evaluated Model B on 100 validation messages and found 96 correct predictions. Which conclusion is justified?

Step 3 of 8
Compare training and validation results to recognize a warning pattern consistent with overfitting, while keeping the conclusion appropriately cautious.
Overfitting is a pattern in which a model learns details that work very well on its training data but do not transfer well to relevant new messages.
A warning sign is a large gap: training accuracy is strong, while validation accuracy is much weaker. The highest training score alone is not the best evidence of generalization.

A large distance between training and validation results is a warning pattern; closely aligned results provide more reassuring evidence of transfer.
Tip
A gap is not proof of one specific cause. For example, the validation messages may differ in difficulty or relevance from the training messages. Treat the pattern as a reason to investigate, not as a final diagnosis.
Compare each model across the two datasets. The training result shows performance on familiar examples; the validation result is the more useful check here for relevant new messages.
Model A falls from 99% on training data to 80% on validation data: a 19-point gap. Model B falls from 97% to 96%: a 1-point gap.
Model A has the higher training accuracy, but Model B has much stronger validation evidence. Therefore, Model A shows the stronger warning pattern consistent with overfitting.
| Model | Training accuracy | Validation accuracy |
|---|---|---|
| Model A | 99% | 80% |
| Model B | 97% | 96% |
Which conclusion is best supported by the comparison?
Using the comparison you just studied, identify the model with the stronger overfitting warning. Explain why its higher training accuracy does not outweigh its validation result.
Write at least 100 characters (0/100).

Step 4 of 8
Recognize data leakage from overlapping records, exposed labels, and information unavailable at prediction time.
Data leakage happens when an evaluation gives a model information it would not legitimately have when making a prediction for a new arriving message. The resulting score can look stronger than real-world performance.
At Northstar Support, the model should use Message text at arrival to predict Urgent or Routine. Confirmed priority is the answer established by review. Resolution time exists only after the case is resolved.

Only information available when the message arrives belongs in the prediction input. The reviewed label is used afterward to score the prediction.
Leakage can enter through the data split or through an input feature. In each case, the model receives a shortcut that would not be available for a genuinely new message.
Each safeguard removes an unfair source of information before evaluating the model.
| Mechanism | Unfair advantage | Basic safeguard |
|---|---|---|
| Related messages cross splits | A near-duplicate reveals patterns from the same conversation | Keep a conversation's related records together |
| Confirmed priority is an input | The input exposes the answer to predict | Use it only after prediction for scoring |
| Resolution time is an input | It reveals information from after resolution | Use only information available at arrival |
Example
A legitimate workflow is: give the model Message text at arrival; let it predict a priority; then compare that prediction with Confirmed priority. The label is necessary for scoring. It becomes leakage only if it is supplied to the model as an input, or otherwise influences development in an inappropriate way.
Match each situation to the issue that makes its evaluation misleading.
Tap an item and then its matching pair.
Northstar Support has three evaluation problems:
For each problem, name the leakage mechanism and explain in one sentence how it gives an unfair advantage. Then explain why using Confirmed priority after prediction to score the model is legitimate, while using it as an input is not.
Write at least 300 characters (0/300).

Step 5 of 8
Learn how repeated test-guided decisions weaken the credibility of a final model score.
A test set is not for choosing what to change. Its job is to provide one final check after development choices are settled.
For Northstar Support, use training data to learn from 1,000 messages. Use validation data to compare Model A and Model B or revise a model. Only then evaluate the selected model on the separate test data.
If you repeatedly look at test results, change the model, and look again, the test results start influencing development. The test set is then no longer an independent final check—even though its messages were never used directly to train the model.

Keep the test data out of development decisions until the final check.
Example
Consider this development loop:
The later score can look better partly because the team adapted to this particular test set. It is not as credible as a score from test data that had no role in those choices.
It is normal to read a final test result. The problem is using that result to decide what to revise, select, or try next.
If test data has already guided development, state that limitation honestly. To support a new final claim, use a fresh independent test set that was not used for those decisions.
True or false: A test set remains independent as long as its examples are never directly used to train the model.
Which sequence both keeps the final check independent and puts the evaluation activities in the correct order?

Step 6 of 8
Look past a high overall accuracy by comparing category counts with a simple baseline on the same evaluation messages.
A single accuracy score combines every message in the evaluation set. When one category is much more common, correct predictions for that common category can dominate the total.
At Northstar Support, a held-out set contains 95 Routine messages and 5 Urgent messages. That means a model can look strong overall while doing little—or nothing—for the less common Urgent messages.
To read the result carefully, look behind the total: count correct predictions and errors separately for each actual category.

Most messages are Routine, so their results can strongly shape the overall accuracy.
Use a simple comparison model, called a baseline, to ask whether a model adds value. The baseline here is Always Routine: it predicts Routine for every incoming message.
On these same 100 validation messages, it gets every Routine message right and every Urgent message wrong. Its total can therefore sound impressive even though it never identifies an Urgent message.
Results on the same validation messages and the same Urgent-or-Routine task:
| Actual category | Total messages | Correct predictions | Errors |
|---|---|---|---|
| Routine | 95 | 95 | 0 |
| Urgent | 5 | 0 | 5 missed urgent messages |
| Routine | 95 | 92 | 3 false alarms |
| Urgent | 5 | 4 | 1 missed urgent message |
Example
The first two rows describe Always Routine. The last two rows describe Model B.
Always Routine: 95 correct out of 100 = 95%.
Model B: 4 correct Urgent predictions plus 92 correct Routine predictions gives 96 correct out of 100 = 96%.
The comparison is meaningful because both models are judged on the same messages and the same task.
What does the 95% accuracy of Always Routine fail to show?
Choose the best answer.
Write at least 70 characters (0/70).
Northstar Support evaluates 100 messages: 95 are Routine and 5 are Urgent. The Always Routine baseline predicts Routine for every message.
Calculate its accuracy, state how many missed urgent messages it makes, and explain in one or two sentences why overall accuracy alone hides this failure.
Write at least 100 characters (0/100).

Step 7 of 8
Compare missed urgent messages and false alarms, then make a conditional judgment based on Northstar Support's stated priority.
Northstar Support makes two different kinds of errors when it classifies an incoming message.
Neither label is a new message category. They describe what went wrong in a prediction.

A missed urgent message can delay attention; a false alarm can consume priority-review time.
The validation results show a tradeoff, not just a total number of errors. Model B finds more Urgent messages than Always Routine, but it also sends some Routine messages to the priority queue.
An error count alone does not tell you which operational burden occurred. Two models could each make three errors, yet one could miss three urgent requests while the other creates three false alarms. The consequences determine whether those results are equally acceptable.
These category counts come from the same 100 validation messages: 95 Routine and 5 Urgent.
| Actual category | Total messages | Correct predictions | Errors |
|---|---|---|---|
| Model B: Urgent | 5 | 4 | 1 missed urgent message |
| Model B: Routine | 95 | 92 | 3 false alarms |
| Always Routine: Urgent | 5 | 0 | 5 missed urgent messages |
| Always Routine: Routine | 95 | 95 | 0 false alarms |
Match each term to its meaning or consequence.
Tap an item and then its matching pair.
On the validation data, Model B has 1 missed urgent message and 3 false alarms. Always Routine has 5 missed urgent messages and 0 false alarms.
Northstar Support says that avoiding missed urgent messages matters more than reviewing extra Routine messages. In 2–4 sentences, state the consequence of each error type, then explain why Model B may be more useful under that priority. Also acknowledge Model B's additional false alarms.
Write at least 180 characters (0/180).

Step 8 of 8
Integrate dataset roles, independence, category counts, baselines, and error consequences into a cautious evaluation judgment.
Northstar Support used 1,000 training messages to learn patterns. It compared model versions using 100 validation messages, then selected Model B. Only after that choice was settled, the team evaluated Model B once on a separate 100-message test set.
The test set was relevant to incoming support messages, had no overlapping conversations with development data, and used only Message text at arrival as an input. Confirmed priority was used only to score predictions. Resolution time was not used because it is known only after a case is resolved.
This setup makes the test result credible evidence about these held-out messages. It is still not a guarantee of performance on every future message.
Model B was chosen using validation data. Its test result was read after development decisions were complete.
| Model | Training accuracy | Validation accuracy | Test accuracy |
|---|---|---|---|
| Model A | 99% | 80% | Not evaluated |
| Model B | 97% | 96% | 94% |
The three flows represent separate jobs: learning, development choices, and the final independent check.

The test messages remain separate until the final check.
On the test set, Model B made 94 correct predictions out of 100 = 94%. Its category counts matter:
The baseline, Always Routine, made 95 correct predictions out of 100 = 95% on these same test messages. But it missed all 5 Urgent messages. A higher overall accuracy alone does not decide which approach is more useful.
Model B's test-category results.
| Actual category | Total messages | Correct predictions | Errors |
|---|---|---|---|
| Urgent | 5 | 4 | 1 missed urgent message |
| Routine | 95 | 90 | 5 false alarms |
A report says: “Model B was selected using validation data and evaluated once on separate test data. The model used Message text at arrival and Resolution time as inputs, achieving 98% test accuracy.”
What is the strongest reason to reject this credibility claim?
Write a short evaluation judgment for the Northstar Support report. Include all of these points:
State what the evidence supports, but do not claim that the model will perform the same way on every future message.
Write at least 300 characters (0/300).
Summary
Use this checklist whenever a model result looks impressive:
A careful conclusion separates two questions: Is the evaluation credible? And is the observed error tradeoff useful for the intended work?
Congratulations! You completed: Reading Model Evaluation Results
Thousands of online courses in video, ebooks and audiobooks.
To test your knowledge during online courses
Generated directly from your cell phone's photo gallery and sent to your email
Download our app via QR Code or the links below:.
+ 10 million
students
Free and Valid
Certificate
60 thousand free
exercises
4.8/5 rating in
app stores
Free courses in
video and ebooks