Learning path · Level 2 · Tutorial 2

Reading Model Evaluation Results

Read small model-performance tables and judge whether apparently strong results provide credible evidence of useful performance on new cases.

  • Level: Beginner
  • Duration: 18 min
  • 8 steps
Start tutorial Back to the path
Reading Model Evaluation Results

What you will go through

  1. Give Each Dataset a Job Assign training, validation, and test data to their distinct roles in the Northstar Support model-development example. 2 min
  2. Read What a Score Actually Says Interpret an accuracy result, identify the evaluated dataset behind it, and make a cautious claim about performance on new cases. 2 min
  3. Recognize an Overfitting Warning Compare training and validation results to recognize a warning pattern consistent with overfitting, while keeping the conclusion appropriately cautious. 2 min
  4. Spot Information That Should Not Be There Recognize data leakage from overlapping records, exposed labels, and information unavailable at prediction time. 3 min
  5. Keep the Final Check Independent Learn how repeated test-guided decisions weaken the credibility of a final model score. 2 min
  6. Look Behind High Accuracy Look past a high overall accuracy by comparing category counts with a simple baseline on the same evaluation messages. 2 min
  7. Ask Which Mistakes Matter Compare missed urgent messages and false alarms, then make a conditional judgment based on Northstar Support's stated priority. 2 min
  8. Make a Cautious Evaluation Judgment Integrate dataset roles, independence, category counts, baselines, and error consequences into a cautious evaluation judgment. 3 min

What you will learn

  • Assign training, validation, and test data their correct roles in a worked example.
  • Identify a pattern consistent with overfitting by comparing training and held-out results.
  • Spot leakage in a described data split or input feature and explain why it makes results misleading.
  • Use category counts and a simple baseline to explain why high overall accuracy can conceal an unhelpful model.
  • Explain why the consequences of errors matter when judging model performance.

Before you start

  • How Models Learn from Data

Step 1 of 8

Give Each Dataset a Job

Assign training, validation, and test data to their distinct roles in the Northstar Support model-development example.

Separate the jobs

One task, three dataset roles

Northstar Support is building a model that classifies each incoming customer message as Urgent or Routine. Each reviewed message has:

  • Message text at arrival: an input available when the message arrives
  • Confirmed priority: the label established by review

The team separates its examples into three datasets because each dataset has a different job. This separation helps distinguish learning from checking how well a chosen model may work on cases it has not used during development.

The three-part data flow

A diagram shows messages used first for learning, then for choosing between model versions, and finally for an independent check.

Use training data to learn, validation data to make development choices, and test data for the final independent check.

Follow the development sequence

What each dataset does

At Northstar Support, the worked split contains:

  • 1,000 training messages
  • 100 validation messages
  • 100 test messages

Training data supplies the examples used to learn patterns. The model is fitted using the training messages and their Confirmed priority labels.

Validation data is held out of that learning process. It supports development choices, such as choosing which model version to keep.

Test data is also held out of learning. It is saved for the final independent check, after the development choices are settled.

Example

Worked example: Northstar Support

A simple development sequence:

  1. Use the 1,000 training messages to learn a classification pattern.
  2. Use the 100 validation messages to compare candidate model versions and make a development choice.
  3. Once that choice is settled, use the separate 100 test messages for the final independent check.

Validation data and test data are both held-out data: they are kept out of the examples used to fit the model. But they are not interchangeable. Validation data helps guide development; test data checks the finished choice independently.

Tip

Keep the roles distinct

“Final” means final for this development cycle: do not use the test data as another source of choices while deciding which version to build. The next steps will examine why this distinction matters.

Practice: give each dataset a job

Assign each dataset

Match each Northstar Support activity to the dataset with the correct job.

Tap an item and then its matching pair.

Step 2 of 8

Read What a Score Actually Says

Interpret an accuracy result, identify the evaluated dataset behind it, and make a cautious claim about performance on new cases.

Turn correct predictions into accuracy

Accuracy: a count turned into a percentage

Accuracy is the share of predictions that match the known labels.

For Northstar Support, compare each predicted priority with its Confirmed priority. If Model B makes 96 correct predictions among 100 validation messages, its accuracy is:

96 correct out of 100 = 96%

Accuracy summarizes agreement on the examples that were evaluated. It does not describe every message the model could receive.

A grid of 100 message cards, with 96 matching predictions and 4 mismatches.

Accuracy compares correct predictions with all evaluated messages.

Read a small result table

Read the dataset before judging the score

Read a result table in context, not as a percentage alone. First identify the dataset, then the number of examples, then the number of correct predictions.

This is a development snapshot. Model B was evaluated on 100 validation messages. Its 96% validation accuracy means 96 correct out of 100 = 96%.

Its 97% training accuracy describes performance on the 1,000 training messages used while learning. It is not, by itself, evidence of performance on new messages.

Model B: development snapshot

Model B results during development. The test result is intentionally not read until development choices are settled.

Performance results must be connected to the dataset on which they were measured.
ModelTraining accuracyValidation accuracyTest accuracy
Model B97%96%Not yet evaluated

Tip

When reading an accuracy claim, ask three quick questions:

  • Which dataset produced this score?
  • How many messages were evaluated?
  • How many predictions matched Confirmed priority?

A percentage without these details has limited meaning.

Connect scores to new cases carefully

What a held-out score can support

Generalization means useful performance on relevant new examples that were not used to develop the model.

A validation result gives development feedback on held-out messages. After choices are settled, an untouched and relevant test set provides a more independent final check.

Training accuracy is about familiar examples. Validation and test results are more useful evidence about new cases, but a score still describes only the evaluated examples and conditions. It does not guarantee the same result for every future message.

A model learns from familiar message cards and is checked on separate unseen message groups.

Held-out results offer evidence about relevant unseen messages; a final test check remains separate from development.

Example

Keep the claim within its evidence

Suppose next month's messages use new wording, describe different issues, or come from a different customer mix. A 96% validation result does not promise that the same percentage will appear then.

A careful claim is: “Model B achieved 96% validation accuracy on these 100 validation messages.”

Check your interpretation

Interpret the result cautiously

Northstar Support evaluated Model B on 100 validation messages and found 96 correct predictions. Which conclusion is justified?

Step 3 of 8

Recognize an Overfitting Warning

Compare training and validation results to recognize a warning pattern consistent with overfitting, while keeping the conclusion appropriately cautious.

What an overfitting warning looks like

A gap can be a warning

Overfitting is a pattern in which a model learns details that work very well on its training data but do not transfer well to relevant new messages.

A warning sign is a large gap: training accuracy is strong, while validation accuracy is much weaker. The highest training score alone is not the best evidence of generalization.

Compare the two patterns

Two pairs of performance bars: one pair has a large difference between training and validation results, while the other pair is nearly equal.

A large distance between training and validation results is a warning pattern; closely aligned results provide more reassuring evidence of transfer.

Tip

Use cautious language

A gap is not proof of one specific cause. For example, the validation messages may differ in difficulty or relevance from the training messages. Treat the pattern as a reason to investigate, not as a final diagnosis.

Compare training with validation

Northstar Support results

Compare each model across the two datasets. The training result shows performance on familiar examples; the validation result is the more useful check here for relevant new messages.

Read across, then compare

Model A falls from 99% on training data to 80% on validation data: a 19-point gap. Model B falls from 97% to 96%: a 1-point gap.

Model A has the higher training accuracy, but Model B has much stronger validation evidence. Therefore, Model A shows the stronger warning pattern consistent with overfitting.

Training and validation accuracy for two message-classification models.
ModelTraining accuracyValidation accuracy
Model A99%80%
Model B97%96%

Practice a cautious conclusion

Check the warning pattern

Which conclusion is best supported by the comparison?

Apply the comparison

Make the evaluation judgment

Using the comparison you just studied, identify the model with the stronger overfitting warning. Explain why its higher training accuracy does not outweigh its validation result.

Write at least 100 characters (0/100).

Step 4 of 8

Spot Information That Should Not Be There

Recognize data leakage from overlapping records, exposed labels, and information unavailable at prediction time.

Use only legitimate prediction-time information

What makes leakage misleading?

Data leakage happens when an evaluation gives a model information it would not legitimately have when making a prediction for a new arriving message. The resulting score can look stronger than real-world performance.

At Northstar Support, the model should use Message text at arrival to predict Urgent or Routine. Confirmed priority is the answer established by review. Resolution time exists only after the case is resolved.

Diagram showing an arriving message flowing to a model, while a reviewed answer and later outcome information are blocked.

Only information available when the message arrives belongs in the prediction input. The reviewed label is used afterward to score the prediction.

Separate answers, future information, and related records

Three common shortcuts

Leakage can enter through the data split or through an input feature. In each case, the model receives a shortcut that would not be available for a genuinely new message.

Each safeguard removes an unfair source of information before evaluating the model.

Leakage mechanisms and basic safeguards
MechanismUnfair advantageBasic safeguard
Related messages cross splitsA near-duplicate reveals patterns from the same conversationKeep a conversation's related records together
Confirmed priority is an inputThe input exposes the answer to predictUse it only after prediction for scoring
Resolution time is an inputIt reveals information from after resolutionUse only information available at arrival

Example

Labels have a valid job

A legitimate workflow is: give the model Message text at arrival; let it predict a priority; then compare that prediction with Confirmed priority. The label is necessary for scoring. It becomes leakage only if it is supplied to the model as an input, or otherwise influences development in an inappropriate way.

Quick check: find the shortcut

Identify the leakage route

Match each situation to the issue that makes its evaluation misleading.

Tap an item and then its matching pair.

Apply the distinction

Explain the evaluation flaws

Northstar Support has three evaluation problems:

  1. Training data contains one message from a customer conversation, while validation data contains a nearly identical follow-up from that conversation.
  2. The model input includes Confirmed priority.
  3. The model input includes Resolution time when each message arrives.

For each problem, name the leakage mechanism and explain in one sentence how it gives an unfair advantage. Then explain why using Confirmed priority after prediction to score the model is legitimate, while using it as an input is not.

Write at least 300 characters (0/300).

Step 5 of 8

Keep the Final Check Independent

Learn how repeated test-guided decisions weaken the credibility of a final model score.

Protect the role of test data

One final check, not a feedback loop

A test set is not for choosing what to change. Its job is to provide one final check after development choices are settled.

For Northstar Support, use training data to learn from 1,000 messages. Use validation data to compare Model A and Model B or revise a model. Only then evaluate the selected model on the separate test data.

If you repeatedly look at test results, change the model, and look again, the test results start influencing development. The test set is then no longer an independent final check—even though its messages were never used directly to train the model.

Protect the test data

A three-stage flow: learning from training messages, choosing with validation messages, then one locked final check with test messages.

Keep the test data out of development decisions until the final check.

Independence can be lost without training on the test set

Example

When the test becomes feedback

Consider this development loop:

  1. The team trains a model using training data.
  2. It checks the test result.
  3. It changes the model because of that result.
  4. It checks the same test data again.

The later score can look better partly because the team adapted to this particular test set. It is not as credible as a score from test data that had no role in those choices.

What to do after test-guided changes

It is normal to read a final test result. The problem is using that result to decide what to revise, select, or try next.

If test data has already guided development, state that limitation honestly. To support a new final claim, use a fresh independent test set that was not used for those decisions.

Check the independence claim

Check the boundary

True or false: A test set remains independent as long as its examples are never directly used to train the model.

Apply the evaluation sequence

Choose the clean evaluation sequence

Which sequence both keeps the final check independent and puts the evaluation activities in the correct order?

Step 6 of 8

Look Behind High Accuracy

Look past a high overall accuracy by comparing category counts with a simple baseline on the same evaluation messages.

See what the total combines

A high total can hide a gap

A single accuracy score combines every message in the evaluation set. When one category is much more common, correct predictions for that common category can dominate the total.

At Northstar Support, a held-out set contains 95 Routine messages and 5 Urgent messages. That means a model can look strong overall while doing little—or nothing—for the less common Urgent messages.

To read the result carefully, look behind the total: count correct predictions and errors separately for each actual category.

See the category imbalance

A large group of Routine message cards and a much smaller group of Urgent message cards, with an accuracy gauge that visually emphasizes the larger group.

Most messages are Routine, so their results can strongly shape the overall accuracy.

Compare the baseline with Model B

A baseline sets a minimum comparison

Use a simple comparison model, called a baseline, to ask whether a model adds value. The baseline here is Always Routine: it predicts Routine for every incoming message.

On these same 100 validation messages, it gets every Routine message right and every Urgent message wrong. Its total can therefore sound impressive even though it never identifies an Urgent message.

Category breakdown

Results on the same validation messages and the same Urgent-or-Routine task:

Counts by actual category reveal what each overall score contains.
Actual categoryTotal messagesCorrect predictionsErrors
Routine95950
Urgent505 missed urgent messages
Routine95923 false alarms
Urgent541 missed urgent message

Example

Read the counts with the score

The first two rows describe Always Routine. The last two rows describe Model B.

Always Routine: 95 correct out of 100 = 95%.

Model B: 4 correct Urgent predictions plus 92 correct Routine predictions gives 96 correct out of 100 = 96%.

The comparison is meaningful because both models are judged on the same messages and the same task.

Practice reading beyond the score

Check the hidden failure

What does the 95% accuracy of Always Routine fail to show?

Explain the limitation

Choose the best answer.

Write at least 70 characters (0/70).

Apply the category-count check

Make the baseline judgment

Northstar Support evaluates 100 messages: 95 are Routine and 5 are Urgent. The Always Routine baseline predicts Routine for every message.

Calculate its accuracy, state how many missed urgent messages it makes, and explain in one or two sentences why overall accuracy alone hides this failure.

Write at least 100 characters (0/100).

Step 7 of 8

Ask Which Mistakes Matter

Compare missed urgent messages and false alarms, then make a conditional judgment based on Northstar Support's stated priority.

See the operational difference

Two errors, two consequences

Northstar Support makes two different kinds of errors when it classifies an incoming message.

  • A missed urgent message is an Urgent message predicted as Routine. It may wait in the routine queue, delaying attention to a time-sensitive customer request.
  • A false alarm is a Routine message predicted as Urgent. It enters the priority queue and uses review time that might have gone to a truly urgent request.

Neither label is a new message category. They describe what went wrong in a prediction.

A diagram shows an urgent message incorrectly routed to a routine queue and a routine message incorrectly routed to a priority queue.

A missed urgent message can delay attention; a false alarm can consume priority-review time.

Compare the tradeoff

Read the error mix

The validation results show a tradeoff, not just a total number of errors. Model B finds more Urgent messages than Always Routine, but it also sends some Routine messages to the priority queue.

An error count alone does not tell you which operational burden occurred. Two models could each make three errors, yet one could miss three urgent requests while the other creates three false alarms. The consequences determine whether those results are equally acceptable.

Validation category breakdown

These category counts come from the same 100 validation messages: 95 Routine and 5 Urgent.

Model B reduces missed urgent messages relative to Always Routine, while adding false alarms.
Actual categoryTotal messagesCorrect predictionsErrors
Model B: Urgent541 missed urgent message
Model B: Routine95923 false alarms
Always Routine: Urgent505 missed urgent messages
Always Routine: Routine95950 false alarms

Practice identifying the tradeoff

Match the errors to their effects

Match each term to its meaning or consequence.

Tap an item and then its matching pair.

Apply the stated priority

Make a priority-based judgment

On the validation data, Model B has 1 missed urgent message and 3 false alarms. Always Routine has 5 missed urgent messages and 0 false alarms.

Northstar Support says that avoiding missed urgent messages matters more than reviewing extra Routine messages. In 2–4 sentences, state the consequence of each error type, then explain why Model B may be more useful under that priority. Also acknowledge Model B's additional false alarms.

Write at least 180 characters (0/180).

Step 8 of 8

Make a Cautious Evaluation Judgment

Integrate dataset roles, independence, category counts, baselines, and error consequences into a cautious evaluation judgment.

1. Read the evidence as a whole

A complete evaluation report

Read the whole report, not just one percentage

Northstar Support used 1,000 training messages to learn patterns. It compared model versions using 100 validation messages, then selected Model B. Only after that choice was settled, the team evaluated Model B once on a separate 100-message test set.

The test set was relevant to incoming support messages, had no overlapping conversations with development data, and used only Message text at arrival as an input. Confirmed priority was used only to score predictions. Resolution time was not used because it is known only after a case is resolved.

This setup makes the test result credible evidence about these held-out messages. It is still not a guarantee of performance on every future message.

Performance results

Model B was chosen using validation data. Its test result was read after development decisions were complete.

Accuracy by dataset
ModelTraining accuracyValidation accuracyTest accuracy
Model A99%80%Not evaluated
Model B97%96%94%

Keep the final check separate

The three flows represent separate jobs: learning, development choices, and the final independent check.

A diagram showing separate message-card flows for learning, model selection, and a sealed final evaluation.

The test messages remain separate until the final check.

Look behind the test score

On the test set, Model B made 94 correct predictions out of 100 = 94%. Its category counts matter:

  • Urgent: 4 correct predictions out of 5; 1 missed urgent message.
  • Routine: 90 correct predictions out of 95; 5 false alarms.

The baseline, Always Routine, made 95 correct predictions out of 100 = 95% on these same test messages. But it missed all 5 Urgent messages. A higher overall accuracy alone does not decide which approach is more useful.

Category breakdown

Model B's test-category results.

Results on 100 test messages
Actual categoryTotal messagesCorrect predictionsErrors
Urgent541 missed urgent message
Routine95905 false alarms

2. Reject a misleading result

Check a flawed claim

A report says: “Model B was selected using validation data and evaluated once on separate test data. The model used Message text at arrival and Resolution time as inputs, achieving 98% test accuracy.”

What is the strongest reason to reject this credibility claim?

3. Make a cautious judgment

Final application

Write a short evaluation judgment for the Northstar Support report. Include all of these points:

  1. Identify the warning pattern in Model A’s training and validation results.
  2. Reject the 98% claim from the previous screen and name its specific evaluation flaw.
  3. Explain why Model B’s independent test has a different purpose from validation data.
  4. Judge Model B using its test-category counts, the 95% Always Routine baseline, and the consequences of missed urgent messages versus false alarms.

State what the evidence supports, but do not claim that the model will perform the same way on every future message.

Write at least 300 characters (0/300).

4. Carry the checklist forward

Summary

Reusable evaluation checklist

Use this checklist whenever a model result looks impressive:

  • Dataset role: Was training data used for learning, validation data for choices, and test data for the final check?
  • Independence: Were test examples separate, relevant, and kept out of development decisions?
  • Gap: Is there a large difference between training and held-out results that could signal overfitting?
  • Category counts: Which categories were correct, and which errors occurred?
  • Baseline: Does the model add value beyond a simple alternative on the same examples?
  • Consequences: Are missed urgent messages or false alarms more costly for this task?

A careful conclusion separates two questions: Is the evaluation credible? And is the observed error tradeoff useful for the intended work?

  • Assign training, validation, and test data different jobs.
  • Treat a large training–held-out gap as a warning, not proof of one cause.
  • Reject results affected by leakage or test-guided development.
  • Compare category counts and a baseline, not only overall accuracy.
  • Connect errors to their real-world consequences.
  • Describe evidence cautiously; a test score does not guarantee every future outcome.

Step complete

Congratulations! You completed: Reading Model Evaluation Results

You completed Reading Model Evaluation Results.

Download the App now to have access to + 5000 free courses, exercises, certificates and lots of content without paying anything!

  • 100% free online courses from start to finish

    Thousands of online courses in video, ebooks and audiobooks.

  • More than 60 thousand free exercises

    To test your knowledge during online courses

  • Valid free Digital Certificate with QR Code

    Generated directly from your cell phone's photo gallery and sent to your email

Cursa app on the ebook screen, the video course screen and the course exercises screen, plus the course completion certificate