
Step 1 of 8
Start with an Evidence-First Test Case
Create a test case from source-based expectations before evaluating an assistant answer.
Learning path · Level 6 · Tutorial 4
Create a small, repeatable test set and a clear rubric for document-grounded answers. Evaluate evidence and meaning rather than relying on whether an answer sounds convincing or matches one preferred wording.
Start with an Evidence-First Test Case
Create a test case from source-based expectations before evaluating an assistant answer. 2 min
Choose a Representative Question Mix
Select four complementary questions that reflect the collection's intended uses and ordinary evidence limits. 2 min
Write Expectations from the Sources
3 min
Define Observable Scoring Anchors
Apply a consistent 0–2 rubric for accuracy, grounding, coverage, and uncertainty. 3 min
Calibrate Your Scores on Sample Answers
Score contrasting Cedar Learning Hub answers with independent criterion ratings and evidence-based notes. 3 min
Record a Repeatable Baseline
Create a run record that preserves the evaluation inputs, actual response, scores, and evidence notes. 2 min
Report Failures Without Hiding Them
2 min
Assemble and Review Your Test Set
Finish a four-case evidence-based test set with a reusable rubric, two scored responses, and a traceable evaluation record. 3 min

Step 1 of 8
Create a test case from source-based expectations before evaluating an assistant answer.
A test case links one question to four things:
Start with the documents, not with a convincing-sounding response. This keeps the test focused on what the Cedar Learning Hub sources actually support.

The workflow protects the evaluation from being shaped by the answer's wording.
A required answer point is a meaning that must be present. It is not a script the assistant must repeat.
For example, an answer can use different wording and still pass if it preserves the supported limit. Conversely, polished wording does not pass if it adds a claim the source does not establish.
Write short, observable unacceptable claims. They tell a reviewer what would materially change the answer.
Example
Question type: Direct lookup
Question: How far in advance may a registered member reserve a room?
Expected evidence: Room Booking Guide > Booking window — “Reservations may be made up to 7 days in advance.”
Required answer points:
7 days in advance.Unacceptable claims:
2 hours.An answer such as “A registered member can reserve a room up to 7 days ahead” preserves the required meaning. It need not copy the excerpt word for word.
Put these actions in the order that keeps expectations independent of the assistant answer.

Step 2 of 8
Select four complementary questions that reflect the collection's intended uses and ordinary evidence limits.
A small test set is useful when its questions cover different ways people will use the collection. Four easy questions about booking time may all look successful while leaving important behavior untested.
For Cedar Learning Hub, begin with four complementary purposes:
This is a bounded quality check for these intended uses—not a statistical claim about every possible question.

Each question type probes a different ordinary use or evidence limit in the collection.
Use these four questions as the starter mix. Do not write full expected evidence or scoring rules yet; first make sure the questions are meaningfully different.
The matrix reveals what each case is designed to exercise. A row should add a distinct purpose, source combination, or evidence limit—not just reword an existing question.
| Case | Question type | Question focus | Source coverage |
|---|---|---|---|
| 1 | Direct lookup | How far in advance can a registered member reserve a room? | Room Booking Guide > Booking window |
| 2 | Conditional rule | A registered member already has one active booking. Can they make another booking? | Room Booking Guide > Booking limits |
| 3 | Synthesis | Which room has both a display and a whiteboard, and what is the maximum duration for one booking? | Room Facilities Guide > Equipment; Room Booking Guide > Booking limits |
| 4 | Partly unsupported request | Does Birch Room have a whiteboard and microphones? | Room Facilities Guide > Equipment; microphone availability is Not stated in sources |
Match each candidate question to the testing purpose it represents.
Match each question to its question type.
Tap an item and then its matching pair.
Which proposed set is the representative four-case starter mix for Cedar Learning Hub?

Step 3 of 8
Build expectations before you look at an assistant answer. For this exercise, the complete source snapshot is below—no external files or tool access are needed.
Both sources are active, version 1.0, updated 2026-02-01. Treat the excerpts as the evidence boundary: do not add likely details from general knowledge. No passage in this snapshot addresses microphone availability.
Use each reference together with its short excerpt when you write expected evidence.
| Passage reference | Short source excerpt |
|---|---|
| Room Booking Guide > Eligibility | Registered members may book. |
| Room Booking Guide > Booking limits | Each member can hold one active booking. Each booking can last up to 2 hours. |
| Room Booking Guide > Booking window | Reservations may be made up to 7 days in advance. |
| Room Facilities Guide > Equipment | Cedar Room has a display and a whiteboard. |
| Room Facilities Guide > Equipment | Birch Room has a whiteboard but no display. |
A test case records the meaning an acceptable answer must preserve—not one preferred sentence.
For every question, fill the same fields:
A document title alone is too broad. Include the relevant section and excerpt, such as Room Booking Guide > Booking limits with the applicable rule.

For a synthesis question, preserve each needed meaning and its own evidence.
Example
Question type: Synthesis across sources
Question: “I need a room with a display for a 2-hour session. Which room can I book?”
Expected evidence
Room Facilities Guide > Equipment — “Cedar Room has a display and a whiteboard.”Room Booking Guide > Booking limits — “Each booking can last up to 2 hours.”Required answer points
Unacceptable claims
This case can accept a concise paraphrase. It does not require these exact words, but it does require both meanings.
A partly unsupported request has two parts: what the sources establish and what they do not establish.
Do not turn a gap into a negative fact. If neither source mentions microphones, write Not stated in sources; that does not mean microphones are unavailable.
For the selected room-equipment question, expected evidence should retain the documented equipment passage and also record the microphone gap. Its required answer points should separately cover the supported equipment fact and the unresolved microphone information. Its unacceptable claims should include invented positive or negative microphone statements.
This structure lets an answer be useful where evidence exists while remaining precise about its limit.
Tip
A source qualification belongs in more than one place:
Not stated in sources where needed.Do not write “the answer must use this exact sentence.” Evaluate whether the required meanings survive paraphrasing.
In your own text document, draft the four-case Cedar Learning Hub starter set. Use these questions:
For each test case, use exactly these labels: Question type, Question, Expected evidence, Required answer points, and Unacceptable claims.
Include passage references and short excerpts in expected evidence. Write required answer points as separate bullets. For the room-equipment question, preserve the source qualification and identify at least one unsupported claim. Paste your complete four-case draft here.
Write at least 500 characters (0/500).

Step 4 of 8
Apply a consistent 0–2 rubric for accuracy, grounding, coverage, and uncertainty.
A rubric keeps evaluation focused on observable evidence, not on whether an answer sounds polished. Score each dimension separately using the same scale: 0 = Not met, 1 = Partly met, 2 = Met.
These are independent questions. Do not replace them with one overall impression.

Each lens can receive a different score; fluent wording does not automatically pass all four.
Use these anchors consistently. Unverified means the sources do not establish a claim; it does not mean the claim has been proven false.
| Dimension | 0 = Not met | 1 = Partly met | 2 = Met |
|---|---|---|---|
| Accuracy | Incorrect or unverified core assertion | Minor error or unverified ancillary assertion; core remains intact | Verified correct |
| Grounding | Substantive claim lacks support, or cited passage does not support it | Support exists, but reference is missing or too broad | Substantive claims are supported by correctly identified passages |
| Coverage | No required answer points present | Some required answer points present | All required answer points present |
| Uncertainty | Guesses a gap or refuses answerable content | Separation is vague or unnecessarily cautious | Clearly separates support from genuine gaps without needless hedging |
The expectations are set before an answer is examined.
Question type: Routine question with a partly unsupported request
Question: According to the sources, who may book, what is the maximum duration of one booking, and are microphones available?
Expected evidence:
Room Booking Guide > Eligibility — “Registered members may book.”Room Booking Guide > Booking limits — “Each booking can last up to 2 hours.”Not stated in sources — microphone availability.Required answer points:
Not stated in sources.Unacceptable claims:
Example
Actual answer
“Registered members may book. Each booking can last up to 2 hours. Microphones are not available. [Room Booking Guide > Eligibility; Room Booking Guide > Booking limits]”
The first two claims are supported, but microphone availability is an explicit part of the question. Turning that evidence gap into a definite negative is an unverified core assertion, not an ancillary detail.
| Criterion | Score | Evidence note |
|---|---|---|
| Accuracy | 0 = Not met | “Microphones are not available” -> Not stated in sources -> An unverified core assertion answers an explicit part of the question. |
| Grounding | 0 = Not met | “Microphones are not available” -> cited Booking Guide passages do not address microphones -> Substantive claim lacks support. |
| Coverage | 1 = Partly met | Eligibility and 2-hour duration are present -> Required gap statement is absent -> Some required answer points are present. |
| Uncertainty | 0 = Not met | Definite microphone claim -> Not stated in sources -> The answer guesses instead of identifying the gap. |
Choose the score set that correctly evaluates all four separate observations.
A. “Cedar Room has a display.” No passage reference is supplied.
B. “Cedar Room has microphones.” It cites Room Facilities Guide > Equipment.
C. “Microphone availability is Not stated in sources.”
D. “I cannot determine whether one booking can last up to 2 hours.”

Step 5 of 8
Score contrasting Cedar Learning Hub answers with independent criterion ratings and evidence-based notes.
Calibration checks whether the rubric rewards evidence-based meaning rather than polished wording. For each answer:
Accuracy, Grounding, Coverage, and Uncertainty independently.A concise paraphrase can meet the rubric without copying a reference sentence. A fluent addition can lower several scores if the sources do not establish it. When reviewers disagree, return to the passage or evidence gap and the relevant scoring anchor—not the answer’s confidence or style.

Compare claims with evidence and required points before assigning independent ratings.
Example
Question type: Synthesis across sources
Question: What equipment does Cedar Room have, and what is the maximum duration of one booking?
Expected evidence
Room Facilities Guide > Equipment — “Cedar Room has a display and a whiteboard.”Room Booking Guide > Booking limits — “Each booking can last up to 2 hours.”Required answer points
Unacceptable claims
These expectations test meanings, not a preferred answer sentence.
Example
Question type: Routine question with a partly unsupported request
Question: Does Birch Room have a whiteboard and microphones?
Expected evidence
Room Facilities Guide > Equipment — “Birch Room has a whiteboard but no display.”Not stated in sources — microphone availability.Required answer points
Not stated in sources.Unacceptable claims
The documented equipment and the information gap must remain separate.
Score this sample answer against test case A from the previous screen.
Actual answer: “Cedar Room has a screen and a whiteboard. A booking can last no more than two hours. [Room Facilities Guide > Equipment; Room Booking Guide > Booking limits]”
Provide four separate scores using 0 = Not met, 1 = Partly met, or 2 = Met. Then write evidence notes in the form Claim -> Evidence -> Rating.
Write at least 100 characters (0/100).
Score this sample answer against test case B from the earlier screen.
Actual answer: “Birch Room has a whiteboard, but it has no microphones. [Room Facilities Guide > Equipment]”
Provide four separate scores using 0 = Not met, 1 = Partly met, or 2 = Met. Then write evidence notes in the form Claim -> Evidence -> Rating.
Write at least 100 characters (0/100).

Step 6 of 8
Create a run record that preserves the evaluation inputs, actual response, scores, and evidence notes.
A repeatable baseline preserves what was available and what was observed at one point in time. It does not promise that a later assistant will use identical wording.
Before judging the result, retain the source snapshot and relevant passages, the complete test case, the test set and rubric versions, and the exact instructions. Then preserve the Actual answer verbatim, followed by separate Scores and Evidence notes.
This order matters: an edited rewrite is not the answer that was scored. Without the sources, instructions, context mode, and original response, a later comparison is not fair.

A baseline connects the available evidence and instructions to the unedited answer and its criterion-level evaluation.
Tip
Do not silently replace a corrected expectation, source snapshot, or rubric. Preserve the earlier record and assign a new version to the corrected artifact. A comparison is meaningful only when you can see which version was used.
Likewise, record unavailable metadata as Unknown. Do not infer a model name, retrieval setting, or hidden configuration from an answer.
Use a text document or spreadsheet with setup, case, and result sections. The run-record labels below preserve the conditions that can affect an answer or its evaluation.
| Section | Run-record label | What to retain |
|---|---|---|
| Setup | Source snapshot | Active source titles, versions, update dates, and relevant passages |
| Setup | Test set version | Version of the test cases and their expectations |
| Setup | Rubric version | Version of the four scoring anchors |
| Setup | Instructions | Complete, exact answer instructions used for the run |
| Setup | Tool/model | Visible tool and model name, or Unknown |
| Setup | Context mode | Document retrieval, Direct context, or Provided sample |
| Result | Run date | Date in YYYY-MM-DD format |
| Result | Actual answer | Unedited response, including source references |
| Result | Scores | Accuracy, Grounding, Coverage, Uncertainty shown separately |
| Result | Evidence notes | Criterion notes in Claim -> Evidence -> Rating form |
Attach the complete test case to the record rather than relying on a short nickname. Preserve its Question type, Question, Expected evidence, Required answer points, and Unacceptable claims.
For Cedar Learning Hub, the source snapshot identifies both active sources: Room Booking Guide, version 1.0, updated 2026-02-01; and Room Facilities Guide, version 1.0, updated 2026-02-01. Retain the relevant passage excerpts alongside those identifiers.
A provided sample is a valid no-tool route. An optional local run requires the full source text and exact instructions that you supplied to your own assistant.
Example
Source snapshot: Active sources: Room Booking Guide, version 1.0, updated 2026-02-01; Room Facilities Guide, version 1.0, updated 2026-02-01.
Test set version: 1.0
Rubric version: 1.0
Instructions: “Answer using only the active Cedar Learning Hub sources. Cite factual claims with source title plus section. When requested information is absent, say Not stated in sources; do not infer or add outside facts.”
Tool/model: Unknown
Context mode: Provided sample
Run date: 2026-02-01
Question type: Routine question with a partly unsupported request
Question: Does Birch Room have a whiteboard and microphones?
Expected evidence: Room Facilities Guide > Equipment — “Birch Room has a whiteboard but no display.” Not stated in sources — microphone availability.
Required answer points
Not stated in sources.Unacceptable claims
Example
Actual answer: “Birch Room has a whiteboard. Microphone availability is Not stated in sources. [Room Facilities Guide > Equipment]”
Scores
2 = Met2 = Met2 = Met2 = MetEvidence notes
Room Facilities Guide > Equipment -> Accuracy 2.Room Facilities Guide > Equipment -> Grounding 2.The Actual answer remains an observation. Scores and Evidence notes are the later evaluation of that observation.
A colleague retained only these parts of a Cedar Learning Hub record: Run date, Test set version, Rubric version, and four separate Scores.
Write the missing record content needed for a fair repeat comparison. Include:
Tool/model: Unknown if no visible tool or model information was saved.Do not invent the old response, instructions, or metadata. State that the original item must be retained verbatim or recorded as Unknown where appropriate.
Write at least 500 characters (0/500).

Step 7 of 8
Report every evaluated test case as a four-part profile, in this order: Accuracy, Grounding, Coverage, and Uncertainty. Keep the four scores separate; do not replace them with an overall average or a label such as “passed.”
A positive-looking profile can still contain an important failure. An answer may include all required answer points while adding a claim that the sources do not support. Separate scores keep that grounding and uncertainty problem visible.

A case-level profile preserves the evidence-based detail that a single overall result could hide.
Use a case status before interpreting results:
Scored means an Actual answer was collected and evaluated.Not run means no answer was evaluated. It is not a pass, a failure, or a score of zero.For a scored case, make each failure note specific: Claim -> Evidence -> Rating. Identify the exact unsupported claim or omitted required answer point, then connect it to the relevant passage or to Not stated in sources.
Report what the evidence demonstrates. An unsupported statement is observable. Saying that retrieval failed, a model ignored instructions, or a source was missed is a hidden-cause explanation that this record does not establish.
Example
Actual answer: “Reservations may be made up to 7 days in advance. Bookings are confirmed by email. [Room Booking Guide > Booking window]”
Evidence notes
Room Booking Guide > Booking window — “Reservations may be made up to 7 days in advance.” -> Supported.Not stated in sources -> An unverified ancillary assertion; Accuracy 1 = Partly met for the case.Room Booking Guide > Booking window does not address confirmation methods -> Grounding 0 = Not met.Not stated in sources -> Uncertainty 0 = Not met.The note reports the unsupported addition without claiming why it appeared.
Keep an unrun case in the report. A partial baseline can be useful, but it must not be presented as a completed four-case result.
| Test case | Status | Accuracy | Grounding | Coverage | Uncertainty |
|---|---|---|---|---|---|
| Case 1: booking timing | Scored | 1 | 0 | 2 | 0 |
| Case 4: equipment and microphones | Not run | — | — | — | — |
Case 1 is Scored. Its answer correctly says reservations may be made up to 7 days in advance and cites Room Booking Guide > Booking window. It also states, “Bookings are confirmed by email.” Neither active Cedar Learning Hub source addresses booking confirmation methods.
Case 4, about Birch Room equipment and microphone availability, is Not run.
Which report is evidence-specific and does not hide either result?

Step 8 of 8
Finish a four-case evidence-based test set with a reusable rubric, two scored responses, and a traceable evaluation record.
A baseline-ready test set is a connected package, not merely a list of questions. Keep each test case linked to expectations written from the sources. Then keep collected answers, scores, and evidence notes separate from those expectations.
For Cedar Learning Hub, preserve the four complementary cases already established: direct lookup, conditional application of a rule, synthesis across sources, and a routine question with a partly unsupported request. This package supports a later comparison, but it establishes quality only for the cases that are actually scored.

Each test case begins with source-based expectations; answers are collected and scored afterward.
Question type
Direct lookup
Question
How far in advance may a registered member reserve a room?
Expected evidenceRoom Booking Guide > Booking window — “Reservations may be made up to 7 days in advance.”
Required answer points
7 days in advance.Unacceptable claims
2 hours.Question type
Conditional application of a rule
Question
A registered member already has an active booking. Can they make another booking?
Expected evidenceRoom Booking Guide > Booking limits — “Each member can hold one active booking. Each booking can last up to 2 hours.”
Required answer points
Unacceptable claims
Question type
Synthesis across sources
Question
I need a room with a display for a 2-hour session. Which room can I book?
Expected evidence
Room Facilities Guide > Equipment — “Cedar Room has a display and a whiteboard.”Room Booking Guide > Booking limits — “Each member can hold one active booking. Each booking can last up to 2 hours.”Required answer points
2 hours.Unacceptable claims
2 hours.Question type
Routine question with a partly unsupported request
Question
Does Birch Room have a whiteboard and microphones?
Expected evidence
Room Facilities Guide > Equipment — “Birch Room has a whiteboard but no display.”Not stated in sources — microphone availability.Required answer points
Not stated in sources.Unacceptable claims
Attach this shared rubric to every test case. Keep the four ratings separate; do not replace them with one combined total.
| Dimension | 2 = Met | 1 = Partly met | 0 = Not met |
|---|---|---|---|
| Accuracy | Verified correctness | Minor error or unverified ancillary assertion | Incorrect or unverified core assertion |
| Grounding | Substantive claims supported by correctly identified passages | Support exists, but reference is missing or too broad | Substantive claim lacks support or cited passage does not support it |
| Coverage | All required answer points present | Some required answer points present | No required answer points present |
| Uncertainty | Clearly separates support from genuine gaps | Separation is vague or unnecessarily cautious | Guesses a gap or refuses clearly answerable content |
Add this setup block before collecting answers. It preserves the exact conditions of test set version 1.0.
Room Booking Guide, version 1.0, updated 2026-02-01; Room Facilities Guide, version 1.0, updated 2026-02-01.1.01.0Not stated in sources; do not infer or add outside facts.”UnknownProvided sample2026-02-01For a local run, replace only metadata you can actually see. Keep unavailable metadata as Unknown.
Two cases are scored below; the other two remain visibly untested. A concise answer can pass when it preserves the required meaning and identifies supporting evidence. A fluent unsupported addition remains visible as a failure rather than being hidden by other strengths.
Cases 1 and 2 belong to the test set, but no Actual answer has been collected for them. They must remain Not run, not be treated as a pass or a score of zero.
| Test case | Status | Record state |
|---|---|---|
| Case 1: booking timing | Not run | No Actual answer, Scores, or Evidence notes |
| Case 2: active-booking limit | Not run | No Actual answer, Scores, or Evidence notes |
| Case 3: display and duration | Scored | Answer and criterion-level notes preserved |
| Case 4: equipment and microphones | Scored | Unsupported microphone assertion remains visible |
Status: Scored
Actual answer: “Cedar Room is the option with a display, and a booking can last up to 2 hours. [Room Facilities Guide > Equipment; Room Booking Guide > Booking limits]”
Scores
2 = Met2 = Met2 = Met2 = MetEvidence notes
Room Facilities Guide > Equipment — “Cedar Room has a display and a whiteboard.” -> Accuracy 2.2 hours maximum are both present -> Required answer points -> Coverage 2.Status: Scored
Actual answer: “Birch Room has a whiteboard, but it has no microphones. [Room Facilities Guide > Equipment]”
Scores
0 = Not met0 = Not met1 = Partly met0 = Not metEvidence notes
Not stated in sources -> Accuracy 0 because an unverified core assertion changes the answer.Room Facilities Guide > Equipment does not address microphone availability -> Grounding 0.Not stated in sources qualification is absent -> Required answer points -> Coverage 1.Not stated in sources -> Uncertainty 0 because the response converts a gap into certainty.Before calling the package ready, check:
Not stated in sources for a genuine gap.Scored; every other case is explicitly Not run.Claim -> Evidence -> Rating form.Summary
Use this sequence whenever you create or repeat a document-grounded answer check.
Review the checklist, then choose one test case from the on-screen set or your local copy. Submit a concise review containing:
Question type, Question, Expected evidence, Required answer points, and Unacceptable claims.Claim -> Evidence -> Rating with a score of 0, 1, or 2.No file upload is needed.
Write at least 180 characters (0/180).
Congratulations! You completed: Building an Evidence-Based Answer Test Set
Thousands of online courses in video, ebooks and audiobooks.
To test your knowledge during online courses
Generated directly from your cell phone's photo gallery and sent to your email
Download our app via QR Code or the links below:.
+ 10 million
students
Free and Valid
Certificate
60 thousand free
exercises
4.8/5 rating in
app stores
Free courses in
video and ebooks