Learning path · Level 7 · Tutorial 5

Testing Workflow Runs, Retries, and Usage Limits

Test complete workflow and agent runs rather than judging only the final answer. Record actions, introduce controlled failures, check retry behavior, and set usage limits using safe drafts or manual simulations instead of live consequential actions.

  • Level: Advanced
  • Duration: 20 min
  • 9 steps
Start tutorial Back to the path
Testing Workflow Runs, Retries, and Usage Limits

What you will go through

  1. Define What a Successful Run Must Do Define observable expectations for the Harbor Helpdesk normal workflow run, including its route, required operations, approval boundary, prohibited actions, output quality, and stopping outcome. 2 min
  2. Record Actions Without Overclaiming Build a minimal, evidence-based action record for Harbor Helpdesk workflow runs without overstating what happened. 2 min
  3. Introduce Failures Safely Use controlled, text-only failure probes to verify that Harbor Helpdesk routes safely to review or stops before an unsafe downstream operation. 3 min
  4. Choose Retry, Stop, or Escalation Choose an evidence-based recovery action for a workflow run while respecting approval decisions and retry bounds. 2 min
  5. Check for Completion Before Retrying Verify an uncertain state-changing operation before any retry, so a lost confirmation does not become a duplicate record. 2 min
  6. Budget the Whole Run, Including Recovery Calculate tool-call, service-call, retry, and illustrative cost totals before allowing another operation. 2 min
  7. Verify Which Limits Actually Stop Execution Distinguish advisory usage controls from verified enforced stops, then use supervised execution when reporting is delayed. 2 min
  8. Retest the Repair Without Weakening Safeguards Select regression cases that show a targeted repair works while preserving approval, duplicate-prevention, and usage safeguards. 2 min
  9. Complete a Safe Run-Test Review Integrate route expectations, action evidence, recovery, usage limits, and regression traces into a concise conclusion without treating simulated evidence as production certification. 3 min

What you will learn

  • Define expected routes, actions, approval decisions, and stopping outcomes for normal and failure cases.
  • Run safe tests for unavailable tools, unexpected outputs, missing data, and denied approval.
  • Maintain an action record that distinguishes attempted, completed, failed, and approval-pending operations without recording sensitive content.
  • Choose whether to retry, stop, or escalate, checking for already completed actions before retrying.
  • Set tool-call, retry, and usage or cost limits, identifying which limits are enforced and which require supervision.
  • Repeat affected tests after a change and document whether the failure was resolved without weakening safeguards.

Before you start

  • Building a No-Code Workflow with a Conditional Branch
  • Configuring a Bounded Tool-Using Agent
  • Building an Evidence-Based Answer Test Set

Step 1 of 9

Define What a Successful Run Must Do

Define observable expectations for the Harbor Helpdesk normal workflow run, including its route, required operations, approval boundary, prohibited actions, output quality, and stopping outcome.

See the run, not just the answer

Test the whole workflow run

A workflow run is more than its final reply. Test the full path: required inputs, transferred fields, selected route, each permitted operation, approval decision, output quality, and the point where the run stops.

For Harbor Helpdesk, use the fictional inputs request_ref and summary. In the normal case, the summary is What time do you close on Friday?. classify_request must return topic with the valid value hours before the normal route can continue.

A reply that says the right closing time is still not a successful run if it took the review route incorrectly, skipped approval, or saved before approval.

Normal route and safety boundaries

A flow diagram shows the normal Harbor Helpdesk path from inputs through classification, lookup, drafting, approval, and simulated saving. Invalid or missing results branch to review; denied approval ends before saving.

Use this as a route reference: invalid inputs or outputs go to review, and approval is a boundary before the simulated save.

Tip

Safe testing boundary

Use a text-only simulation first: write the fictional inputs and supplied observations, then compare them with expectations. If you also inspect a no-code tool, use disconnected or draft-only operations. Do not send messages or perform a live consequential action.

Specify the normal case

Write observable expectations first

Before observing a result, write expectations that can be checked. Include what must happen and what must not happen. This prevents a polished final reply from hiding an incorrect route or an unauthorized operation.

A compact expected-run record

Expected normal-run record for workflow configuration v1

These are planned expectations, not evidence that any external action occurred.
AspectExpectation
CaseNormal Friday-hours request
Expected route`classify_request` → `lookup_hours` → `draft_reply` → approval → `save_draft_record`
Expected actionsRequired fields are present; `topic` is `hours`; the approved note is used; a nonempty `reply_text` is drafted.
Expected approvalApproval is required before `save_draft_record`.
Expected stopA granted approval permits a simulated save, then the workflow run ends.
Must not occurNo `review` route, no save before approval, and no message sending.
Output quality`reply_text` is consistent with `Hours note v1`: Monday–Friday, 09:00–17:00.

Example

Example: expectation record before a test

Case: Normal Friday-hours request

Expected route: classify_request returns topic = hours; then lookup_hours, draft_reply, approval, and save_draft_record.

Expected actions: Transfer the supplied request_ref and summary; use Hours note v1; draft a nonempty reply_text consistent with the note.

Expected approval: Required before the simulated save.

Expected stop: End after the simulated save is permitted.

The later test record can add Observed result and a verdict. Do not fill those from an assumption before the workflow run is tested.

Practice: identify a final-answer-only test

Find the gap

Which proposed test is incomplete because it judges only the final reply?

Practice: make the run test complete

Select the needed expectations

A proposed Harbor Helpdesk test supplies the normal inputs and checks only that a final draft gives the Friday closing time. Which added expectation set would make it capable of exposing an incorrect route or an unauthorized save?

Step 2 of 9

Record Actions Without Overclaiming

Build a minimal, evidence-based action record for Harbor Helpdesk workflow runs without overstating what happened.

Status and evidence fields

Record evidence, not assumptions

An action record reports only what the evidence supports for one workflow run; it is not a retrospective success story. Each event links Run, Event, Operation, Attempt, Status, Input summary, Evidence, Approval, and Error.

For Harbor Helpdesk, a minimal record can state that request_ref and summary are present without copying the request content. It can retain the approved reference Hours note v1 when relevant, while omitting a full reply_text, private message body, secrets, and unrestricted raw service responses.

A trace keeps events separate

A chronological workflow trace separates an operation request, approval state, dispatch, and an unconfirmed timeout instead of turning them into one final result.

Keep the initial attempted event and later evidence in order. A timeout without completion evidence becomes outcome_unknown, not completed or failed.

Use precise action statuses

Status describes the evidence for an operation event. Approval is a separate field with not_required, pending, granted, or denied. An approval decision itself is neither an operation nor a service call.

Status values for an action record
StatusUse whenSafe evidence example
attemptedThe operation request was dispatched; no result is established yet.Dispatch recorded
completedEvidence confirms that the operation finished.Returned required result
failedAn explicit failure result was observed.Unavailable-operation error
approval_pendingA proposed operation awaits a decision and has not been dispatched.Approval request shown
outcome_unknownA timeout or connection loss leaves completion unconfirmed.Confirmation missing after timeout
not_runThe operation was never dispatched.Approval was denied

Tip

Evidence labels matter

Prefix evidence with Simulated for a supplied manual result and Observed for a result actually seen in the test. A simulated completion never means an external action occurred. If execution details are unavailable, record them as unknown.

If approval is denied, the proposed operation is not_run. Put the denial in Approval, not in Status as a failed operation.

Preserve the timeline

Preserve the complete event timeline

This fictional Harbor Helpdesk trace includes every operation dispatched on the valid hours route: classify_request, lookup_hours, draft_reply, and save_draft_record. For each dispatched operation, the initial attempted event remains visible beside its later result. The save has no completion evidence after its timeout, so the record does not rewrite it as a success or failure.

Core event timeline

Each repeated operation name below refers to the same initial operation and attempt; it is not a new operation. Field presence and approved references are sufficient for this record.

Action-record timeline for fictional `Run` R-17
RunEventOperationAttemptStatusInput summary
R-171`classify_request`1attempted`request_ref` present; `summary` present
R-172`classify_request`1completed`request_ref` present; `summary` present
R-173`lookup_hours`1attempted`request_ref` present; `topic` present
R-174`lookup_hours`1completedApproved source: `Hours note v1`
R-175`draft_reply`1attemptedApproved source: `Hours note v1`
R-176`draft_reply`1completedDraft reference only; no message body retained
R-177`save_draft_record`1approval_pending`request_ref` present; draft reference only
R-178`save_draft_record`1attempted`request_ref` present; approved draft reference only
R-179`save_draft_record`1outcome_unknown`request_ref` present; approved draft reference only

Supporting evidence and approval

Approval and evidence support the statuses, but do not replace them. Saving is simulated; no message is sent in this example.

Evidence fields for the same fictional trace
RunEventEvidenceApprovalError
R-171Simulated: dispatch recordednot_requirednone
R-172Simulated: returned `topic=hours`not_requirednone
R-173Simulated: dispatch recordednot_requirednone
R-174Simulated: approved hours reference returnednot_requirednone
R-175Simulated: dispatch recordednot_requirednone
R-176Simulated: nonempty draft consistent with `Hours note v1`not_requirednone
R-177Simulated: valid draft awaits a decisionpendingnone
R-178Observed: approval granted and dispatch recordedgrantednone
R-179Observed: timeout; no completion confirmationgrantedConfirmation timeout

Example

Do not replace an attempt with a claim

Event 8 remains attempted: it records dispatch. Event 9 records the later evidence for that same save attempt. Because completion was not observed, the supported outcome is unknown. This trace does not decide what to do next; it records only what happened and what remains unconfirmed.

Practice: evidence-based statuses

Classify the trace evidence

Match each Harbor Helpdesk trace event to the action status supported by its evidence.

Tap an item and then its matching pair.

Step 3 of 9

Introduce Failures Safely

Use controlled, text-only failure probes to verify that Harbor Helpdesk routes safely to review or stops before an unsafe downstream operation.

Keep the Test Controlled

One Changed Condition, One Trace

A controlled failure probe starts from the Harbor Helpdesk normal case in configuration v1: request_ref is present, summary is What time do you close on Friday?, and the approved source remains Hours note v1.

Change exactly one condition for each probe. Preserve the other inputs, source, and configuration so that any route change or blocked operation has a clear cause.

Use supplied fictional observations rather than disrupting a real service. Prefix the evidence with Simulated. Check the whole workflow run: selected route, dispatched and blocked operations, approval decision, relevant action status, and stopping outcome. A plausible reply never makes an unsafe path acceptable.

How Controlled Probes Isolate a Failure

A baseline workflow branches into unavailable-operation, invalid-output, missing-data, and denied-approval probes. Each path blocks unsafe downstream work and ends in review or a stop.

Each probe changes one condition while preserving the normal Harbor Helpdesk context.

Specify the Safe Path for Each Failure

Four Failure Categories

For an unexpected-output probe, substitute one invalid result at a time—for example, an invalid topic value or a missing topic field. Do not combine both substitutions in one workflow run.

For every case, decide what must not happen as well as what should happen. Record only minimal, permitted details: field presence, the approved source reference, the simulated result category, and any error. Keep an approval decision separate from an operation status.

Scenario Matrix

Use this matrix before simulating a case. It makes the required route, blocked work, and stopping boundary observable without using a live service.

All inputs, records, and observations in this matrix are fictional text-only simulations.
CaseExpected routeExpected actionsExpected approvalExpected stopAction-record evidence
Unavailable operation`review` after `lookup_hours``lookup_hours` is attempted, then failed; no `draft_reply` or `save_draft_record`not_requiredStop at review after the lookup failureSimulated unavailable response; `lookup_hours` status `failed`
Unexpected output`review` after `classify_request`Invalid or missing `topic` is not consumed; no `lookup_hours`not_requiredStop at reviewSimulated invalid output; record the classification result and no downstream operation
Missing data`review` before classificationDo not invent a required field; `classify_request` is not dispatchednot_requiredStop for correction or reviewSimulated required-field absence; `classify_request` status `not_run`
Denied approvalApproval boundary after a valid draftCompleted draft remains available for review; `save_draft_record` is not dispatcheddeniedStop after denial; no alternate save operationSimulated valid draft; approval `denied`; save status `not_run`

Tip

Approval Is Its Own Boundary

A denied approval is not a failed operation. Record the completed draft_reply separately, record approval as denied, and record the proposed save_draft_record as not_run. The simulated save never occurs, and no alternate operation may bypass the decision.

Practice All Four Failure Categories

Interpret Four Simulated Probes

For each supplied text-only observation, match it to the complete safe test conclusion: route, relevant action status, blocked operation, and stopping outcome.

Tap an item and then its matching pair.

Order the Test Procedure

Sequence a Controlled Failure Probe

Put the controlled-test procedure in the only order that supports a clear comparison.

  1. Preserve the normal `v1` inputs, source, configuration, and expected safety boundary.
  2. Introduce one supplied failure condition in a text-only simulation.
  3. Record the observed route, operations, status, approval, evidence, and stopping outcome.
  4. Compare the observation with the expected record and assign `pass`, `fail`, or `not_verified`.

Apply the Procedure and Check Coverage

Write a Safe Failure-Test Conclusion

Write a concise test note for this supplied text-only scenario:

Configuration v1, request_ref, summary, and Hours note v1 are preserved. classify_request, lookup_hours, and draft_reply reached a valid draft consistent with the approved note. The approval decision is recorded as denied. No save observation is supplied.

Use numbered lines to provide:

  1. The four-step controlled-test procedure in order.
  2. The route reached, the blocked operation and relevant action status, and the stopping outcome.
  3. A coverage check stating whether the test suite includes all four required failure categories, with their names.

Do not infer an external save from this simulation.

Write at least 350 characters (0/350).

Step 4 of 9

Choose Retry, Stop, or Escalation

Choose an evidence-based recovery action for a workflow run while respecting approval decisions and retry bounds.

Classify the reason before acting

Recover from evidence, not preference

A retry is an additional attempt after an initial attempt. It is a recovery choice, not a response to any result you dislike.

First classify the evidence. A failed operation with observed evidence of a temporary service problem may justify one bounded retry. Respect a service-specified wait when one is available.

Do not automatically retry missing request_ref or summary, an invalid topic, or a permission failure. Those conditions need correction, review, or escalation. A valid hours route still requires approval before save_draft_record.

Treat approval separately from an operation status. A denied approval is a decision to stop the save, not a technical failure to overcome.

A recovery decision follows the evidence

A diagram separates temporary service failure, missing data, permission failure, and denied approval into retry, review, escalation, and stop paths.

Only observed temporary failure can support a bounded retry. Approval denial blocks the proposed save.

Apply the bounds to each workflow run

Count retries precisely

Use the stated limits across the whole workflow run:

  • 1 retry per operation
  • 2 retries per run

The initial attempt does not count as a retry. For example, an initial lookup_hours attempt followed by one additional attempt uses 1 retry for that operation and 1 retry for the run.

Before dispatching a retry, check the operation’s prior attempts, retries already used in the workflow run, the failure evidence, and any stated waiting period. Both an automatic retry and a person’s manual retry count toward the same allowance.

Recovery guide

Use this decision guide after reading the action record. It does not replace the required approval gate.

Evidence determines the next action; an undesirable reply alone is not temporary-failure evidence.
Observed conditionNext actionInformation to check first
Temporary service failureBounded retry may be allowedFailure evidence, stated wait, remaining retry bounds
Missing required inputRequest correction or route to reviewWhich required field is absent
Invalid output such as `topic`Route to review; do not continue downstreamOutput contract and recorded invalid value
Permission failureStop and escalatePermission scope and authorized owner
Approval deniedStop `save_draft_record`Recorded approval decision; no alternate save path
Retry allowance exhaustedStop or escalateOperation and run retry counters

Example

Example: bounded recovery

If draft_reply initially fails with observed evidence that its service is temporarily unavailable, one later additional attempt can be considered only after the stated wait and both retry counters are checked. If that retry also fails, its per-operation retry allowance is exhausted. Record a stop or escalation instead of adding another attempt.

Protect against an uncertain state-changing operation

Unknown is not failed

An outcome_unknown status needs special care when the operation could change state. A lost confirmation for save_draft_record does not prove that the simulated save failed, and it does not prove completion either.

Do not issue a retry merely to obtain a clearer answer. First seek appropriate completion evidence. If completion cannot be established safely, stop and escalate rather than assuming another attempt is harmless. The next step develops the full read-only verification procedure.

Example

Example action-record entry

Run: HH-204
Event: 5
Operation: save_draft_record
Attempt: 1
Status: outcome_unknown
Input summary: request_ref present; approved draft reference only
Evidence: Observed confirmation unavailable after request dispatch
Approval: granted
Error: confirmation unavailable

Decision: do not label this operation failed or completed without evidence. Preserve the same operation reference and record a stop or escalation if safe completion evidence is unavailable.

Tip

Keep the decision auditable

For any recovery decision, keep the initial attempt in the record. A later retry does not erase it. If you stop, record why: invalid input, invalid output, permission failure, denied approval, exhausted retry allowance, or uncertain outcome.

Practice: select the next action

Choose the recovery decision

Match each Harbor Helpdesk condition to the appropriate next action and the information required before acting.

Tap an item and then its matching pair.

Step 5 of 9

Check for Completion Before Retrying

Verify an uncertain state-changing operation before any retry, so a lost confirmation does not become a duplicate record.

Treat an uncertain save as unresolved

A timeout is not a safe retry signal

A timeout after save_draft_record does not prove that the save failed. The destination may have completed the operation while its confirmation was lost. Mark the operation as outcome_unknown and preserve its operation reference.

Do not restart the whole workflow run. Re-running earlier operations can create a second save attempt for the same draft. First establish what happened to the original state-changing operation.

Why verification comes before a repeat

A save operation with a lost confirmation branches into a read-only check and three possible evidence outcomes.

After an uncertain save, verify the destination before deciding whether any retry is safe.

Check the destination, not the final reply

Verify the specific operation

Use the same operation reference when you check the destination. In Harbor Helpdesk, check_draft_record is read-only: it can look for the draft record without creating or changing one.

Evidence of a matching record means save_draft_record is already completed: do not repeat it. An authoritative destination result that confirms no matching record can support a possible retry. But a missing result from a delayed or incomplete search is still inconclusive—not proof of noncompletion.

Before a retry after confirmed noncompletion, confirm that the granted approval still covers the unchanged proposed save, the run is at a safe recovery point, and retry allowance remains. Otherwise, stop and escalate.

Example

Example: preserve the uncertainty

Run: HH-27
Event: 4
Operation: save_draft_record
Attempt: 1
Status: outcome_unknown
Input summary: request_ref present; approved hours route; draft reference only
Evidence: Simulated save confirmation timed out; completion was not observed
Approval: granted
Error: Simulated timeout

Event: 5
Operation: check_draft_record
Attempt: 1
Status: attempted
Input summary: same operation reference
Evidence: Simulated read-only destination check requested
Approval: not_required
Error: none

The record does not relabel the save as failed or completed without evidence. The next observed check result determines the safe outcome.

Practice the safe sequence

Order the recovery sequence

A simulated save_draft_record confirmation timed out. Put the recovery actions in the only safe order.

  1. Record attempt 1 as `outcome_unknown` and retain its operation reference.
  2. Classify the returned evidence as completion, authoritative noncompletion, or inconclusive.
  3. Take the permitted final outcome; consider a retry only if noncompletion is confirmed and safeguards remain valid.
  4. Run the read-only `check_draft_record` using that same reference.

Choose a different response for each result

Choose the outcome supported by evidence

Match each simulated verification result to its safe outcome.

Tap an item and then its matching pair.

Step 6 of 9

Budget the Whole Run, Including Recovery

Calculate tool-call, service-call, retry, and illustrative cost totals before allowing another operation.

Separate the counting units

Count the layers, not just the route

A visible workflow step is not a complete usage unit. Track each layer separately:

  • An operation is the named action in the workflow run.
  • A tool call is one dispatched operation request.
  • A service call is an underlying request made by that operation.
  • Estimated cost is calculated from the applicable service calls.

In Harbor Helpdesk, most operations use one tool call and one service call. draft_reply is different: its one tool call triggers two model-service calls—drafting and checking. Approval decisions use neither tool calls nor service calls.

One operation can use more than one service call

A four-operation workflow diagram with one tool-call marker per operation. The draft operation has two service-call markers, while approval has none.

Use the operation route to plan usage, but total the dispatched tool calls and underlying service calls separately.

Illustrative rate card

Use this fictional rate card only for the exercises in this tutorial. It is an accounting example, not provider pricing.

Cost per service call
OperationService calls per tool callEstimated cost
`classify_request`1USD 0.01
`lookup_hours`1USD 0.00
`draft_reply`2USD 0.02 each
`save_draft_record`1USD 0.00
`check_draft_record`1USD 0.00

Calculate the normal route

Example

Worked normal-run ledger

For normal workflow run N-01, request_ref and summary are present. classify_request returns topic: hours; lookup_hours uses approved Hours note v1; draft_reply produces a nonempty reply_text consistent with that note. Approval is granted, so the simulated save_draft_record operation can run.

The route has four operations. It has four tool calls, but five service calls because draft_reply makes two. Its estimated cost is USD 0.05. A correct final reply alone would not replace the required approval or save boundary.

Normal route accounting

This compact ledger is linked to Run and uses the counter headings consistently. The save is simulated; no message is sent.

Usage for workflow run N-01
RunOperationTool calls usedService calls usedEstimated cost
N-01`classify_request`11USD 0.01
N-01`lookup_hours`11USD 0.00
N-01`draft_reply`12USD 0.04
N-01`save_draft_record`11USD 0.00
N-01Total45USD 0.05

Tip

Plan before dispatch

Before dispatching an operation, reserve its known possible usage. For example, draft_reply needs one tool call, two service calls, and USD 0.04 under this rate card. Do not infer its cost from the number of visible operations.

Budget recovery before it happens

Include verification and retry

Recovery also has a budget. Suppose a save_draft_record attempt has outcome_unknown. A read-only check_draft_record is an additional operation, not free bookkeeping: it uses one tool call and one service call. If authoritative evidence confirms noncompletion, an allowed retry of the same save operation adds another tool call and service call.

A failed request can also consume quota or incur a charge under a service's rules. When usage or pricing is hidden or variable, record the estimate as uncertain rather than calling it an exact bill.

Recovery projection

Assume this recovery occurs after the normal-prefix counts: the original save attempt was already dispatched. check_draft_record confirms that no record was saved, so one retry of save_draft_record is permitted. The rate card assigns USD 0.00 to both operations.

Projected counters for workflow run R-42
RunEventTool calls usedService calls usedEstimated cost
R-42Before completion check45USD 0.05
R-42`check_draft_record` completed56USD 0.05
R-42Allowed save retry projected67USD 0.05

Tip

Check every applicable allowance

The planned targets are: one retry per operation, two retries per workflow run, six tool calls per workflow run, eight service calls per workflow run, USD 0.08 per workflow run, and USD 0.20 per test session.

For R-42, the projected retry remains within the per-operation retry target, the per-run retry target, and the projected run counters. A new workflow run resets per-run counters only. It does not reset the test session's accumulated estimated cost or a shared allowance.

Warning

Do not reset the session total

These are planning targets, not a claim that any platform automatically blocks excess use. Keep the workflow run ledger and the test-session total separate; a run that fits its own target can still exceed the session allowance.

Practice: complete and check the ledger

Normal-run total

For a normal Harbor Helpdesk workflow run with the normal route, complete this compact ledger total. Enter the values in this order: tool calls / service calls / estimated cost.

Recovery total

In workflow run R-32, the original save_draft_record attempt has outcome_unknown. A completed check_draft_record gives authoritative evidence of noncompletion, and the proposed save retry is allowed. Complete the projected total after that retry: tool calls / service calls / estimated cost. Before R-32, the test session has already recorded USD 0.12.

Dispatch decision

Does the proposed retry in R-32 fit every applicable retry, workflow-run, and test-session allowance?

Step 7 of 9

Verify Which Limits Actually Stop Execution

Distinguish advisory usage controls from verified enforced stops, then use supervised execution when reporting is delayed.

Build a limit inventory

Inventory limits, not promises

A limit inventory separates what a limit says from what it can actually do. For each limit, record its scope, value, enforcement mechanism, visible evidence, and the supervision still required.

Use the inventory before dispatching an operation. A cap is a hard cap only when evidence shows that the application blocks the relevant service call or tool call at the stated scope. A displayed target alone is not proof of enforcement.

Different controls have different effects

A diagram contrasts a checklist, dashboard, alert, and locked application gate; only the gate visibly blocks an operation path.

Information and notifications can support a decision, but only a verified stop blocks dispatch.

Example limit inventory

For Harbor Helpdesk v1, document evidence rather than assuming that every displayed target is enforced.

A limit applies only to the scope demonstrated by its evidence.
ControlScope and valueMechanismVisible evidenceSupervision needed
Run tool-call target6 per workflow runWritten instructionConfiguration noteCheck before each dispatch
Run service-call target8 per workflow runManual ledgerCounter summary by `Run`Reserve multi-call operations
Run cost targetUSD 0.08 per workflow runDashboard alertUsage alert after reportingTreat delayed totals as uncertain
Simulated tool-call guardOne simulated workflow run; 6 tool callsApplication stop in simulationSimulated seventh dispatch blockedProduction enforcement not established

Do not confuse visibility with enforcement

Classify the level of control

These controls provide different protection:

  • A written instruction tells a person or agent what to do. It does not itself prevent a service call.
  • A dashboard display reports usage. Reporting may be delayed or incomplete.
  • An alert signals a threshold. Unless tested otherwise, it is a notification, not a stop.
  • An enforced application stop blocks dispatch within its verified scope.

Do not extend a claim beyond its evidence. A verified stop for one connected service does not automatically cover another service, the full workflow run, or session-wide spending.

Warning

Why a displayed total may be unsafe

Counters can lag behind reality. Queued operations may run later; concurrent workflow runs may consume the same allowance; automatic retries add attempts; and reporting may arrive after service calls were made.

A manual ledger or alert is useful for supervision, but neither proves that all remaining allowance is available. Do not use delayed reporting as a reason to dispatch a bypass retry.

Stop safely when the next action cannot fit

Use a supervised fallback

When reliable enforcement is unavailable, use supervised execution:

  1. Before dispatch, calculate the next operation's known tool calls, service calls, and estimated cost.
  2. Compare them with a conservative remaining allowance for the same workflow run and test session.
  3. Dispatch one operation only when the allowance is established and sufficient.
  4. Update the compact counters linked by Run after the observation.
  5. If reporting is delayed or the remainder cannot be established safely, stop and escalate for review.

This fallback reduces risk; it does not turn a manual ledger into an enforced cap.

Example

Simulate a boundary without dispatching

Simulated boundary test for Run R-18:

draft_reply attempt 1 had a simulated temporary service failure after the valid classify_request and lookup_hours operations. The current counters are Tool calls used 3, Service calls used 4, and Estimated cost USD 0.05. The test-session total is USD 0.17. Usage reporting is delayed.

A permitted retry would be draft_reply attempt 2. It would require 1 tool call, 2 service calls, and USD 0.04 under the fictional rate card. The projected workflow run cost would be USD 0.09 and the projected session cost would be USD 0.21. Both exceed their planned targets.

Do not dispatch the retry. Record the stopped proposal accurately:

Run: R-18
Event: 7
Operation: draft_reply
Attempt: 2
Status: not_run
Input summary: required fields present; approved source Hours note v1
Evidence: Simulated boundary check: projected run and session cost exceed targets; reporting delayed
Approval: not_required
Error: remaining cost allowance insufficient

The counters remain unchanged. Do not add a bypass retry; the recorded stop is the expected outcome.

Apply the limit evidence

Control classification under delayed reporting

A written instruction says “stay under the budget.” A dashboard displays usage. An alert appears at a threshold. A verified application stop blocks a dispatch for one service. A manual ledger is maintained by an operator.

Usage reporting is delayed before the next operation. Which classification and response is safe?

Step 8 of 9

Retest the Repair Without Weakening Safeguards

Select regression cases that show a targeted repair works while preserving approval, duplicate-prevention, and usage safeguards.

Define a regression run

Regression checks the whole path

A regression workflow run asks two questions at once: did v2 fix the observed defect, and did it preserve the controls that were already correct in v1?

For Harbor Helpdesk, do not judge only whether a final reply looks plausible. Compare the route, the transferred permitted fields (request_ref, summary, and topic), each dispatched operation, approval, and the stopping outcome.

The normal path remains: classify_request returns hours; lookup_hours uses Hours note v1; draft_reply produces a nonempty reply_text consistent with that note; approval is granted; then save_draft_record is simulated. other, missing inputs, or invalid outputs go to review without saving.

A repair is not demonstrated if it gets a good-looking result by skipping approval, hiding an error, or removing a stopping limit.

What must stay protected

A before-and-after workflow diagram where a repaired validation checkpoint redirects a flawed draft to review while approval, usage, and verification safeguards remain in place.

Compare the entire trace: the targeted route can change, while approval, stopping limits, and verification protections remain.

Make traces comparable

Link the defect to a specific adjustment

Record the change narrowly enough to test it. The source note stays unchanged: Hours note v1 contains Monday–Friday, 09:00–17:00. A configuration change is not evidence by itself; the before-and-after traces provide the evidence.

For each comparable case, inspect the selected route, permitted field transfers, operations and attempts, approval decision, counters, and stop. A verdict is pass, fail, or not_verified—not an action status. If a behavior cannot be observed, mark it not_verified rather than assuming success.

Worked change record

A compact change record for a fictional validation repair. It describes what v2 must prove; it does not claim that a live save occurred.

Change record: v1 to v2
FieldRecorded entry
Configurationv1 → v2
Observed defectAn empty `reply_text` was allowed past draft validation
Specific adjustmentRequire nonempty, note-consistent `reply_text` before approval
Unchanged source`Hours note v1`
Must remain unchangedApproval before simulated save; review for invalid output; planned limits

Cover the repair and its boundaries

Build the affected test set

Select regression cases from the changed boundary outward. Start with the original failing case, then include the normal route that shares the changed validation point. Add safety cases whose protections could be bypassed by the changed route.

Use a compact test record with these headings: Case, Expected route, Expected actions, Expected approval, Expected stop, Observed result, and Verdict. Cite observable evidence in Observed result; use not_verified when the needed behavior was inaccessible.

All Harbor Helpdesk saving remains simulated. A simulated completion is evidence about the test trace, not an external action.

Regression case coverage

For the v2 validation repair, these cases check both the fix and retained safeguards.

Cases to compare against v1 expectations
CaseWhat it checksRequired safe outcome
Original invalid draftEmpty or invalid `reply_text``review`; no approval or save
Normal requestValid hours route and field transferApproval still precedes simulated save
Denied approvalValid draft, approval deniedStop; `save_draft_record` is not_run
Uncertain simulated saveLost confirmation after a save attemptUse `check_draft_record` before any retry
Limit boundaryNext operation would exceed planned allowanceDo not dispatch; record the stop

Evaluate the repair claim

Choose a defensible repair test

In v1, an empty reply_text after an hours classification incorrectly continued toward approval. In v2, validation was changed so an invalid draft routes to review. A demonstration shows a normal draft being saved, but approval was omitted and the planned limit of six tool calls per run was disabled.

Which conclusion and regression plan is valid?

Step 9 of 9

Complete a Safe Run-Test Review

Integrate route expectations, action evidence, recovery, usage limits, and regression traces into a concise conclusion without treating simulated evidence as production certification.

Assemble the evidence packet

One conclusion, connected evidence

A final run-test review connects expectations, action evidence, recovery decisions, remaining allowances, enforcement evidence, and regression results. It is an evidence-backed conclusion—not a claim that a workflow is ready for production.

This capstone uses supplied fictional Harbor Helpdesk records only. No external account, message, or live write is used. A Simulated completion describes the test trace; it never means an external action occurred.

The integrated test packet

Connected visual cards show route expectations, action evidence, verification, approval, usage limits, and regression comparison.

A defensible conclusion links what was expected, what the trace shows, what recovery was allowed, and what remains unverified.

Preserve the uncertain save evidence

Read the `R-17` timeout record

R-17 is a v1 timeout case using request_ref, the summary What time do you close on Friday?, and unchanged Hours note v1. The valid route reaches granted approval, then the first save_draft_record attempt loses its confirmation.

The proposed direct retry is recorded as not_run: it was blocked because no completion check occurred. It uses no additional tool call or service call. This run stops for escalation; it is not continued or relabeled as v2.

Action record: `R-17` in `v1`

Each dispatched operation retains an attempted event and a later result event. Event 10 records a blocked proposal, not a dispatch.

Chronological action timeline for the timeout case.
RunEventOperationAttemptStatusInput summary
R-171`classify_request`1attempted`request_ref` present; `summary` present
R-172`classify_request`1completed`request_ref` present; `summary` present
R-173`lookup_hours`1attempted`request_ref` present; `topic` present
R-174`lookup_hours`1completedApproved source: `Hours note v1`
R-175`draft_reply`1attemptedApproved source: `Hours note v1`
R-176`draft_reply`1completedDraft reference only; nonempty and note-consistent
R-177`save_draft_record`1approval_pending`request_ref` present; approved draft reference
R-178`save_draft_record`1attempted`request_ref` present; approved draft reference
R-179`save_draft_record`1outcome_unknownSame save operation reference; confirmation unavailable
R-1710`save_draft_record`2not_runUnchanged approved draft reference; proposed direct retry

Supporting record: `R-17` in `v1`

Evidence, approval, and error support the action statuses; they do not replace them.

Evidence and approval history keyed to the action timeline.
RunEventEvidenceApprovalError
R-171Simulated: dispatch recordednot_requirednone
R-172Simulated: returned `topic=hours`not_requirednone
R-173Simulated: dispatch recordednot_requirednone
R-174Simulated: approved hours reference returnednot_requirednone
R-175Simulated: dispatch recordednot_requirednone
R-176Simulated: nonempty draft consistent with `Hours note v1`not_requirednone
R-177Simulated: valid draft awaits a decisionpendingnone
R-178Observed: approval granted and dispatch recordedgrantednone
R-179Observed: confirmation timeout; completion not establishedgrantedConfirmation timeout
R-1710Simulated: direct retry proposal blocked before dispatchgrantedVerification required before retry

Keep the regression run separate

Start the separate `R-21` regression run

R-21 is a separate v2 regression simulation of the same timeout condition. It is not a continuation or rerun of R-17. After the timeout, v2 checks the same save operation reference with read-only check_draft_record before considering a retry.

The unchanged approval remains granted for the unchanged proposed save. The supplied check result is simulated and authoritative for this fictional exercise: no matching record exists with immediate visibility.

Action record: `R-21` through timeout

The first nine events reach the uncertain save outcome without assuming either failure or completion.

Initial chronological events in the separate `v2` workflow run.
RunEventOperationAttemptStatusInput summary
R-211`classify_request`1attempted`request_ref` present; `summary` present
R-212`classify_request`1completed`request_ref` present; `summary` present
R-213`lookup_hours`1attempted`request_ref` present; `topic` present
R-214`lookup_hours`1completedApproved source: `Hours note v1`
R-215`draft_reply`1attemptedApproved source: `Hours note v1`
R-216`draft_reply`1completedDraft reference only; nonempty and note-consistent
R-217`save_draft_record`1approval_pending`request_ref` present; approved draft reference
R-218`save_draft_record`1attempted`request_ref` present; approved draft reference
R-219`save_draft_record`1outcome_unknownSame save operation reference; confirmation unavailable

Supporting record: `R-21` through timeout

The timeout evidence belongs in Evidence, not Input summary. Approval remains a separate field.

Evidence and approval history for the initial `R-21` events.
RunEventEvidenceApprovalError
R-211Simulated: dispatch recordednot_requirednone
R-212Simulated: returned `topic=hours`not_requirednone
R-213Simulated: dispatch recordednot_requirednone
R-214Simulated: approved hours reference returnednot_requirednone
R-215Simulated: dispatch recordednot_requirednone
R-216Simulated: nonempty draft consistent with `Hours note v1`not_requirednone
R-217Simulated: valid draft awaits a decisionpendingnone
R-218Simulated: approval granted and dispatch recordedgrantednone
R-219Observed: confirmation timeout; completion not establishedgrantedConfirmation timeout

Verify, then recover within the budget

Action record: `R-21` recovery events

The read-only check has its own operation and attempt. Only after its authoritative simulated noncompletion result does the original save operation receive attempt 2.

The verification and bounded retry continue the `R-21` timeline.
RunEventOperationAttemptStatusInput summary
R-2110`check_draft_record`1attemptedSame save operation reference; `request_ref` present
R-2111`check_draft_record`1completedSame save operation reference; no matching record
R-2112`save_draft_record`2attemptedSame save operation reference; unchanged approved draft
R-2113`save_draft_record`2completedSame save operation reference; simulated draft record

Supporting record: `R-21` recovery events

The check is read-only and has not_required approval. The retry preserves the prior granted approval because the proposed save is unchanged.

Evidence and approval history for verification and retry.
RunEventEvidenceApprovalError
R-2110Simulated: read-only destination check dispatchednot_requirednone
R-2111Simulated: authoritative no matching record with immediate visibilitynot_requirednone
R-2112Simulated: bounded retry dispatched after verified noncompletiongrantednone
R-2113Simulated: completion confirmed for retry attemptgrantednone

Check recovery allowances and controls

Counters remain linked to each Run. R-17 used 4 tool calls, 5 service calls, and USD 0.05; its blocked proposal changes none of these. Before the R-21 retry, the completion check brings R-21 to 5 tool calls, 6 service calls, and USD 0.05. The remaining per-run targets are 1 tool call, 2 service calls, and USD 0.03. The retry needs 1 tool call, 1 service call, and USD 0.00.

This recovery-comparison session started at USD 0.04. R-17 and R-21 add USD 0.05 each, so the session estimate after R-21 is USD 0.14, leaving USD 0.06 against USD 0.20. The simulated seventh-call guard is enforced only in this supplied simulation. The session-cost total is a manual estimate requiring supervision, not verified platform enforcement.

Reach a bounded test conclusion

Compare the affected test cases

The two tables below form one compact test record. The first table preserves the required route, action, approval, stop, and observed-result fields. The linked verdict table keeps verdicts explicit without exceeding the table column limit.

Test and regression record

Each row is a separate text-only case. R-19 and R-20 are safeguard cases, not additions to the recovery-comparison session ledger.

Expected behavior compared with supplied trace evidence.
CaseExpected routeExpected actionsExpected approvalExpected stopObserved result
R-16 normal, `v1``hours` normal routeFour operationsgrantedAfter simulated saveRoute and stop matched
R-17 timeout, `v1`Verify before any retryNo second save before checkgrantedAt unknown outcomeDirect retry proposed without verification; blocked before dispatch
R-21 timeout, `v2`Check, then bounded retryCheck plus one save retrygranted unchangedAfter simulated saveAuthoritative no-record check; retry completed
R-19 denied approval, `v2``hours` to approvalSave is `not_run`deniedAt denialNo bypass operation
R-20 limit boundary, `v2`Do not dispatch seventh callAt most 6 tool callsnot_requiredAt guardSeventh dispatch blocked in simulation

Test and regression verdicts

A verdict assesses the supplied test evidence. It is not an action status and does not establish production readiness.

Verdicts linked to the corresponding test cases.
CaseVerdict
R-16 normal, `v1`pass
R-17 timeout, `v1`fail
R-21 timeout, `v2`pass
R-19 denied approval, `v2`pass
R-20 limit boundary, `v2`pass

Summary

Final review sequence

  • Define route, required actions, approval, and stopping expectations before judging a workflow run.
  • Keep every dispatch event and its later evidence separate; preserve uncertainty when completion is not established.
  • For an uncertain state-changing operation, verify completion before deciding whether any retry is safe.
  • Check remaining retries, tool calls, service calls, and estimated cost at the relevant run and session scope.
  • Use a distinct workflow run for a regression test in a changed configuration.
  • Conclude from observable evidence and identify behavior that remains not_verified.

State the evidence-backed conclusion

Final capstone review

Write a concise final review of the supplied Harbor Helpdesk capstone records. Include all of these points:

  1. The status of the first save_draft_record attempt in R-17 and the trace evidence supporting it.
  2. The duplicate-safe decision for R-17, including the status of its proposed retry.
  3. How distinct R-21 in v2 tests the recovery rule, including the verification result and retry decision.
  4. The remaining per-run tool-call, service-call, and estimated-cost allowance before the R-21 retry, plus the session allowance after R-21.
  5. The enforcement classification of the seventh-call guard and the session-cost control.
  6. The regression verdicts for the normal route, timeout repair, denied approval, and limit boundary, with trace evidence.

Identify at least one behavior that remains not_verified. End with an evidence-based conclusion, not a production-certification claim.

Write at least 450 characters (0/450).

Step complete

Congratulations! You completed: Testing Workflow Runs, Retries, and Usage Limits

You completed a safe, evidence-backed run-test review connecting recovery, usage limits, approval safeguards, regression evidence, and remaining uncertainty.

Download the App now to have access to + 5000 free courses, exercises, certificates and lots of content without paying anything!

  • 100% free online courses from start to finish

    Thousands of online courses in video, ebooks and audiobooks.

  • More than 60 thousand free exercises

    To test your knowledge during online courses

  • Valid free Digital Certificate with QR Code

    Generated directly from your cell phone's photo gallery and sent to your email

Cursa app on the ebook screen, the video course screen and the course exercises screen, plus the course completion certificate