Learning path · Level 9 · Tutorial 3

Designing a Bounded AI Pilot

Write a bounded pilot plan that can establish whether a proposed approach improves the current process without concealing review effort, operating costs, or quality failures. Keep the work at the planning stage rather than building or testing the final solution.

  • Level: Advanced
  • Duration: 25 min
  • 8 steps
Start tutorial Back to the path
Designing a Bounded AI Pilot

What you will go through

  1. Define the Pilot's Boundaries Define a bounded pilot charter for Cedar Office Supplies before collecting evidence or changing a live process. 3 min
  2. Specify the Current-Process Baseline Define a baseline record for the current process so that comparable completed work, rework, waiting, and unfinished effort remain visible. 3 min
  3. Plan a Comparable Measurement Plan a representative, paired comparison that makes future pilot evidence interpretable without running the pilot. 4 min
  4. Set Benefit, Quality, and Safety Gates Define measurable benefit, quality, recurring-capacity, and safety gates so faster output alone cannot establish pilot success. 3 min
  5. Account for the Full Effort and Cost Create a transparent inventory of pilot and operating effort, separating staff time from cash spending and avoiding hidden or duplicated work. 3 min
  6. Model Volume and Interpret the Savings Estimate recurring staff effort and cash spending under Low, Expected, and High monthly demand scenarios, while distinguishing time released from cash savings that can actually be avoided. 4 min
  7. Predeclare the Review Decision Predeclare how Cedar Office Supplies will interpret pilot evidence so that safety gates, comparison limitations, and resource limits remain visible in the final review. 3 min
  8. Complete and Check the Pilot Plan Synthesize the Cedar Office Supplies pilot charter into a concise, internally consistent evidence-gathering plan, then check its claims and limits before any later pilot is run. 4 min

What you will learn

  • Define a pilot's hypothesis, scope, participants, duration, exclusions, and resource limits.
  • Specify how to measure a current-process baseline and comparable pilot outcomes using a representative sample.
  • Set measurable benefit targets alongside minimum quality and safety thresholds.
  • Estimate one-time and recurring effort and costs using a plain-language table, distinguishing time savings from cash savings.
  • State in advance what evidence would justify considering wider adoption, further investigation, revision, or stopping.

Before you start

  • Prioritizing AI Use Cases for a Team
  • Comparing AI and Non-AI Solution Options
  • Building an Evidence-Based Answer Test Set
  • Testing Workflow Runs, Retries, and Usage Limits

Step 1 of 8

Define the Pilot's Boundaries

Define a bounded pilot charter for Cedar Office Supplies before collecting evidence or changing a live process.

Frame the exercise before defining it

A pilot is not a rollout

A pilot is a limited evidence-gathering exercise, not a commitment to deploy a solution. Its charter makes the limits visible before any evaluation begins.

For Cedar Office Supplies, the proposed change is specific: the current process has support staff draft routine returns-policy replies; the AI-assisted process uses an existing assistant to draft from approved policy documents for human review.

A useful improvement hypothesis connects that change to an observable benefit while preserving constraints. For example: the AI-assisted process may reduce active staff handling effort for the permitted task, provided that human review remains in place and quality and safety boundaries are respected.

This tutorial creates a planning document only. It does not build, test, deploy, or authorize the assistant.

What a bounded charter makes visible

A pilot charter diagram showing hypothesis, scope, people, time, exclusions, separate resource limits, and internal replay with no customer sending.

A charter links the question being investigated to explicit operating boundaries.

Tip

Planning is not execution

Keep two boundaries separate:

  • This tutorial: write the pilot charter.
  • A future pilot, if approved: evaluate the existing assistant within the charter.

Do not describe a planned pilot as though it has already produced results. Label assumptions and limits clearly, and do not imply a production change.

Read the example charter as a set of limits

Example

Example: a bounded pilot charter

Worked pilot charter — Cedar Office Supplies

Question and hypothesis: Planning estimate — Using the existing assistant to draft permitted replies from approved policy documents may reduce active handling effort compared with the current process, while retaining human review and the established safety boundaries.

Task scope: Routine returns-policy replies only. Planning estimate — 40 unique, authorized, de-identified historical cases: 24 straightforward cases, 12 multi-part cases, and 4 insufficient-information cases. This is a planning assumption, not a claim about actual demand.

Participants and duration: Planning estimate — 2 support staff over 2 weeks. Accountable pilot contact: Cedar Support Operations Lead.

Delivery boundary: Internal replay only. No replies are sent to customers and no production process changes.

Exclusions: Account-specific requests, refund decisions, disputes, and legal claims.

Resource limits: Planning estimate — no more than 24 staff-hours across all pilot activities and no more than USD 100.00 in incremental cash spending.

Charter fields to make explicit

A charter should state each boundary in plain language. The values below are planning estimates for this example, not measured outcomes.

The limits constrain the proposed evidence-gathering exercise.
Charter fieldWhat Cedar Office Supplies specifies
Change under reviewExisting assistant drafts permitted replies; staff review them
Included workRoutine returns-policy replies from approved policy documents
ParticipantsPlanning estimate: 2 support staff
DurationPlanning estimate: 2 weeks
WorkloadPlanning estimate: 40 unique historical cases under internal replay
Excluded workAccount-specific requests, refund decisions, disputes, legal claims
Staff-time ceilingPlanning estimate: 24 staff-hours across all pilot activities
Cash ceilingPlanning estimate: USD 100.00 incremental cash spending

Practice: write the charter

Draft the bounded charter

Using the Cedar Office Supplies planning constraints from the preceding screen, draft a compact pilot charter in 120–180 words.

Include:

  • a testable improvement hypothesis comparing the current process and AI-assisted process;
  • the permitted task scope and internal-replay delivery boundary;
  • participants, duration, and accountable pilot contact;
  • exclusions;
  • separate staff-time and incremental cash ceilings; and
  • a clear statement that this is a planned evaluation, not a build, deployment, or customer-facing activity.

Do not invent results or claim that the proposed approach has already improved the process.

Write at least 300 characters (0/300).

Step 2 of 8

Specify the Current-Process Baseline

Define a baseline record for the current process so that comparable completed work, rework, waiting, and unfinished effort remain visible.

Start and finish the same way

Measure the work that exists now

A current process baseline measures how comparable work is completed today, without the proposed change. It is evidence to collect, not a story about how the process usually feels.

For Cedar Office Supplies, record what staff do when preparing routine returns-policy replies from approved policy documents under the current process. A draft alone is not enough: the record follows the work to a defined usable outcome.

Planning estimate: current active handling time is 10 min/case. Keep this explicitly marked as an estimate until baseline records produce a measured mean.

A workflow shows active staff work ending in either a ready reply, a clarification, or an escalation, while waiting time is shown on a separate timeline.

Define the same start and finish boundaries for every baseline record. A valid clarification or escalation can be a completed case.

Keep handling and waiting separate

Use a usable outcome as the boundary

Set an operational boundary before collecting times:

  • Start: a staff member opens an assigned case and begins task-specific work.
  • Finish: a policy-supported reply is ready for the normal next step, or an appropriate clarification or escalation is documented and ready for that step.

Count active handling from start to finish: reading the case, finding policy support, drafting, checking, correcting, and any rework. If work stops before a valid finish, record its effort and status; do not make it disappear.

The planned pilot uses internal replay, so no reply is sent to a customer. That delivery boundary does not change what counts as a completed case.

Separate the clocks

Active staff time and elapsed time answer different questions. Keep them in separate fields; do not add them together as though both were staff effort.

Recording rules for baseline timing
ObservationRecord asReason
Staff reads policy, checks a draft, and fixes an errorActive handling timeThis is staff work, including rework
A case sits in a queue before work beginsWaiting timeNo staff member is actively handling it
A case waits for a permitted internal inputElapsed turnaround or waiting timeUseful service context, but not automatically staff effort
Staff returns after a wait and continues workAdditional active handling timeCount the work when it occurs

Record the evidence, not only the successes

Make omitted work visible

A baseline record needs enough detail to show both the outcome and the effort behind it. For each started case, record:

  • task type;
  • completion status and route: ready reply, clarification, escalation, or unfinished;
  • active handling time, including corrections and rework;
  • elapsed turnaround or waiting time as a separate measure;
  • whether an error was found at the first check, plus the rework performed; and
  • the reason if work remains unfinished.

Calculate mean active handling time using completed cases, but report the completion rate alongside it: completed cases divided by started cases. This prevents a low average from concealing cases that were abandoned, deferred, or excluded after consuming staff effort.

Baseline record structure

Use one row per started case. The exact task categories and case set will be planned next; this is the record structure, not a claim about actual demand.

Fields to collect before comparing another process
FieldWhat to captureStatus
Task typeNature of the returns-policy requestTo measure
Completion statusReady reply, clarification, escalation, or unfinishedTo measure
Active handlingAll staff work from start through correctionTo measure
First-check error and reworkWhether a correction was needed and the work performedTo measure
Waiting or turnaroundElapsed delay recorded separately from staff workTo measure
Current meanMean active handling time per completed caseTo measure
10 min/caseInitial current-process assumption onlyPlanning estimate

Apply the baseline rules

Check a defensible baseline

Match each hypothetical baseline observation to the correct recording decision.

Tap an item and then its matching pair.

Step 3 of 8

Plan a Comparable Measurement

Plan a representative, paired comparison that makes future pilot evidence interpretable without running the pilot.

Build a reproducible workload sample

Define the eligible pool before selecting cases

A comparable measurement starts with the intended workload, not with cases that seem likely to make the AI-assisted process look good.

First, define an eligible case pool: authorized, de-identified historical cases available for internal replay that concern routine returns-policy replies and can be assessed under the shared completion rules. Exclude account-specific requests, refund decisions, disputes, legal claims, duplicate records, and records that cannot be authorized or de-identified. Record every exclusion and its reason before comparing either process.

Classify the eligible pool before observing process outcomes. The planned mix includes straightforward, multi-part, and insufficient-information cases. An insufficient-information case remains eligible: an appropriate clarification or escalation can be a completed case.

A documented sample begins with a defined pool

A diagram showing eligible historical support cases screened, grouped by task category, selected from numbered lists, and sent through two parallel process lanes.

Planning estimate: select 40 unique cases from a defined eligible pool—24 straightforward cases, 12 multi-part cases, and 4 insufficient-information cases. Each selected case is assessed under both processes.

Select within categories without convenience picking

Check the planned mix against available workload records before calling it representative. Compare the category definitions and the planned proportions with the records, and document any mismatch. The fixed mix is a Planning estimate for this pilot, not an established description of Cedar's actual demand.

Use a repeatable selection rule within each category. Number the eligible cases after sorting them by a stable record identifier. Before either process is assessed, create and save a random ordering for each category. Select the first 24, 12, and 4 cases from those saved orderings. Keep the pool list, category decision, ordering, and selection date in the planning record.

If a selected record is later found objectively ineligible under the predeclared rules, retain its exclusion record and take the next unused record from that category's saved ordering. Do not replace a case because it was slow, required clarification, produced a weak draft, or was unfinished: those are outcomes to record, not reasons to improve the sample.

Design the paired comparison

Pair the work and counterbalance the order

Use the same 40 unique cases under the current process and the AI-assisted process. Apply the same completed-case definition and the same evaluation rubric in both runs. This keeps the comparison focused on the process change rather than different cases or standards.

Repeated exposure can distort active handling time: a staff member may be faster on a familiar case regardless of the process used. Planning estimate: with 2 support staff, divide each task category across both staff members and counterbalance which process comes first. For example, distribute the 24 straightforward cases evenly between staff and have half begin with each process. Record staff member, category, process order, and case sequence.

Keep approved policy source material and tool configuration fixed where possible. Log relevant differences such as practice time, source version, access problems, or configuration changes. These records do not remove every difference, but they make possible distortions visible during review.

Example

Worked planning example

Planned comparison procedure — not a result

  1. Create the authorized, de-identified eligible pool, classify each record, and preserve documented exclusions.
  2. Save a randomized ordering within each category. Select the Planning estimate of 24 straightforward, 12 multi-part, and 4 insufficient-information cases; use the next saved record only for a documented eligibility replacement.
  3. Assign every selected case to both the current process and the AI-assisted process. Use the shared completion rules and evaluation rubric.
  4. Balance categories, staff assignments, and first-process order across the 2 support staff. Vary case sequence rather than completing all cases in one process first.
  5. For each run, record active handling time, completion outcome, rubric result, process order, and any changed condition.

Internal replay remains the delivery boundary: no reply is sent to a customer and no production process is changed.

Plan reporting before the replay

Preserve evidence and limitations

Plan the evidence record before internal replay begins. Report outcomes by task category as well as for the full planned sample, because an overall average can conceal a category-specific difference.

Keep extra risk-focused challenge cases separate from the representative-workload summary. They can expose an edge case or boundary concern, but they do not belong in the workload mix unless they were selected through the same documented rule.

State limitations beside the results. A Planning estimate of 40 cases is a small sample; internal replay does not test live customer interaction; and unobserved conditions may differ from those in the future. These limitations qualify the evidence—they do not erase the records collected.

Evidence to collect

Use one shared record format so a later review can trace selection, comparison conditions, and outcomes.

Plan these records before the internal replay begins.
EvidenceWhat to captureResponsible personWhen
Eligible poolCase identifier, category, eligibility decision, exclusion reasonPilot contactBefore selection
Selection logSaved category ordering, selected cases, documented replacementsPilot contactBefore replay; when replaced
Paired process recordCase identifier, staff member, process, process order, sequenceAssigned support staffDuring replay
Comparable outcomeCompletion status, active handling time, rubric result, unfinished-work reason if applicableAssigned reviewerFor each assessed case
Condition logPractice time, source version, configuration, access issue, or other differencePilot contactWhen a difference occurs
Challenge-case recordRisk-focused case and finding, reported outside workload summaryAssigned reviewerIf challenge cases are used
Attempt recordModel attempts recorded separately from completed casesAssigned support staffDuring AI-assisted replay

Apply the measurement design

Draft a defensible measurement plan

For Cedar Office Supplies, write a concise measurement plan using the fixed Planning estimate of 40 unique cases and the stated task mix.

Use three labeled bullets:

  • Sample: define the eligible pool, the within-category selection rule, and the documented exclusion/replacement rule.
  • Comparison: state how the same cases will be assessed under both processes and explain how staff assignment and process order limit selection or learning effects.
  • Limitation: identify one limitation that must accompany reported results.

Keep this at the planning stage. Do not claim that the pilot has run or that either process achieved a result.

Write at least 250 characters (0/250).

Step 4 of 8

Set Benefit, Quality, and Safety Gates

Define measurable benefit, quality, recurring-capacity, and safety gates so faster output alone cannot establish pilot success.

Use gates, not a blended score

Independent gates

A pilot must not pass because it is faster alone. Predeclare separate gates for benefits, quality, and safety. Each gate must pass independently; do not average them into one score.

For Cedar Office Supplies, the AI-assisted process is an existing assistant that drafts permitted routine returns-policy replies from approved policy documents for human review. The primary benefit gate tests whether active handling time is lower than in the observed current process. A second benefit gate checks whether recurring staff capacity is actually released after the full recurring AI-assisted effort is counted.

A fast result cannot compensate for weak first-pass quality, an observed safety-boundary breach, or negative recurring staff time released.

Four gates, not one blended score

A workflow must clear four separate gates for handling time, recurring staff capacity, quality, and safety.

A pilot needs evidence for every independent gate; speed does not outweigh a failure elsewhere.

Define every gate before results exist

Specify the calculation and evidence

Specify each gate before results exist. State the observed quantity or numerator, denominator or comparison point, unit, evidence source, and threshold.

Use the observed current-process baseline, not an unverified estimate, for the handling-time comparison. Planning estimate: the current process takes 10 min/case and the AI-assisted process takes 7 min/case. These inputs help plan the pilot; they do not show that a gate has passed.

For recurring staff time released, compare equivalent recurring work at the same stated monthly volume. Its calculation will include AI-assisted handling and the separate recurring maintenance and support effort. The detailed Low, Expected, and High projections come later.

Cedar's predeclared gate specifications

The same completed-case rules and evaluation rubric apply to the current process and AI-assisted process. Thresholds are requirements To measure, not results.

Each requirement is assessed independently during the planned comparison or stated recurring scenario analysis.
GateObserved quantity or numeratorDenominator or comparisonUnitEvidence sourceRequirement
Handling-time benefitCurrent mean active handling time minus AI-assisted meanObserved current-process mean active handling time% reductionTimed records for completed casesTo measure: at least 20% lower
Recurring-capacity benefitCurrent recurring staff effort minus total AI-assisted recurring staff effortEquivalent stated monthly volumestaff-hours/monthScenario assumptions and recorded effortTo measure: positive net recurring staff time released
First-pass qualityEligible case runs whose first work product passes the rubric without correctionAll eligible case runs for that process% first-pass qualityShared rubric recordsTo measure: at least 95% for AI-assisted process and no decline from current-process rate
SafetyObserved boundary breachesObserved count; no rate denominatorcountHuman-review and boundary logTo measure: 0 observed breaches

Do not let correction conceal a failure

Keep failures and corrections visible

First-pass quality asks whether the initial work product meets the shared rubric before a reviewer corrects it. The rubric checks factual correctness, support in approved policy documents, completeness, and an appropriate clarification or escalation when information is insufficient.

Use like-for-like denominators. For the AI-assisted process, the denominator is all eligible AI-assisted case runs, including a run where the first model attempt produces no usable output. A no-usable-output first attempt is a first-pass failure, even if a retry or manual fallback later produces a completed case.

For the current process, the denominator is all eligible current-process case runs. Its numerator is the initial current-process work products that pass the same rubric without correction. Compare the two rates; the AI-assisted rate must not decline from the observed current-process rate.

Record stages separately

A completed case and a first-pass-quality pass answer different questions. Record both, rather than allowing later human work to hide an initially weak result.

Appropriate clarification or escalation can be a completed case, but its initial work product still receives a rubric result.
Record stageWhat to recordWhy it matters
AI-assisted first work productRubric pass or failure; no usable first model attempt is a failureMeasures initial AI-assisted quality
Current-process first work productSame rubric pass or failure before correctionCreates a like-for-like no-decline comparison
Post-review outcomeCompleted, clarified, escalated, or unfinishedShows what happened after review, correction, retry, or fallback
Safety checkAny boundary breach and when it was identifiedProtects non-negotiable limits

Protect the pilot boundary

Warning

Safety boundaries are non-negotiable

Human review is required for every completed reply. The planned pilot uses only authorized, de-identified historical cases and approved policy documents in internal replay. No reply is sent to a customer.

To measure: 0 observed safety-boundary breaches is a required result for this sample. It does not prove that the process is risk-free beyond the cases and conditions observed. Any observed breach requires an immediate pause for review; a favorable handling-time result does not override that boundary.

Check an independent-gate result

Apply the gate specification

Complete the metric specification and apply the independent-gate rule.

markdown
Benefit metric:
((observed current-process mean active handling time − observed AI-assisted mean active handling time) / ) × 100
Requirement: To measure — at least 20% lower

AI-assisted first-pass quality:
(first work products passing the rubric without correction / ) × 100

Current-process first-pass quality:
(first work products passing the same rubric without correction / ) × 100
Requirement: To measure — at least  for the AI-assisted process, with no decline from the observed current-process rate

Net recurring staff time released requirement: To measure — 

A pilot is faster but has one observed safety-boundary breach.
Status: 

Step 5 of 8

Account for the Full Effort and Cost

Create a transparent inventory of pilot and operating effort, separating staff time from cash spending and avoiding hidden or duplicated work.

Inventory every activity

Make all work visible

A complete inventory makes the pilot’s resource limits testable. It does not try to prove savings yet.

For every activity, record whether it is one-time or recurring; staff effort or cash outlay; fixed or volume-dependent; and included in the pilot ceiling, a routine-operation projection, or neither.

Include the work that can otherwise disappear: setup, training, both process replays, human review, corrections, retries, manual fallback, measurement, coordination, maintenance, and support. A missing activity can make an approach appear more efficient than it is.

For Cedar Office Supplies, the planned pilot assesses the same 40 unique cases under both the current process and the AI-assisted process. Both assessments consume pilot staff time.

See the whole inventory

Diagram showing both current-process and AI-assisted case handling flowing into pilot staff effort, with setup, measurement, subscription, usage, and recurring support shown separately.

Count both process replays in the pilot; keep recurring routine operation separate.

Tip

Separate the accounting boundaries

Keep three boundaries separate:

  1. Pilot-only work: setup, training, current-process replay, AI-assisted replay, measurement, and coordination. These count toward the Planning estimate of 24 staff-hours.
  2. Routine operation: future case handling, maintenance, support, subscriptions, and usage. Record these separately; they are not pilot-only work.
  3. Unchanged shared work: an existing cost that does not change between processes. Record it for context, but do not label it an incremental AI cost.

The delivery boundary is unchanged: the planned work uses authorized, de-identified historical cases in internal replay. No replies are sent to customers.

Build a traceable inventory

Use one traceable structure

Use a plain-language table so another reviewer can trace each total back to an activity and assumption. All numerical entries below are Planning estimates, not pilot findings.

Pilot and routine-operation inventory

Planning estimate for Cedar Office Supplies. “Included” identifies which ceiling receives the item; it does not make staff effort an additional cash payment.

The pilot includes both process replays. Routine monthly operation is recorded separately.
ActivityTimingQuantity, rate, or unit priceStatusCeiling treatment
Setup and trainingOne-time pilot work6 staff-hoursPlanning estimate; staff effortInclude in 24 staff-hours
Current-process replay handlingPilot work40 cases × 10 min/case = 6.67 staff-hoursPlanning estimate; includes checking, corrections, and reworkInclude in 24 staff-hours
AI-assisted replay handlingPilot work40 cases × 7 min/case = 4.67 staff-hoursPlanning estimate; includes preparation, review, corrections, retries, and manual fallbackInclude in 24 staff-hours
Measurement and coordinationPilot-only workEstimate before launchTo measure or estimate; do not omitInclude in 24 staff-hours
Subscription for 2 usersFull monthly billing period2 × USD 15.00/user/month = USD 30.00Planning estimate; cash outlayInclude in USD 100.00
Model usagePilot use48 model attempts × USD 0.05 = USD 2.40Planning estimate; retries and unsuccessful attempts countInclude in USD 100.00
Maintenance and supportRecurring routine operation2 staff-hours/monthPlanning estimate; staff effort outside pilot-only totalProject separately
Unchanged shared systemExisting operationNo incremental change identifiedTo identify for contextRecord separately

Prevent omissions and double counting

Example

Reserve capacity before launch

Check the ceilings with the work already quantified. The two process replays must be counted separately because the same cases are assessed twice. However, each process’s full handling estimate should be counted only once.

Planning estimate — quantified pilot staff effort: 6 staff-hours for setup and training + 6.67 staff-hours for current-process replay + 4.67 staff-hours for AI-assisted replay = 17.33 staff-hours. This reserves at most 6.67 staff-hours within the 24 staff-hour ceiling for measurement and coordination.

Planning estimate — quantified pilot cash: USD 30.00 subscription + USD 2.40 model usage = USD 32.40. This leaves USD 67.60 within the USD 100.00 incremental cash ceiling. Confirm that planned measurement and coordination fit the reserved staff capacity before the pilot starts.

Count once; label honestly

Avoid two common distortions.

Do not double count bundled handling work. The Planning estimate of 7 min/case for the AI-assisted process already includes preparation, human review, corrections, retries, and manual fallback. Name those activities so they remain visible, but do not add separate review or retry hours to that same per-case total.

Do not convert staff value into cash spending. At USD 30.00/staff-hour, the Planning estimate of 6 staff-hours for setup and training has a staff-effort value of USD 180.00. That value is useful for understanding effort, but it is not automatically an additional cash payment.

For a later comparison, retain the relevant current-process effort and identify costs shared by both processes. An unchanged shared cost is not an incremental AI cost.

Warning

Check the billing rule

A two-week pilot does not necessarily mean a half-month subscription. Cedar’s Planning estimate assumes a full monthly charge: 2 users × USD 15.00/user/month = USD 30.00. Confirm billing periods, authorized-user limits, and model-usage rules before treating access as free or prorated.

Practice: retain every necessary activity

Classify without hiding or duplicating work

Match each inventory entry to its correct treatment.

Tap an item and then its matching pair.

Step 6 of 8

Model Volume and Interpret the Savings

Estimate recurring staff effort and cash spending under Low, Expected, and High monthly demand scenarios, while distinguishing time released from cash savings that can actually be avoided.

Set up three transparent scenarios

Keep the projection separate from the pilot

The planned internal replay remains bounded; these monthly scenarios do not enlarge its sample or produce pilot results. They are Planning estimates for recurring operation at three stated volumes.

Use Low, Expected, and High rather than one optimistic demand total. Each scenario assumes that every case is eventually a completed case, including a case completed through manual fallback. If later evidence shows unfinished work, report its count and completion rate separately. Do not make unresolved cases disappear from the operating picture.

Use explicit scenario assumptions

A model attempt is not a completed case. Planning estimate: use 1.2 model attempts/case, including unsuccessful attempts and retries. Attempts determine billable usage; completed cases determine case-handling volume.

Planning estimate: 7 min/case for the AI-assisted process already includes preparation, human review, corrections, retries, and manual fallback. Do not add separate hours for those activities again. Planning estimate: the current process is 10 min/case.

Monthly scenario assumptions

All numerical values are Planning estimates.

Operating projections for Cedar Office Supplies, not expansions of the 40-case pilot sample.
ScenarioCompleted cases/monthModel attempts/caseCurrent processAI-assisted process
Low1001.210 min/case7 min/case
Expected3001.210 min/case7 min/case
High6001.210 min/case7 min/case

Separate cases, attempts, staff effort, and cash

A three-scenario flow diagram separating completed cases from model attempts, staff effort, fixed support, subscriptions, and usage charges.

Completed cases drive handling effort. Model attempts drive usage charges. Fixed support is added once per month, not once per case.

Calculate staff effort and recurring cash separately

Calculate the recurring measures

Use the same calculation structure for each volume. Convert minutes to staff-hours by dividing by 60. Planning estimate: add 2 staff-hours/month for recurring maintenance and support after calculating AI-assisted case handling.

The recurring-capacity gate remains separate from the handling-time gate: To measure — net recurring staff time released must be positive. For each stated equivalent volume, calculate current-process staff effort minus total AI-assisted recurring staff effort. A positive Planning estimate is not evidence that the gate has passed; observed handling and support effort must later support it.

Projection formulas

All formulas use Planning estimates. Display staff-hours to two decimal places when needed.

Apply these formulas consistently to every monthly scenario.
MeasureFormulaPurpose
Paid model attemptscompleted cases × 1.2Counts unsuccessful attempts and retries
Current-process staff effortcompleted cases × 10 ÷ 60Comparable monthly handling effort
AI-assisted case handlingcompleted cases × 7 ÷ 60Includes review, corrections, retries, and fallback
AI-assisted total staff effortAI-assisted case handling + 2Adds fixed maintenance and support
Net recurring staff time releasedcurrent-process effort − AI-assisted totalRequired to be positive; reusable capacity
Recurring cash spendingUSD 30.00 + (model attempts × USD 0.05)Subscription plus billable usage

Three-scenario operating projection

All values are Planning estimates. The final column tests the direction of the recurring-capacity calculation only; it is not a measured gate result.

Staff effort includes fixed maintenance and support; cash is recurring incremental spending.
ScenarioModel attempts/monthCurrent staff-hours/monthAI staff-hours/monthTime releasedAI cash/month
Low12016.6713.673.00 staff-hours/month; positive projectionUSD 36.00
Expected36050.0037.0013.00 staff-hours/month; positive projectionUSD 48.00
High720100.0072.0028.00 staff-hours/month; positive projectionUSD 66.00

Tip

Do not hide effort or incomplete work

Keep incomplete work visible. These scenarios assume eventual completion, including manual fallback. If that assumption fails in a later evaluation, report unresolved cases and completion rate beside the per-completed-case figures.

Avoid double counting: the 7 min/case Planning estimate already carries human review for every completed reply, corrections, retries, and manual fallback. Naming this work makes it visible; adding it again as separate case-handling time inflates the projection.

Interpret the first period and savings claims

Show the first-period effect separately

Planning estimate: setup and training require 6 staff-hours once. At the Planning estimate of USD 30.00/staff-hour, this has a staff-effort value of USD 180.00. It is not automatically an additional cash payment.

Show this one-time effect separately. For the first period in which setup occurs, subtract 6 staff-hours from recurring time released. This first-period view does not replace the required recurring gate, which concerns positive net recurring staff time released after routine recurring effort is counted.

Recurring capacity versus first-period capacity

All values are Planning estimates. “After setup” includes the one-time 6 staff-hours in that period.

Staff effort only; none of these figures claims payroll savings.
ScenarioCurrent effortAI-assisted recurring effortRecurring time releasedTime released after setup
Low16.67 staff-hours/month13.67 staff-hours/month3.00 staff-hours/month-3.00 staff-hours/month
Expected50.00 staff-hours/month37.00 staff-hours/month13.00 staff-hours/month7.00 staff-hours/month
High100.00 staff-hours/month72.00 staff-hours/month28.00 staff-hours/month22.00 staff-hours/month

Warning

Do not turn capacity into unsupported cash savings

Hypothetical claim: “The Expected scenario releases 13.00 staff-hours/month, so Cedar saves USD 390.00/month.” This is unsupported merely because 13.00 × USD 30.00 equals USD 390.00.

Call 13.00 staff-hours/month time released: it is potential reusable capacity. Call something cash savings only when a credible expense can actually be avoided, such as documented reduced paid overtime. If no avoidable expenditure is identified, report recurring cash spending and time released separately.

Practice: complete and interpret the projection

Practice: complete and interpret the projection

Complete the selected cells. Use digits and two decimal places where the worksheet shows currency or rounded staff-hours.

markdown
Projection worksheet — all figures are Planning estimates

LOW — 100 completed cases/month
paid model attempts = 
AI-assisted total staff effort = (100 × 7 ÷ 60) + 2 =  staff-hours/month

EXPECTED — 300 completed cases/month
paid model attempts = 
recurring cash spending = USD 30.00 + (attempts × USD 0.05) = USD /month

HIGH — 600 completed cases/month
AI-assisted total staff effort = (600 × 7 ÷ 60) + 2 =  staff-hours/month

Cash savings conclusion when no avoidable cash expenditure is identified: 

Step 7 of 8

Predeclare the Review Decision

Predeclare how Cedar Office Supplies will interpret pilot evidence so that safety gates, comparison limitations, and resource limits remain visible in the final review.

Predeclare the decision logic

Decide before seeing results

A review decision should be predeclared: write the evidence requirements before results exist. This prevents a promising-looking result from changing the definition of success after the fact.

For Cedar Office Supplies, the review compares the measured current process with the AI-assisted process against the declared benefit, quality, safety, and resource requirements. It does not authorize a rollout. It selects the appropriate next decision for this bounded pilot.

Predeclared review logic

A review flow in which evidence is assessed before four possible outcomes are selected, with a safety alert leading first to a separate pause point.

An immediate pause protects the pilot while evidence is reviewed. It is not one of the four review outcomes.

Tip

Safety comes first

If an observed safety-boundary breach occurs, pause the pilot immediately for review. Do not continue merely to collect a larger sample. After the evidence is examined, the later review can determine whether the appropriate outcome is Revise or Stop.

Record the accountable pilot review contact and set a decision date before the pilot begins. For this example, the accountable contact is the Cedar Support Operations Lead.

Link evidence to an outcome

Link evidence to a proportionate outcome

Use exactly four review outcomes. A short review record should connect the selected outcome to baseline comparisons, gate results, resource use, limitations, and unresolved questions. This makes the reasoning inspectable rather than based on an overall impression.

Four predeclared review outcomes

Use these conditions before the pilot begins. They are decision rules, not predictions about what Cedar Office Supplies will observe.

“Consider wider adoption” is a limited next-stage consideration, not an immediate rollout authorization.
OutcomeEvidence basisWhat the review record must state
Consider wider adoptionAll required gates pass; resource use is acceptable; limitations do not undermine the bounded comparisonBaseline comparison; threshold results; staff effort and cash spending; limitations; unresolved questions
Investigate furtherEvidence is insufficient, or comparison conditions are too unclear to interpret a resultSpecific unanswered question; additional evidence needed; renewed bounded limit
ReviseA remediable weakness requires a changed approach before another evaluationWeakness found; proposed change; reason a new bounded evaluation is needed
StopAn unacceptable safety failure, infeasible resource demand, or no credible benefit is establishedFailure evidence; resource implications; limitations; unresolved questions

Tip

Do not trade one gate for another

For Consider wider adoption, every independent gate must pass: To measure — at least 20% lower mean active handling time; positive net recurring staff time released; at least 95% first-pass quality with no decline from the current-process rate; and 0 observed safety-boundary breaches. Resource use must also remain acceptable.

Use time released for reusable staff capacity. Do not call it cash savings unless the evidence identifies spending that can actually be avoided.

Protect the boundary during the pilot

Apply the immediate safety action

During internal replay, an AI-assisted draft appears to handle an excluded account-specific request. Human review catches it before anything is sent externally. What must happen first?

Choose the required immediate action.

Make the review decision

Assess a hypothetical evidence summary

Hypothetical result — All 40 unique cases reached completed-case status under both processes. The measured current-process mean active handling time was 10 min/case; the AI-assisted mean was 7.5 min/case, a 25% reduction. First-pass quality was 38 of 40 (95%) for each process, and there were 0 observed safety-boundary breaches. Pilot resource use was 22.00 staff-hours and USD 32.40 in incremental cash spending.

For the Expected scenario of 300 completed cases/month, the observed handling means plus 2.00 staff-hours/month for maintenance and support project 50.00 current-process staff-hours/month and 39.50 AI-assisted staff-hours/month. This is 10.50 staff-hours/month of positive net recurring staff time released.

However, process order was not balanced: staff completed the current process first for every case and the AI-assisted process second. Repeated exposure could explain some or all of the time difference.

Name exactly one predeclared review outcome and justify it in 2–4 sentences. Address the safety evidence, the positive recurring time result, and why the comparison issue represents insufficient evidence or demonstrated failure.

Write at least 220 characters (0/220).

Step 8 of 8

Complete and Check the Pilot Plan

Synthesize the Cedar Office Supplies pilot charter into a concise, internally consistent evidence-gathering plan, then check its claims and limits before any later pilot is run.

Assemble a coherent pilot charter

Turn the components into one plan

A complete pilot charter is a connected argument, not a list of attractive estimates. It links one bounded change to a fair comparison, independent gates, full resource accounting, and a predeclared review decision.

For Cedar Office Supplies, keep three kinds of statements visibly separate:

  • Known for this plan: the existing assistant drafts routine returns-policy replies from approved policy documents; every completed reply receives human review; work uses authorized, de-identified internal replay; no message is sent to a customer.
  • Planning estimate: a proposed scope, duration, workload mix, effort, cost, or operating projection used to design the pilot.
  • To measure: the observed current-process and AI-assisted outcomes, quality, completion, resource use, and any safety-boundary breach.

This distinction prevents a plan from presenting assumptions as results.

The pilot plan as a connected system

A visual map showing boundaries, comparison, quality and safety, resources, and decision rules connected through one evidence record.

Use one coherent plan: boundaries determine the cases; the same completion rules support comparison; measured evidence is checked against gates; resource use and limitations inform the review decision.

Tip

Keep the claim bounded

A bounded plan does not claim that the assistant works, creates cash savings, or is ready for rollout. It states what a later evidence-gathering exercise could answer—and what it cannot answer outside its replay sample and stated conditions.

Run the internal consistency check

Check boundaries, arithmetic, and measures

Before finalizing, test the internal links. The same definition of a completed case must apply to both processes; an appropriate clarification or escalation can count as completed. The AI-assisted active-time estimate already includes preparation, human review, corrections, retries, and manual fallback, so do not add those activities again.

Check that the proposed work fits the ceilings before the pilot begins. The figures below are planning calculations, not results. If measurement and coordination need more than the remaining staff time, reduce the scope or revise the ceiling before proceeding.

Pilot consistency checks

Each row is a pre-pilot check. Numerical requirements are marked To measure: they are predeclared requirements, not observed results.

Planning estimates and predeclared measures for the bounded pilot.
CheckRecord or calculationPlanning conclusionStatus
Boundaries2 support staff; 2 weeks; internal replay onlyNo live customer messages or production changesPlanning estimate
Sample40 unique cases assessed under both processes24 straightforward; 12 multi-part; 4 insufficient-informationPlanning estimate
Comparable completionSame start and finish rules and evaluation rubricReport completion rate alongside per-completed-case measuresTo measure
Active handling40 × 10 min/case + 40 × 7 min/case = 680 min11.33 staff-hours across both comparisonsPlanning estimate
Staff ceiling6 staff-hours setup/training + 11.33 staff-hours handling = 17.33 staff-hoursAt most 6.67 staff-hours remain within 24 staff-hoursPlanning estimate
Pilot cashUSD 30.00 subscription + 48 model attempts × USD 0.05 = USD 32.40Within the USD 100.00 incremental cash ceilingPlanning estimate
Handling-time gateMean active handling time at least 20% lowerCompare with the measured current-process baselineTo measure
Recurring-capacity gateNet recurring staff time released is positiveCompare equivalent monthly volumes after maintenance and supportTo measure
Quality and safety gatesAt least 95% first-pass quality; no decline from baseline; human review; 0 observed breachesEvaluate independently from speed and capacityTo measure
Review outcomesFour predeclared outcome labelsLink each outcome to gates, costs, limitations, and unanswered questionsTo measure

Check recurring effort and cash claims

Check recurring projections separately

The handling-time projection includes 2 staff-hours/month for recurring maintenance and support. Cash includes a full USD 30.00 monthly subscription for two users plus billable model attempts. A model attempt is not a completed case.

At the Planning estimate of USD 30.00/staff-hour, 6 setup and training hours have a USD 180.00 staff-effort value. That value is not automatically an additional cash expense, and released time is not automatically cash savings. The positive recurring-time gate must later be supported by measured handling and maintenance effort.

Planning estimate. All cases are assumed eventually completed; each scenario uses 1.2 model attempts/case, including retries and unsuccessful attempts.
ScenarioCases/monthCurrent staff-hours/monthAI staff-hours/monthNet time releasedAI cash/month
Low10016.6713.673.00 staff-hours/monthUSD 36.00
Expected30050.0037.0013.00 staff-hours/monthUSD 48.00
High600100.0072.0028.00 staff-hours/monthUSD 66.00

Tip

Interpret the projection honestly

The projected positive values above are Planning estimates, not proof of a benefit. In a later review, call these values time released only when observed current-process effort exceeds total observed AI-assisted recurring effort at the same volume. Call anything cash savings only when Cedar can credibly avoid an actual expense, such as paid overtime. A staff-hour value alone does not establish an avoided cash payment.

Apply the complete planning framework

Final application: draft the pilot plan

Now write the plan as a concise decision document. Use the supplied Cedar Office Supplies assumptions from the earlier screens; do not invent results.

Your draft must connect the hypothesis and boundaries to the baseline, paired comparison, independent gates, complete effort and cost inventory, and all four review outcomes. Finish by naming one uncertainty that requires evidence rather than assertion.

Write the integrated plan

Write a concise plain-text pilot charter for Cedar Office Supplies. Include:

  1. The improvement hypothesis, permitted task, exclusions, internal-replay boundary, participants, duration, sample, and separate resource ceilings.
  2. What the current-process baseline and AI-assisted process will record, including compatible completed-case rules and comparison controls.
  3. The benefit, quality, and safety requirements and the evidence source for each, including positive net recurring staff time released.
  4. One-time and recurring staff effort and cash estimates, including Low, Expected, and High monthly scenarios. Distinguish time released from cash savings.
  5. All four predeclared review outcomes—Consider wider adoption, Investigate further, Revise, and Stop—and the evidence that would lead to each.
  6. One unresolved planning assumption and the specific evidence needed to resolve it.

Mark assumptions as Planning estimate and future observations as To measure. State what the pilot can answer and one claim it cannot support.

Write at least 700 characters (0/700).

Close at the planning stage

Summary

Final plan check

  • A pilot charter is bounded evidence-gathering work, not deployment, a test result, or rollout authorization.
  • Use the same completed-case rules, workload categories, and evaluation rubric for the current process and the AI-assisted process.
  • Keep Planning estimates, To measure evidence, and any Hypothetical result separate.
  • Speed does not compensate for weak first-pass quality, negative recurring staff time released, or a safety-boundary breach; gates remain independent.
  • Count review, corrections, retries, fallback, maintenance, subscriptions, and billable model attempts without double counting.
  • Time released is reusable staff capacity. Call it cash savings only when a real spending reduction can be avoided.
  • Predeclare exactly four review outcomes so that later evidence cannot move the goalposts.

Tutorial completed

Congratulations! You completed: Designing a Bounded AI Pilot

You have completed the planning stage for a bounded AI pilot. Your deliverable is an evidence-gathering plan: it can guide a later internal replay, but it does not establish results or authorize rollout.

Download the App now to have access to + 5000 free courses, exercises, certificates and lots of content without paying anything!

  • 100% free online courses from start to finish

    Thousands of online courses in video, ebooks and audiobooks.

  • More than 60 thousand free exercises

    To test your knowledge during online courses

  • Valid free Digital Certificate with QR Code

    Generated directly from your cell phone's photo gallery and sent to your email

Cursa app on the ebook screen, the video course screen and the course exercises screen, plus the course completion certificate