Many AI pilots are set up to produce a yes. Someone picks the friendliest workflow and the keenest users, then writes the summary, and you approve a build on evidence someone else controlled. A real pilot is a decision with a kill switch agreed in advance. It answers one question before you spend the build budget.
We build and scope AI pilots through our AI integration service, so weigh that. Every fact below about Anthropic’s API and the NIST AI Risk Management Framework was checked against their own documentation in October 2026.
Short answer: Pick one workflow, one metric and one team. Measure the work as it runs today. Before you start, write down the result that means stop, the result that means build, and what must not get worse. Fix the end date, keep the evaluation in your hands and log the same numbers weekly. Then decide go, fix and re-test, or stop. Model costs are usually small: on Anthropic’s prices, a few thousand Claude Sonnet 5.5 calls cost tens of dollars, and you can cap spend. Price production separately.
What the Official Guidance Says
As of October 2026, from Anthropic’s documentation and NIST:
| Topic | What the source says |
|---|---|
| Success criteria | Should be specific, measurable, achievable and relevant |
| Evals | Mirror the real task, include edge cases, automate grading where possible |
| Grading | Code-based is fastest and most reliable; human grading is flexible but slow and expensive |
| Claude Sonnet 5.5 | $2 per million input tokens, $10 per million output tokens |
| Claude Haiku 5.5 | $0.10 per million input tokens, $0.50 per million output tokens, for prompts up to 100,000 tokens |
| Batch API | 50% off input and output; batches not done in 24 hours expire |
| Spend limits | You can set your own monthly limit below your tier’s cap, and per workspace |
| NIST AI RMF, Measure | Test before deployment and regularly in use; independent review can reduce internal bias |
Sources: Define success criteria and build evaluations, Anthropic pricing, Batch processing, Rate and spend limits, NIST AI RMF 1.0, NIST AI RMF overview.
What a Pilot Is, and What It Isn’t
Three terms get used as if they mean the same thing. They don’t.
Proof of Concept vs. Pilot vs. Full Deployment
A proof of concept asks: can this technology do the task at all? It’s a short technical test on sample data, with no real users and no conclusion about business value.
A pilot asks: will this work for us, with our people, our data and our volume? It runs in a real setting, with real users, against a measured baseline. It ends in a go/no-go decision with evidence behind it.
A full deployment assumes the pilot said yes. Skip the pilot and you’re betting the deployment budget on a proof of concept that only worked under controlled conditions.
Who Runs the Pilot Matters
Many vendors offer to run your pilot for you. The catch is incentive. A vendor wants a yes, so the friendliest workflow and the best measurement window tend to get picked. If they also hold the data and write the summary, they control the result.
That isn’t hostile, just misaligned. NIST’s framework says independent review can improve testing and reduce internal biases and conflicts of interest (source). Whoever runs the evaluation shapes the outcome. Make sure that’s you.
Three Questions to Answer Before You Start
If you can’t answer all three clearly, the pilot will give you an ambiguous result whatever the technology does.
Can You Define the Problem in One Sentence?
“Use AI to improve operations” is not a problem statement. “Our support team re-reads the whole account history before each first response” is. The more specific the sentence, the tighter the pilot and the easier the decision.
Do You Have Baseline Data to Measure Against?
If you don’t know what the current process costs in time and errors, you can’t show improvement. Before you design the pilot, measure the workflow as it runs today, including its normal ups and downs: time per item, error rate and volume.
Anthropic’s guidance says targets should rest on things like prior experiments or expert knowledge (source). Your baseline is that prior experiment.
Is There a Real Cost to Doing Nothing?
If the current process is slow but not a bottleneck, or its errors don’t touch revenue or customers, a successful pilot still might not justify a build. Know what you’re solving for.
How to Structure a Pilot That Produces Honest Results
Scope: One Process, One Metric, One Team
A pilot across several workflows gives results you can’t read. One thing improves, another gets worse, and you can’t tell which effect came from where. Lock the scope: one process, one primary metric, one team. Choose people who do the work every day.
Timeline: Long Enough to Pass the Learning Curve
No standard says how long a pilot must run. The length should come from the work. A pilot that ends while users are still learning the tool measures novelty, not value. Run it until the learning curve flattens and at least one full cycle of the workflow, such as a month-end close, has passed.
Fix the end date and the decision date at the start. A pilot with no end date drifts into a permanent trial nobody approves and nobody stops.
Success Threshold: Set the Bar Before You See Results
This is the hardest rule to keep and the most important. Agree the threshold before the pilot begins and write it down. If you decide what counts as success after the results, every unclear result gets read kindly.
Anthropic’s guidance gives a good test. “Good performance” is not a criterion; a specific metric on a defined test set, against a baseline, is. Most use cases need several criteria, such as accuracy, consistency, response time and price (source).
We set three levels:
- Minimum: below this, we stop. This is the kill switch.
- Target: this justifies a full build.
- Stretch: this changes what we would build.
Guardrail Metrics: What Must Not Get Worse
Guardrail metrics measure what you can’t afford to break. A customer-facing tool might be faster but fail if satisfaction drops. An internal tool might hit its speed target while quality slips downstream. Define two or three guardrails up front. If one is breached, the pilot fails, whatever the primary metric shows.
Test guardrails before real users see the tool. Anthropic recommends evals that mirror your real tasks, including edge cases such as missing input, overly long input and cases where even people would disagree (source). Build that test set from your own past work.
Running the Pilot Without Letting the Vendor Run It
Who Owns the Data and Evaluation
This belongs in the contract. Before the pilot begins, confirm in writing that you own the pilot data and the evaluation, and can share the results. If a partner resists, treat that as information.
For a direct review of vendor contract terms, get in touch.
What to Document Weekly
Keep a short weekly log:
- the primary metric
- each guardrail metric
- how many people in the pilot group used the tool that week
- one verbatim piece of user feedback, good or bad
Don’t summarise feedback until the end. Early summaries smooth out the frictions that show whether a problem is fixable or built in.
Where you can, grade outputs in code. Anthropic calls code-based grading the fastest and most reliable, and human grading flexible but slow and expensive (source).
When to Pause or Stop Early
Two things justify an early stop. The first is a guardrail breach you can’t reverse quickly, such as wrong customer data shown or a compliance line crossed. Stop at once. The second is usage collapse. If most of the group stops using the tool, you’re measuring resistance to adoption, not the technology, and that needs a different fix. Decide in advance what level of usage counts as collapse.
Stopping early isn’t failure. It’s the pilot doing its job.
The Go/No-Go Decision Framework
Scoring Results Against Pre-Set Criteria
At the end, pull the weekly log. Calculate the change in the primary metric against your baseline, check each guardrail and compare with your three levels. The decision should be close to mechanical, because the hard thinking happened when you set the levels. Don’t move the numbers now.
There are three honest outcomes:
- Go: the target was met and no guardrail was breached. Plan the build.
- Fix and re-test: the result landed between minimum and target. Change one specific thing and run a short, defined second round.
- Stop: the minimum wasn’t met, or a guardrail broke. Document what you learned.
The Cost-to-Scale Reality Check
Pilot cost and production cost are different numbers. Production adds work the pilot skipped: connecting to your real systems, error handling, logging, monitoring, real traffic and an owner after launch. Don’t multiply the pilot invoice. Ask for a line-item production estimate.
For businesses on WordPress, much of that cost depends on how the AI tool connects to your site and plugins. A custom WordPress development scoping conversation should cover that before any money goes into production.
How much does an AI pilot project cost for a small business?
The build and evaluation work is the main cost, and it depends on your workflow, data and systems, so we scope each pilot individually. The model cost is easier. Here’s our own worked example from Anthropic’s published prices. Say the pilot handles 2,000 support tickets, each using about 3,000 input tokens and 400 output tokens, on Claude Sonnet 5.5 at $2 and $10 per million tokens (source):
- Input: 6 million tokens, $12
- Output: 800,000 tokens, $8
- Total: about $20
Claude Haiku 5.5 costs a twentieth as much per token on prompts up to 100,000 tokens. For offline test runs, the Batch API gives 50% off; most batches finish within an hour, and one not done in 24 hours expires (source). Count tokens on your own data before relying on the estimate.
To bound the bill, set a spend limit. Anthropic lets you set a monthly limit below your tier’s cap, and limits per workspace; once you hit it, requests are refused until it resets or you raise it (source). A pilot workspace with its own limit avoids surprises.
How to Present Findings to Stakeholders
Use one structure: what we tested, what we measured, what we found, what scaling will cost, and the decision. If the pilot hit the minimum but not the target, say so. A plain, honest result builds more trust than a polished case for a decision already made.
Related Posts
- Phased rollout strategy for a custom AI tool: what happens after the pilot says go
- AI integration time to value: the operational and financial clocks
- AI integration budget: pricing pilot and production as separate lines
- Ownership vs dependency: what your contract should say about code and data
Frequently Asked Questions
How long should an AI pilot project run?
There’s no standard length. Run it until users are past learning the tool and at least one full cycle of the workflow has passed; if the work has monthly peaks, cover two. Fix the end and decision dates first.
What’s the difference between a proof of concept and a pilot?
A proof of concept tests whether the technology can do the task. A pilot tests whether it works for your workflow, users, data and constraints. The first can succeed while the second fails.
How many users should be involved in an AI pilot?
Enough to see patterns, not so many that the pilot becomes a rollout. Usually that’s one team that does the work daily, a realistic mix of the people who would use it at full scale, not only enthusiasts or sceptics.
What should I do if pilot results are ambiguous?
Check whether you set thresholds in advance; if not, that’s usually why. If the result sits between minimum and target, treat it as a conditional no. Consider a short second round with one specific change, such as better data preparation, and write down what you’re testing differently.
What happens to the code and data if the pilot fails?
That depends on the agreement, so settle it before the pilot begins. You should own the pilot code, the prompts, any configuration and all the data generated. If a vendor keeps any of these as a condition, negotiate it out or walk away.
Why do AI pilots fail?
In our experience, the technology is rarely the main cause. The problems are usually upstream: a problem nobody defined in one sentence, no baseline to compare against, data that’s incomplete or hard to reach, people who were never brought in, and success criteria written after the results came in. Each of these is avoidable before the pilot starts.