Most AI projects don’t fail in build, they fail in the conversation after, when nobody can agree on whether the thing actually worked. The acceptance criteria were never written down. The finish line was assumed, not defined. We have seen this pattern enough times that we now refuse to start a build without it in writing.
If your agency or developer tells you your AI integration is “done,” ask them to show you the acceptance criteria. If they can’t produce a document, the project isn’t done, it was abandoned at a convenient moment.
Why “Done” Is Different for AI Projects
Most software has a predictable finish line. You test it, it passes, you ship it. AI doesn’t work that way. The system you test in week three behaves differently in week twelve, because the data it processes changes, the model it depends on updates, and edge cases accumulate.
Treating AI like conventional software leads to handoffs that look complete and break silently.
AI Systems Degrade, Model Drift Is Real
Model drift happens when the real-world inputs your AI processes diverge from the data it was trained or tuned on. A customer classification tool trained on Q1 language starts misfiring in Q4 after a product rebrand. An email triage system calibrated for 200 tickets/day starts hallucinating priorities at 800 tickets/day.
This isn’t a bug. It’s the nature of probabilistic systems operating in changing environments. “Done” for AI must account for it explicitly.
Non-Determinism Means You Can’t Test Once and Ship
Run the same input through a language model ten times and you’ll get ten slightly different outputs. That’s by design. It also means traditional QA, pass/fail on a fixed test suite, is insufficient.
You need statistical acceptance criteria: “The system produces an acceptable output on at least 95% of test cases” is meaningful. “It passed our tests” is not.
The Two-Part Definition of Done
A complete definition of done for an AI integration has two distinct parts. Both must be agreed in writing before the project starts, not negotiated after delivery.
Part 1, The Pre-Launch Finish Line
This is the hard criteria your system must meet before it goes live. It’s a specific, measurable list, not a narrative. It should include numerical thresholds, not descriptions.
Examples of what belongs here:
- Output accuracy rate: ≥ 94% on a representative test set of 200 samples
- Latency ceiling: API response under 3 seconds at the 95th percentile
- Fallback trigger rate: < 8% of requests escalated to manual handling
- Error rate: < 1% hard failures in a 72-hour staging soak test
- Integration test coverage: all connected systems (CRM, email, database) tested end-to-end with real data
These numbers are not universal, they depend on your use case. A customer-facing chatbot has different latency requirements than an internal document processor. But you need actual numbers, agreed before build begins.
Part 2, The Post-Launch Responsibility Transfer
This is the piece most SMBs never receive. After launch, the AI system needs monitoring. Accuracy needs to be tracked. Drift needs to be caught. Fallback triggers need to be reviewed.
The question “who owns this after handoff?” must have a written answer before you sign off. Three outcomes are legitimate:
- You own it, the vendor delivers monitoring dashboards, alert configurations, and documentation. You run it in-house.
- Vendor retains it, a retainer covers ongoing monitoring, model updates, and incident response.
- Shared responsibility, defined split: you watch business metrics, they watch technical metrics.
If your vendor delivers option zero, “it’s built, you’re on your own, good luck”, that’s not a handoff. That’s abandonment.
The Criteria Checklist: What Done Actually Looks Like
A complete definition of done covers four categories. Run through each before you sign off.
Functional Criteria
These are the core performance thresholds the system must hit before launch.
- Accuracy/precision targets: defined and tested on a held-out sample, not just the happy path
- Latency targets: p50, p95, p99 response times documented and verified
- Fallback behavior: what happens when the AI can’t produce a confident answer? Is there a graceful degradation path, or does it fail hard?
- Edge case handling: at least 20 documented edge cases tested, with acceptable behavior defined for each
Operational Criteria
The AI is running; but is it observable?
- Logging: inputs, outputs, and confidence scores logged to a queryable store
- Alerting: automated alerts configured for error rate spikes, latency breaches, and fallback rate increases
- Monitoring dashboard: at minimum, a live view of daily accuracy, error rate, and request volume
- Incident runbook: documented steps for “what to do if the system starts misbehaving at 2am”
Without these, you have no way of knowing when the system stops performing. You’ll find out from a frustrated customer, not a dashboard.
Business Criteria
Technical completion is not the same as business completion.
- Baseline measurement: the metric this integration was meant to improve (hours saved, tickets resolved, error rate reduced) was measured before launch, so you have a comparison point
- User acceptance: the actual humans using the system have completed structured testing and signed off, not just the project lead
- Stakeholder sign-off: a named person with authority has reviewed and approved against the criteria list, in writing
- Documentation: an operations manual exists that a non-technical team member can follow
Compliance and Ethics Criteria
Often skipped for SMB projects. Skipping it creates liability.
- Data handling documentation: which data the AI processes, where it’s stored, and whether it leaves your jurisdiction is written down
- PII exposure check: if the system processes customer data, a named person has confirmed GDPR/CCPA handling is correct
- Bias/fairness check: if the system makes decisions affecting customers (recommendations, pricing, support prioritisation), a documented bias audit has been completed, even a lightweight one
- Model dependency documentation: which third-party model or API the system depends on, what happens if it’s deprecated, and what the migration path looks like
What to Demand From Your Vendor Before You Sign Off
A serious vendor can produce these documents. If they can’t, the project is not done.
The Three Documents That Signal a Serious Vendor
1. The Acceptance Criteria Document A numbered list of pass/fail criteria, including the specific thresholds discussed above, signed by both parties before build begins. Not written after. Before.
2. The Monitoring and Alerting Specification What’s being logged, what alerts are configured, what thresholds trigger them, and who receives them. This should be a technical document, not a slide deck sentence.
3. The Responsibility Transfer Agreement Who owns what after handoff. Who to call when something breaks. What the escalation path is. This can be a single page, but it must exist.
At Designodin, these three documents are produced before a single line of code is written. Not because it’s good practice, because without them, projects drift indefinitely and nobody can agree when they’re done.
Red Flags That Mean the Project Isn’t Actually Done
- “It’s working great in testing” with no documented test methodology
- Success metrics defined after the build, not before
- No monitoring setup, or monitoring that requires manual checking
- “We’ll set up alerts later”, later never comes
- Handoff documentation that’s a README, not an operations manual
- Vendor reluctance to put acceptance criteria in writing
- No answer to “what does failure look like, and who handles it?”
Any one of these in isolation warrants a direct conversation. Several of them together means the project is incomplete, regardless of what the invoice says.
Frequently Asked Questions
What is the definition of done for an AI project?
The definition of done for an AI project is a written, pre-agreed list of measurable criteria that must be met before the project is considered complete. It covers functional performance (accuracy, latency, fallback behavior), operational readiness (monitoring, logging, alerting), business validation (user acceptance, baseline measurement), and responsibility transfer (who owns ongoing monitoring). Agreeing this list before build begins, not after delivery, is what separates projects that succeed from those that stall.
How do you measure success for an AI integration?
Success is measured against the specific metric the integration was designed to improve, and that metric must be recorded before launch, so you have a comparison point. If the AI was built to reduce manual invoice processing time, you measure processing time per invoice before and after. If it was built to resolve support tickets without human intervention, you measure resolution rate and escalation rate. Defining “what improvement looks like” in numbers before the project starts is the only way to know whether it delivered.
What happens if an AI model drifts after launch?
Model drift, where the system’s performance degrades as real-world inputs change, is expected, not exceptional. The response is a monitoring arrangement that catches it early: logged outputs, automated accuracy checks against a reference set, and alerts when performance drops below threshold. Who runs that monitoring (you or your vendor) should be agreed before launch. If no monitoring exists, drift goes undetected until a user complains.
Should a small business accept an AI integration without ongoing monitoring?
No. Even a lightweight monitoring setup, a weekly accuracy spot-check, an alert for error rate spikes, is non-negotiable. AI systems running in production without any observability are technical debt accumulating silently. Whether monitoring is handled by your vendor under a retainer, a third party, or in-house staff, it needs to be assigned to a named person with defined responsibilities before you sign off.
What’s the difference between acceptance criteria and definition of done for AI?
Acceptance criteria are the specific conditions a feature or deliverable must meet, for example, “the document classifier must achieve 93% accuracy on a 150-document test set.” The definition of done is broader: the complete checklist that covers all categories (functional, operational, business, compliance) that must be satisfied before the project is considered finished. Acceptance criteria are inputs into the definition of done. You need both, in writing, before the project starts.
If you’re starting an AI integration and want acceptance criteria agreed in writing before build begins, tell us what you’re working on, we’ll be direct about whether we can help. You can also see how we scope and build this at designodin.com/ai.