Many custom AI builds go wrong before anyone writes code. The brief is a paragraph about a desired outcome, with no input format, no output format and no definition of done. A brief that works states the job, the inputs, the output and how you’ll judge the result, so both sides can see the finish line.
We build custom AI tools through our AI integration service, so weigh that. Every fact below about Anthropic’s guidance and the NIST AI Risk Management Framework was checked against their own pages in October 2026.
Short answer: Write the tool’s job in one sentence. List every input with its format, source and known messy cases. Define the output as named fields. Set specific, measurable success criteria and collect real examples to test against. Name the point where a person approves before anything hard to undo happens, and list what the tool depends on. Anthropic’s documentation puts success criteria and tests before prompt work, which is a good order for a brief too.
What Anthropic’s Documentation Says About Defining Success
As of October 2026, from Anthropic’s documentation:
| Topic | What the documentation says |
|---|---|
| Before prompting | Have clear success criteria, ways to test against them and a first draft prompt |
| Good criteria | Specific, measurable, achievable and relevant |
| Common criteria | Task fidelity, consistency, relevance, tone, privacy, context use, latency and price |
| Test design | Mirror the real task, include edge cases, automate grading where you can |
| Test volume | More test cases with automated grading beat fewer hand-graded ones |
| Grading | Code-based is fastest and most reliable; human is flexible but slow and expensive |
| Structured outputs | Responses can be held to a JSON schema you define |
| Schema limits | A refusal or hitting the token limit can still return output that doesn’t match |
| Model choice | Haiku 5.5 is the model described for high-volume classification, extraction and routing |
Sources: Define success criteria and build evaluations, Prompt engineering overview, Structured outputs, Models overview.
Why Most AI Briefs Fail Before the Build Starts
In our experience, the most common mistake is scoping by capability instead of by job. A client says: “I want it to handle customer service, draft quotes and flag overdue invoices.” That’s three tools in one sentence, each with different inputs, outputs and ways of being wrong. Built as one, nobody can say which part works, because there’s no single task to test.
Anthropic makes the same point for developers. Its prompt engineering guide assumes you already have clear success criteria and a way to test against them, and says to set those up first (source). A brief without success criteria leaves the builder to guess.
The one-sentence test
Before any scoping meeting, write the tool’s job in one sentence: subject, verb, object. “The tool reads inbound lead emails and outputs a CRM entry with company name, job title and deal stage.” That’s a buildable brief. “The tool improves our sales process” isn’t.
If you can’t write the sentence plainly, the project isn’t ready to scope. That’s our rule, not a standard.
How do I know if my scope is too broad?
If the job sentence needs an “and” to join two different outputs, it’s two tools. A tool that “handles customer service and drafts quotes” is two builds. Do them one after the other; the first teaches you what your data really looks like.
The Four Things a Brief Must Contain
A brief that gets built correctly contains four elements. Timelines, tech stack and integrations follow from these.
1. The job, in one sentence
State what the tool does, not what problem it solves. “Solves our document chaos” is a problem statement. “Extracts vendor name, invoice number, amount and due date from uploaded PDF invoices and posts them to our accounting software” is a job statement. The difference decides whether a developer can write a test for it.
2. The exact inputs
Define what data the tool receives, in what format and from what source. “It gets the invoices” isn’t a definition. “It receives PDF files uploaded through a web form, one invoice per file, in English” is. Otherwise a tool can pass on tidy demo files and fail on the first phone photo of a paper invoice.
For each input, write down the data type, the format, the source and the edge cases you know about. Anthropic’s example edge cases include missing or irrelevant input, overly long input and cases where even people would disagree (source). If your CRM exports phone numbers in three different formats, put that in the brief.
3. The exact output
Define what the tool returns and how correctness is judged. “It drafts a reply” tells a developer nothing. “It returns a plain-text email draft of 80–120 words, addressed by first name, referencing the subject of the original enquiry, with a proposed meeting time in the recipient’s time zone” is an output you can check.
If the output feeds another system, describe it as named fields. Claude’s structured outputs feature holds responses to a JSON schema, so the fields you list in the brief can become the schema the tool must follow (source).
Every output also needs a success criterion. Anthropic’s guidance is that good criteria are specific and measurable: “accurate sentiment classification”, not “good performance” (source). Without one, “is this working?” has no answer both sides accept. Most tools need several, such as accuracy, tone, latency and price.
4. The one human checkpoint
Name the point where a person approves before action is taken. It keeps a mistake contained: an AI that drafts a reply for checking has a limited failure mode, and one that sends on its own doesn’t. Put it where the output is hard to undo or reaches a customer, a financial record or published content.
What Happens When You Skip Each Element
These are the patterns we’d expect when one element is missing. It’s our view, not measured rates.
No job sentence: The scope grows during the build, and the tool ends up doing several things unevenly instead of one thing reliably.
No input definition: The tool works on clean demo data. Real data arrives in other formats, and the tool errors or returns something plausible and wrong. Then both sides argue about whether that data was in scope.
No output definition: There’s no agreed standard to test against. The client says the output is wrong, the builder says it’s what was asked for, and neither can prove it.
No human checkpoint: A wrong figure or category reaches live systems before anyone notices. Then you’re fixing the AI and cleaning up the records it touched.
How to Map Dependencies Before the Build Starts
Every custom AI tool depends on inputs and systems it doesn’t control. Before signing, list every dependency and decide who owns each one, or expect change requests later.
Run through these five questions for any proposed tool:
-
What data sources does it read from? List every system, file format, API and export. Note who controls each and what happens to the tool if its format changes.
-
What human decisions does the current workflow include? AI handles defined, repeatable steps well. Judgment calls that vary by person need writing down as rules first, or they stay with a person.
-
What third-party systems does it connect to? List every API, database or SaaS platform the tool reads from or writes to. For each, confirm the rate limits, the sign-in method and the fallback if the connection drops.
-
What triggers the automation? A file upload, a webhook, a schedule, a person clicking a button? A tool a person starts fails differently from one that runs on a timer.
-
Who owns each dependency? For every item above, name a person or team. “IT” isn’t an owner. Unowned dependencies are the ones nobody checks when they change.
For a wider risk frame, the NIST AI Risk Management Framework is voluntary and covers the design, development, use and evaluation of AI systems. NIST also publishes a Generative AI Profile, released July 26, 2024, for risks specific to generative AI (source). Small tools rarely need the full framework, but it helps when a tool touches customers or money.
What Good AI Scoping Looks Like in Practice
Here’s an illustrative brief, written for this post, not taken from a client. The job: route inbound freight enquiries to one of three operations teams.
- Job sentence: “The tool reads inbound enquiry emails and returns a routing decision: bulk, refrigerated, hazardous or unclear.”
- Input: The plain-text email body and subject line, read from one monitored inbox.
- Output: A JSON object with
category(one of the four values) andreason(one short sentence). - Success criterion: On a set of past enquiries labelled by the ops team, the category matches their label, with hazardous enquiries never routed as bulk.
- Human checkpoint: Anything marked hazardous or unclear waits in a review queue before routing.
- Dependencies: The inbox connection (owner: ops manager), the routing rules document (owner: head of ops), the chat webhook that posts routed enquiries (owner: IT systems lead).
Two details make this testable. The output has fixed values, so a script can grade it, and Anthropic calls code-based grading the fastest and most reliable method (source). And rules anyone can check trigger the checkpoint, not the model’s sense of how sure it is. One detail from the docs: structured outputs don’t guarantee the capitalisation of fixed values, so compare them case-insensitively (source).
A one-paragraph brief has nothing to grade against, so every review becomes a debate.
What to Ask Before You Sign Anything
Before you commit to a custom AI build with any agency, including ours, ask these questions and get written answers:
- Can you show me the job definition in one sentence?
- What is the exact input schema?
- What is the exact output format and success criterion?
- What is the dependency map and who owns each item?
- What is the human review checkpoint?
- How will we QA the output before go-live?
- What is out of scope, explicitly?
Discovery should answer these questions, not postpone them.
Related Posts
- How to define AI automation inputs and outputs: the input and output sections in more depth
- AI automation dependency mapping: the dependency list, step by step
- How to define done for an AI integration: turning success criteria into acceptance
- Human review checkpoints in AI automation: designing the checkpoint the brief names
Frequently Asked Questions
What is a custom AI tool brief and what should it contain?
A document that defines the tool’s job in one sentence, its exact inputs, its exact output with a measurable success criterion, and the human checkpoint before any automated action. Add a dependency list with named owners. Without these, nobody can test whether the build is finished, or correct.
How detailed does the input definition need to be?
Detailed enough that a developer could write a check for it. That means the data type (text, number, PDF, CSV), the format (fields, column names, limits), the source (which system sends it) and the edge cases you know about, such as suppliers who send photos instead of PDFs.
What is a human checkpoint in an AI automation and why is it required?
A named step where a person reviews the AI’s output before it acts: sends an email, updates a record or routes a request. It stops an error before it spreads, and makes someone responsible for what gets actioned, not only for what the AI suggested.
What happens if the job definition changes mid-build?
That’s fine, as long as the change is written down as a scope revision and planned and priced as one. Trouble starts when changes are treated as corrections, which happens when the original brief supported two readings.
Can I write a brief myself without a technical background?
Yes. The job sentence, the output format and the human checkpoint need you to know your workflow, not the technology. For inputs, talk to whoever manages the data; you don’t need to understand APIs to say what your CRM exports. A good partner fills in the technical parts during discovery.
How long should it take to write a proper brief?
We don’t put a number on it, because it depends on how well the workflow is already documented. A focused tool with one job should be quick. If you keep finding new branches as you write, the workflow is more than one tool can handle, and it’s better to know that before the build.