An AI automation that works in the demo can still fail in production. The demo runs on clean, hand-picked examples. Production gets missing fields, odd formats and duplicates. The fix rarely lies in the model. It lies in writing down, before anything is built, exactly what the automation receives and exactly what it must return.
We design and build AI automations through our AI integration service, so weigh that. Every fact below about model features was checked against Anthropic’s and OpenAI’s API documentation in October 2026.
Short answer: before building, write three things down: the trigger that starts the automation, the input schema (every field it receives, its type, and what happens when it’s missing) and the output schema (every field it returns, its valid values and where it goes next). Ask the model for structured output so the response always matches that shape. Then validate what the schema can’t check, such as ranges, totals and whether the answer is right, and send anything that fails to a person.
What Structured Outputs Guarantee
Both Anthropic and OpenAI now offer structured outputs, which force the model’s response to match a JSON schema you supply. That removes a whole class of failures, but not all of them. As of October 2026, from the Anthropic and OpenAI docs:
| Check | Handled by structured outputs? |
|---|---|
| The response is valid JSON | Yes |
| Required fields are present, with the right types | Yes |
| A value comes from your list of allowed values | Yes, but Claude may change capitalization |
| A number is within a range (minimum, maximum) | No: not supported in Claude’s schema |
| A text field is within a length limit | No: not supported in Claude’s schema |
| The model refuses the request for safety reasons | No: the refusal can break the schema |
| The response hits the token limit | No: the output may be incomplete |
| The values are actually correct | No: a schema checks shape, not truth |
Sources: Anthropic structured outputs, OpenAI structured outputs.
What “Defined Inputs and Outputs” Actually Means in AI Automation
An AI automation is a pipeline. Something triggers it, data goes in, the model processes it, and a result comes out. That result either gets used or it doesn’t.
Defining the inputs and outputs before building means writing the contract the automation has to honor. Without one, you’re relying on the model to work out what you meant. Sometimes it will. When it doesn’t, you can’t predict how or test for it.
Inputs: What the AI Receives and in What Format
An input isn’t just “the data”. It’s the data, its format, the required fields and the acceptable values.
For an invoice automation, the input isn’t “an invoice”. It’s a PDF or a structured record with a vendor name, an invoice number (required), line items (description, quantity and unit price for each), a total with two decimal places and a due date in ISO 8601 format.
Any field the spec leaves out is a field the model will improvise on. Some improvisation is fine. Improvisation on invoice totals, order amounts or customer details isn’t.
Outputs: What the AI Returns and How You Validate It
The output spec is the other half of the contract. It says what the automation produces, in what structure, and what a valid result looks like.
“The AI will summarize the invoice” isn’t an output spec. “The AI returns a JSON object with vendor_name, invoice_total, due_date and approval_status, which is one of approved, flagged or error, and flagged sends an alert to the accounts channel” is.
If you can’t write down what a correct output looks like, you can’t test whether the automation works.
Why Leaving Inputs and Outputs Vague Guarantees Problems
A vague scope feels faster at the start. It costs more later, because nobody can say whether the tool is working.
The Testing Problem: You Cannot QA What Has No Spec
A test compares a result with an expected value. With no expected value, there’s no test, only a demo.
That’s why demos so often look fine. They run on hand-picked examples, walked through by someone who knows the system’s limits. Production runs on real data: edge cases, inconsistent formatting, missing fields and duplicates. Without an output spec, you can’t write the tests that catch those failures before your staff or customers do.
The Ownership Problem: Who Is Responsible When Output Is Wrong?
Accountability follows the spec. If the spec says the output is a JSON object with five named fields and the automation returns a paragraph of text, something is clearly broken, and it’s clear whose job it is to fix it. If the output was never specified, “it’s working as intended” is hard to argue with.
Settle this at the scoping stage, in writing, before the build starts.
Design Principles for AI Automation That Actually Works
Three rules cover most of the common failure patterns. None of them is technically hard. They take discipline, on your side and the builder’s.
Define the Trigger, the Input Schema, and the Output Schema Before Any Build
Before any code is written, three things should exist as plain-language documents:
- The trigger: the event that starts the automation, such as a new row in a spreadsheet, a form submission or an inbound email that matches a rule.
- The input schema: every field the automation receives, its type, whether it’s required and what happens when it’s missing.
- The output schema: every field the automation returns, its type, its valid values and which system receives it.
If a builder can’t produce these before the build is priced, the scope isn’t finished.
Use Structured Formats Wherever Business Logic Requires It
Structured formats such as JSON or a CSV with headers can be tested. Free text can’t.
Say an automation reads a customer email, classifies the intent, routes it to a team and logs it in the CRM. The classification needs to be one category from a fixed list, not a summary the model wrote that day. A summary can’t be routed reliably. A category label can.
This doesn’t mean AI can’t handle unstructured inputs like emails or PDFs. Pulling information out of them is one of the things it does well. But its output still has to be structured. The model’s job is to turn the unstructured input into structured fields your systems can act on. When it can’t do that cleanly, the validation step should catch it and pass it to a person, not let it flow downstream.
With structured outputs switched on, the shape is enforced for you. Anthropic says its structured outputs use constrained decoding, so responses are always valid JSON with the required fields and the right field types (source). OpenAI says its version means you don’t need to worry about the model omitting a required key or hallucinating an invalid enum value (source).
Build Validation Into the Output: Every AI Action Should Be Traceable
Every automation should log what it received, what it returned and whether the output passed validation. This isn’t overhead. It’s how you’ll know the tool still works three months after launch.
A simple validation step checks that values fall within expected ranges, that totals add up and that anything unusual raises an alert instead of failing silently. Without it, you only find problems when they show up as a missed payment, a wrong order or a customer request nobody answered.
What Structured Outputs Don’t Cover
A schema guarantees the shape of the answer. Your own code still has to check the rest. From the providers’ documentation:
- Ranges and lengths. Claude’s structured outputs don’t support numerical constraints such as
minimumandmaximum, or string constraints such asminLengthandmaxLength. A schema that uses them returns an error (source). Check “total is above zero” or “SMS is under 160 characters” in code. - Refusals. If Claude refuses a request for safety reasons, the response has
stop_reason: "refusal"and may not match your schema (source). OpenAI notes the same for refusals (source). - Token limits. If the response is cut off at the token limit, both providers say the output may be incomplete and not match the schema. Check the stop reason before you use the result.
- Capitalization. Anthropic says structured outputs don’t guarantee the capitalization of allowed text values, so compare them case-insensitively.
- Correctness. A well-formed invoice total can still be the wrong number. Checks like “line items add up to the total” are yours to write.
Our guide to AI automation input validation covers the same discipline on the way in.
Worked Examples: Inputs and Outputs Done Right
Here are three common automations and what a proper spec looks like for each. They’re illustrations of the method, not client results.
Invoice Processing Automation
Trigger: a new PDF arrives in the accounts inbox.
Input schema: the PDF file (with a size limit), the sender’s email domain (to match the vendor) and the email subject (to spot duplicates).
Output schema: a JSON object with vendor_id (matched from your records, or UNMATCHED), invoice_number, invoice_total, line_items, due_date (ISO 8601) and status (one of queued, flagged-duplicate, flagged-unmatched-vendor or error).
Validation: the line items must add up to the total, the due date must be in the future and the invoice number must not already exist. Anything that fails, or comes back as error or UNMATCHED, goes to a review queue. Nothing reaches accounts payable until it passes.
Lead Triage and CRM Entry
Trigger: a new contact form submission.
Input schema: name (required), email (required, valid format), phone (optional), message (free text, with a length limit) and source_page (required).
Output schema: lead_score (1–10), intent_category (one of pricing-enquiry, support-request, partnership, spam or other), crm_entry (matching your CRM’s fields), assigned_to (from your routing rules), priority (high, normal or low) and flag (true sends a team notification).
The spec can then say exactly what happens next: a lead scored 8 or above with intent_category: pricing-enquiry notifies sales straight away. That’s testable. “The AI will handle leads intelligently” isn’t.
Appointment Booking Confirmation Logic
Trigger: a new booking event from the scheduling system.
Input schema: customer_name, customer_email, service_type (from a fixed list), appointment_datetime (ISO 8601), assigned_staff_id and location if relevant.
Output schema: confirmation_email (from an approved template), calendar_invite (ICS file), sms_confirmation (160 characters at most, checked in code), crm_log_entry and status (sent, failed or skipped-duplicate).
The duplicate check runs before any message goes out, and every booking is logged whether or not the customer message succeeds.
What to Ask Any AI Agency Before You Sign Anything
These questions aren’t adversarial. A builder who has done this properly will answer them quickly.
- Ask for the input schema. What fields does the automation receive, in what format, and what happens when one is missing or malformed?
- Ask for the output schema. What does a correct output look like, what are the valid values and how is it validated?
- Ask how it’s tested. Is there a set of test cases, including the awkward ones?
- Ask who fixes a wrong output. The answer should be in the contract, with a process for reporting and correcting errors.
- Ask what monitoring looks like after launch. How will you know if outputs start going wrong three months from now, and what raises the alert?
Related Posts
- How to scope an AI project so it doesn’t fail: the scope document these specs belong in
- AI automation input validation: checking data before the model sees it
- Human review checkpoints in AI automation: where failed outputs should go
- AI automation error handling: designing fallbacks when something breaks
FAQ
What is the difference between inputs and outputs in AI automation?
Inputs are what the automation receives: the trigger, the fields and their formats. Outputs are what it returns: the result, its structure and the values it’s allowed to contain. Both should be written down before the build begins, or there’s nothing to test the tool against.
Why do AI automation projects fail after a successful demo?
Demos run on hand-picked data. Production runs on real data, with missing fields, unexpected formats and edge cases the demo never saw. Without an output spec and a validation step, those failures stay invisible until they cause a problem downstream.
What does a structured output mean in an AI workflow?
A result in a predictable, machine-readable format, usually JSON, where every field has a name, a type and a set of valid values. Anthropic and OpenAI both offer structured outputs that force the model’s response to match a schema you supply.
Do structured outputs mean I don’t need validation?
No. They guarantee the shape, not the content. Claude’s version doesn’t enforce number ranges or text lengths, a safety refusal or a cut-off response can break the schema, and a correctly formatted value can still be wrong. Check those in your own code.
Can AI handle unstructured inputs like emails or PDFs?
Yes. Extracting information from emails and documents is a common use. The result should still be structured: the model turns the unstructured input into defined fields, and a validation step catches anything that doesn’t come out cleanly.
What is an input schema and why does it matter?
A spec of every piece of data an automation receives: field names, types, which are required, format rules and what happens when a field is missing or malformed. It sets what the automation can reliably act on and gives you a baseline to test the build against.