A custom AI tool has more moving parts than its code: the prompt, the model it calls, the settings on each call and a vendor schedule you don’t control. Change one without a record and a test, and you can’t tell what broke or how to put it back.
We build and maintain custom AI tools through our AI integration service, so weigh that. Every fact below about Claude models, model IDs and retirement was checked against Anthropic’s own documentation in October 2026.
Short answer: Treat the prompt, the model ID, the call settings and the output schema as one release with one version. On the Claude API every model ID is a pinned snapshot, so name the exact ID and upgrade on purpose. Keep real test inputs with known-good outputs and rerun them before every change. Change things in staging, then promote. Keep the last working release so rollback is a config swap. And read retirement notices: Anthropic gives at least 60 days’ notice, and requests to a retired model fail.
What Anthropic’s Docs Say About Model Versions
As of October 2026, from Anthropic’s documentation:
| Topic | What the docs say |
|---|---|
| Model IDs | Every model ID is a pinned snapshot; the model stays constant for the life of the ID |
| 4.6 generation on | Dateless IDs such as claude-sonnet-4-6 are the snapshot, not a pointer |
| Older aliases | An alias such as claude-sonnet-4-5 points to the latest dated snapshot |
| Serving changes | Infrastructure updates can cause minor behavior differences on an unchanged ID |
| Retirement notice | At least 60 days, to customers with active deployments |
| Retired models | Requests fail |
| Sampling settings | Non-default temperature, top_p or top_k return a 400 error on Claude 4.7 and later |
| Tokenizer | Claude 4.7 and later produce about 30% more tokens for the same text |
| Claude Sonnet 5.5 | $2 / $10 per million input / output tokens; retirement not sooner than September 28, 2027 |
Sources: Model IDs and versioning, Models overview, Model deprecations, Pricing.
Why AI Tool Versioning Is Different from Regular Software
Software versioning tracks code. An AI tool’s behavior also depends on the prompt, the model, the call settings and the vendor’s model lifecycle. Your code can stay the same while any of those changes.
Your Prompt Is Code, and Almost Never Treated That Way
The system prompt is the most important logic in the tool. It says what the model does, how it formats the output and what it must not do. Yet it often lives in a shared doc or a settings field, edited in place.
Treat it as code. Keep it in version control, log every change and tag the version known to work. Without that, there’s no rollback, only guessing.
How Your Tool Can Change Without Touching Your Code
On the Claude API, the model behind an ID doesn’t change. Anthropic doesn’t update the weights or configuration of an existing model ID; a new version ships under a new ID (source). Your tool can still change without a code change:
- An alias moves. For models before the 4.6 generation, short aliases point to the most recent dated snapshot, and the fixed-model guarantee doesn’t cover them (source).
- Serving changes. Request routing, safety classifiers and sampling logic can change and occasionally cause minor differences on the same ID (source).
- The model retires. Requests to a retired model fail (source).
What to Version in a Custom AI Tool
Version control for an AI tool covers three layers. Each one is a failure point if you don’t track it.
The Prompt Layer
The system prompt, message templates and any examples that shape the output. Every change gets a version tag (v1.0, v1.1), a date and one sentence on what changed and why. Store them in the project repo, or in a document with version history that you own.
Model Configuration
Record the exact model ID and every setting sent with the call: max_tokens, effort, thinking and tools. Don’t rely on defaults. Default effort is medium on Claude Opus 5.5 and high on Claude Sonnet 5.5, so switching models can change behavior even when your code sends nothing new (source).
Older advice says to log temperature changes. On Claude 4.7 and later, a non-default temperature, top_p or top_k returns a 400 error, and Anthropic recommends prompting instead (source). If your tool still sends temperature, a model upgrade will break the call.
Integration Touchpoints
If the tool connects to a CRM, a form backend or a WordPress site, document the input schema and the output parser at each point. When a field name changes upstream, a poorly documented integration fails quietly, and the AI gets the blame.
Pin down the output too. Claude’s structured outputs constrain the response to a JSON schema you send in output_config.format, so the parser gets valid JSON (source). Version that schema with the prompt.
For tools embedded in a custom WordPress development project, this matters even more. The integration layer sits between the AI logic and the site, and both sides need to know what data shape to expect.
A Practical Update Management Process for Small Teams
Enterprise MLOps assumes a data team, a model registry and a CI/CD pipeline. A small business running one AI tool needs something lighter that it will actually use.
Before You Deploy Any Change, the Baseline Test
Define what “working” means first. Anthropic’s evaluation guidance says success criteria should be specific and measurable, and tests should mirror your real tasks, edge cases included (source). Save real inputs and their good outputs. That’s your baseline.
After any change, rerun it and compare. Anthropic recommends code-based grading where possible as the fastest and most reliable, and LLM-based grading for judgment calls once you’ve checked it’s reliable (source). In practice:
- Check structure in code: valid JSON, required fields, values in range.
- Check facts by exact or string match where there’s a right answer.
- Grade tone or quality with a clear rubric, or review a sample by hand.
- Decide whether each difference is an improvement or a regression.
Anthropic favors more test cases with automated grading over a few hand-graded ones (source). Add every real failure you find.
Staging vs. Production: the Minimal Viable Version for a Small Team
For most small AI tools, staging is a second API key, a second config file and a test URL. Change things there, check them against the baseline, then promote by swapping the config. Never edit the production prompt directly. For a bigger change, such as a new model, send a small share of real traffic to it first.
Expect a cost blip after a prompt release. A prompt cache hit needs an identical prefix, and editing the cached part invalidates it from that point on (source), so the first calls write a fresh cache, which is priced above base input (source).
Rollback: What It Means and How to Actually Do It
Rollback means restoring the prompt, model ID, settings and schema to a known-good release. Versioned together, that’s a config swap. Tag each production release (prod-v2.1), keep previous releases in version control, and when something breaks, restore the last working tag and rerun the baseline.
One limit: you can’t roll back to a retired model, because requests to it fail (source). Test upgrades well before the date.
What Good Documentation Looks Like, and What Clients Should Demand
A custom AI tool should come with documentation that lets you maintain it, or brief a new developer without starting from scratch.
The One-Page Tool Spec Every Client Should Have
- What the tool does, in one sentence.
- The exact model ID in production and its retirement date.
- Where the versioned prompt and output schema live.
- What inputs it accepts and what it returns.
- Who manages it, where it runs and how to trigger a rollback.
This should exist before go-live. If you want to talk through a proper handoff for your build, start a conversation.
What to Log When You Ship a Change
Date, who made the change, what changed (prompt, model, setting, schema or integration), why, and the baseline result. A shared doc or a changelog file in the repo is enough. The habit matters more than the format.
How Model Retirements Affect Your Tool
On the Claude API, retirements come with warning. Anthropic notifies customers with active deployments at least 60 days before a publicly released model retires, by email and in its documentation (source). Tools that break “without warning” are usually ones where nobody read that email.
A current example: on September 30, 2026, Anthropic announced that claude-sonnet-4-5-20250929 retires on November 30, 2026, with claude-sonnet-5-5 as the recommended replacement (source). The upgrade can change more than outputs:
- Settings: non-default temperature calls fail on Claude 4.7 and later.
- Token counts: Claude 4.7 and later produce about 30% more tokens for the same text (source).
- Price: Sonnet 4.5 is $3 / $15 per million input / output tokens; Sonnet 5.5 is $2 / $10 (source). Our own worked example: input that was 1 million tokens on Sonnet 4.5 ($3) could be about 1.3 million on Sonnet 5.5 ($2.60). Measure it on your baseline.
- Platform: Amazon Bedrock and Google Cloud set their own retirement dates (source).
Send Anthropic’s notices to a shared inbox, check the deprecations page on a schedule, and test the replacement as soon as a notice arrives. The Console’s usage export, broken down by API key and model, shows where an old model is still in use (source).
Related Posts
- AI prompt version control: managing prompts across automations
- AI automation rollback: what to do when outputs degrade
- Testing an AI automation before production: building the test set
- AI model updates and your integration: planning for model changes
FAQ
What is custom AI tool versioning and why does it matter for small businesses?
Knowing exactly what’s in production: the prompt, model ID, call settings and output schema, released together under one version. Without it, you can’t explain a change in output, update safely or roll back.
How do I know if a vendor model update has changed my custom AI tool’s behavior?
Rerun a baseline: saved real inputs with known-good outputs. On the Claude API the model behind an ID doesn’t change, but serving changes can cause minor differences, and older aliases move to newer snapshots. A baseline catches both.
What is the simplest rollback process for a custom AI workflow?
Keep the prompt, model ID, settings and schema in version control, tagged per release. When behavior breaks, restore the last working tag and rerun the baseline. It only works while the old model is still available.
How often should I audit a custom AI tool after launch?
Our suggestion: rerun the baseline on a regular schedule and after every change. Check your model’s retirement date, and test the replacement as soon as a notice arrives. Anthropic advises testing with newer models well before the retirement date.
What documentation should I receive when an agency builds a custom AI tool?
A tool spec (model ID, prompt and schema location, inputs and outputs), a changelog, a written rollback procedure, the baseline test set and access to the repo or config. Without these you own a black box, not a tool.
Does pinning a model version affect performance or cost?
Pinning itself doesn’t. Every Claude model ID is already a pinned snapshot, priced per model. Moving to a new model is what changes cost and behavior: prices differ, and Claude 4.7 and later use a tokenizer that produces more tokens for the same text. Pinning means you choose when to upgrade, and test first.