Why do you need to version prompts?

Because a prompt is production code — it determines what your users get — and if you change it without versioning, you can't say what produced yesterday's output, can't roll back a change that made things worse, and can't tell whether a regression came from your edit or from somewhere else. Versioning a prompt means the exact text (and its settings) is tracked, tied to a version, and deployed deliberately, the same way you'd treat any code that shapes the product. The moment a prompt is doing real work for real users, 'I tweaked it in the dashboard' stops being acceptable — you need to know which version is live, what changed, and how to go back.

What happens when the model provider updates the model?

Your product can change behavior even though you changed nothing — because the model is a dependency you don't fully control. A provider update, a version deprecation, or a silent change to a model alias can shift outputs, break an output format your code depends on, or alter quality in ways your users notice before you do. The defenses are pinning to specific model versions rather than floating aliases where you can, running an eval set against any new model version before adopting it, and monitoring live output so a shift shows up as data rather than as support tickets. Treat the model as an external dependency with its own release notes, because that's exactly what it is.

What is an eval dataset for an AI product?

An eval dataset is a curated set of representative inputs with known-good expected outputs (or scoring criteria), that you run a prompt or model against to measure quality objectively — the AI equivalent of a regression test suite. Instead of eyeballing a few examples and hoping, you run the whole set and get a score, so a prompt change or a new model version is judged on evidence: did it improve, hold, or regress. It's what turns 'this feels better' into 'this passes,' and it's the single practice that most separates an AI product you can change safely from one you're afraid to touch.

Prompt & Model Versioning: Production Governance for AI Products

A prompt is production code, and the model behind it is a dependency you don't control. Most AI products treat both like scratch notes — and it shows. This is a big part of why so many AI-built apps feel like demos: the prompt lives in a dashboard, nobody knows which version is live, the model floats on whatever alias the SDK defaults to, and "testing" is trying three examples by hand. That's fine for a prototype and a slow-motion disaster for a product real people rely on. Production AI needs the same governance as any other production system — adapted to the two things that make it different: prompts you author and a model you rent. Here's how I keep it from silently drifting.

Two things change your output — and you own one of them

Every AI feature's behavior is a product of two inputs:

Your prompt (and its settings) — which you write and control.
The model — which the provider controls and can change under you.

The failure mode is treating the first as casual and the second as fixed. It's the opposite: the prompt deserves the discipline of code, and the model deserves the wariness of an external dependency with its own release schedule. Get those two right and most "why did it suddenly get worse?" mysteries never happen.

What to version — and why

"Versioning" is vague until you list what actually needs it. For an AI feature, the versioned unit is bigger than just the prompt text:

What to version	Why it matters
Prompt text	The literal instructions — the core of the behavior
Model + version	A different model (or version) is a different product
Settings	Temperature, max tokens, tools — they change output too
Output contract	The shape your code depends on (JSON schema, fields)
Eval dataset	The bar you measure against — versioned with the prompt

The one people forget is the output contract. If your code parses the model's response, the shape of that response is an API between the model and your app — and a prompt or model change can break it silently, producing output that's still plausible prose but no longer parses. Pin the contract, validate against it, and fail loudly when it breaks rather than shipping a malformed result downstream.

Prompts are code: treat them like it

Concretely, that means:

The prompt lives in the codebase, in version control, not only in a provider's dashboard where changes are untracked and un-reviewable.
Changes are deployed, not tweaked live. A prompt edit is a change like any other — reviewed, versioned, releasable, reversible.
You can always say which version is live, and roll back to the previous one in one step when a change regresses.

This is the same principle as turning a rough idea into a proper ticket rather than hacking in the dark: the discipline that feels like overhead on day one is what lets you move fast without breaking users on day one hundred.

The model is a dependency you don't control

You can pin your own code; you can't fully pin someone else's model. So you manage it like any external dependency that can change:

Pin to specific model versions rather than floating aliases wherever the provider lets you — so "the model changed" is a decision you make, not a surprise you receive.
Read the release notes. Model versions get deprecated and updated; a deprecation you didn't plan for is an outage with a countdown.
Never adopt a new model version blind. Run it against your eval set first (next section). A newer, "better" model can be better on average and worse on your specific tasks or output format.

Your product can regress while your code sits untouched, because the model moved. If you're not running evals against model changes, your users are your regression test — and they report failures as churn.

Eval datasets: the regression suite for AI

The single practice that most separates a product you can change safely from one you're scared to touch is an eval dataset: a curated set of representative inputs with known-good expected outputs (or scoring criteria) that you run any prompt or model change against, and get a score.

It does for AI what a test suite does for code:

Objective judgment. "It feels better" becomes "it scored higher on the set." Vibes don't survive contact with a real change; scores do.
Regression safety. A change that quietly breaks a case you'd forgotten shows up immediately, before users find it.
Safe model migration. A new model version is judged on evidence — improved, held, or regressed — not on a hopeful skim of three outputs.

Build it deliberately: cover the normal cases, the edge cases, and the ones that have burned you before (every production failure becomes a new eval case). Version it alongside the prompt, because the bar and the thing being measured belong together.

The production loop

Put together, governed AI features run on a loop that looks a lot like normal engineering, with two AI-shaped additions:

Change the prompt, settings, or model — as a versioned change.
Eval against the dataset — improved, held, or regressed?
Validate the output contract — does it still parse and conform?
Deploy deliberately, knowing the live version.
Log and monitor real outputs — so drift (yours or the model's) shows up as data.
Roll back in one step if something regresses; add the failure to the eval set so it can't recur silently.

Steps 2 and 3 — evals and the output contract — are the AI-specific ones, and they're exactly the ones prototypes skip. They're also what let you keep a human in the loop on quality without hand-checking every output — the same spirit as using AI without letting it decide.

What good looks like

Prompt, model version, settings and output contract are all versioned in the codebase, not tweaked live.
The model is pinned and its release notes are read; a new version is adopted only after it passes evals.
An eval dataset scores every change, and every production failure becomes a new eval case.
Outputs are validated against a contract and monitored live, so drift is data, not tickets.
Rollback is one step, because you always know what's deployed.

None of this is heavy — it's a prompt in git, a pinned model, a folder of eval cases, and the discipline to run them before you ship. But it's the difference between an AI product that improves safely over time and one that's a haunted house where nobody dares touch the prompt because no one knows what will break. Treat prompts as code and the model as a dependency, and the "magic" becomes something you can actually engineer, change and trust.

Part of the Building AI Products guide. See also why most AI-built apps feel like demos and how I use AI without letting AI decide. The newsletter sends one finance-systems pattern, product decision or build lesson every two weeks.