What is AI observability in production?

AI observability is being able to see what your AI features are actually doing in production — the quality of outputs, the cost and latency per call, whether behaviour is drifting, and how often things fail or fall back — so problems surface as data you can act on rather than as user complaints. It's the AI-shaped extension of normal application monitoring: you still watch errors and latency, but you also watch things a database-backed app never has, like output quality and cost-per-call, because those are exactly the things that can degrade silently while every request still returns a 200. Without it, your users are your monitoring, and they report failures as churn.

What should you monitor for an AI feature?

Six things: quality (are outputs still good, sampled and scored against your evals), cost (tokens and spend per call and per user, so a cost regression is visible immediately), latency (AI calls are slow and variable, and slow is a quality problem), failures and fallbacks (how often calls error, retry or hit a fallback path), drift (whether outputs are changing over time, from your own edits or a model update), and usage (which features are used, so you know what to watch most). The point is that an AI feature can be failing on quality or bleeding money while returning perfectly valid responses — so the standard 'is it up?' monitoring misses exactly the failures that matter most.

How do you detect AI quality problems in production?

By sampling real production outputs and scoring them — ideally against the same eval criteria you use before shipping — rather than waiting for users to complain. Log inputs and outputs (with privacy care), sample a slice regularly, and score them, so a drop in quality shows up as a falling number, not a rising support queue. Two things make it work: logging enough to reproduce a bad output, and having a defined 'good' to score against. Quality is the hardest thing to monitor because it's not a 200-or-500 signal — a confidently wrong answer looks identical to a right one at the HTTP level, so you have to look at the content, not just the status code.

Production AI Observability: Watching What the Model Does

An AI feature can return a valid response, on time, with no error — and be completely wrong, or quietly costing you triple what it did last week. That's the thing normal monitoring misses. Your uptime dashboard says green; every request is a clean 200; and meanwhile output quality has drifted, a model update changed the behaviour, or a prompt tweak doubled the token cost. In a database-backed app, "it returned successfully" mostly means "it worked." In an AI product, success and correctness are different signals, and if you only watch the first, your users are your quality monitoring — and they report problems by leaving. This is how I think about seeing what the model actually does.

Success ≠ correctness

Start from the failure that makes AI observability different: a call can succeed and still fail. A confidently wrong answer, a subtly-off summary, an output that no longer parses the way your code expects — all of these return 200. Standard monitoring (errors, latency, uptime) is necessary and completely blind to them.

So AI observability is two layers: the normal one (is it up, fast, erroring?) plus an AI-specific one (is it good, is it drifting, is it costing what it should?). The second layer is the one prototypes skip and products can't afford to — it's a big part of why so many AI apps feel like demos: they're monitored like static apps, so quality degrades invisibly.

The six things worth watching

Dimension	The question	Why it's silent without monitoring
Quality	Are outputs still good?	A wrong answer returns 200 like a right one
Cost	Tokens/spend per call and per user?	A cost regression bills you, not errors
Latency	How slow, how variable?	Slow AND correct is still a bad experience
Failures/fallbacks	How often do calls error, retry, or fall back?	A fallback firing constantly hides a real problem
Drift	Are outputs changing over time?	Your edit — or a model update — shifts behaviour quietly
Usage	Which features are actually used?	You can't prioritise monitoring blind

The two most-neglected are quality and cost, precisely because neither shows up in error rates. Cost ties straight to unit economics — a monitoring gap here is a margin leak you find on the invoice. Quality ties to your evals — which is the only practical way to monitor it at all.

The AI failures that hurt most are the ones that return 200. If your monitoring only knows "up or down," it's blind to every failure that matters — cost, drift and quality all degrade with a perfectly healthy status code.

Monitoring quality: sample and score

Quality is the hard one, because there's no status code for "good." The practical approach is the same discipline as pre-ship evals, pointed at production:

Log inputs and outputs — enough to reproduce a bad result later (with real care about privacy and sensitive data; log what you need, not everything).
Sample a slice of real traffic regularly — you don't score every call, you score enough to see a trend.
Score the sample against your defined "good" — the same criteria your eval set uses.
Watch the number. A drop in the score is a quality regression, visible as data before it's visible as churn.

The key insight: you already built the "good" definition for your evals; production quality monitoring is that same definition, run on live samples instead of a fixed test set. Build the eval once, use it twice.

Drift: your product changes when you didn't

Drift deserves its own attention because it's the failure that happens while you did nothing. A model update from the provider, a deprecated version, a silent change behind an alias — any of these can shift behaviour under you, and without observability the first you hear of it is a user noticing before you do. Monitoring outputs over time turns "the model changed" from a surprise into a chart: a shift in your quality score, cost, or output shape that flags the change so you investigate it deliberately instead of discovering it through support tickets.

Failures and fallbacks: watch the safety net

If you built fallbacks — and you should — you have to watch how often they fire. A fallback path is a safety net, and a net catching things constantly means something upstream is broken. The trap is that fallbacks make failures invisible by design: the user still gets a response, so nothing looks wrong, while the primary path fails silently behind the scenes. Monitor fallback rate as its own signal, or your safety net becomes a blindfold.

Keep it proportionate

None of this has to be heavy — and for a solo product, it shouldn't be. This is the same instinct as using analytics as a solo builder: measure the few things that change a decision, not everything. A minimal, honest setup is logging inputs/outputs/cost/latency, a periodic sampled quality score against your evals, and an alert on cost and fallback rate. That's enough to catch the failures that return 200. Elaborate observability platforms can come later; the mistake isn't having a simple setup, it's having none and calling green-uptime "monitoring."

What I would decide

Monitor the AI layer separately from the app layer, because success and correctness are different signals and only one of them has a status code. Reuse your eval definition to sample-score production quality — build the "good" once, run it live. Watch cost and fallback rate as first-class alerts, because both fail silently while every request looks healthy. And keep it proportionate: the goal isn't a monitoring cathedral, it's making sure that when an AI feature degrades — in quality, cost or behaviour — you find out from a number, not from a user who's already gone.

Part of the Building AI Products guide. See also prompt & model versioning and AI product unit economics. The newsletter sends one finance-systems pattern, product decision or build lesson every two weeks.