[{"data":1,"prerenderedAt":329},["ShallowReactive",2],{"blog-\u002Fblog\u002Fai-output-contracts-and-failure-handling":3,"blog-surround-\u002Fblog\u002Fai-output-contracts-and-failure-handling":311,"blog-related-\u002Fblog\u002Fai-output-contracts-and-failure-handling":320},{"id":4,"title":5,"audience":6,"body":10,"cluster":274,"contentRole":275,"conversionGoal":275,"date":276,"description":277,"draft":278,"extension":279,"factCheckedAt":275,"faq":280,"featured":278,"language":275,"meta":290,"navigation":291,"order":292,"originalAsset":275,"path":293,"pillar":294,"primaryKeyword":295,"relatedProject":275,"releaseScope":275,"reviewCycle":296,"reviewStatus":297,"reviewedBy":298,"searchIntent":299,"seo":300,"sources":275,"stem":301,"tags":302,"type":309,"updated":275,"__hash__":310},"blog\u002Fblog\u002Fai-output-contracts-and-failure-handling.md","AI Output Contracts & Failure Handling",[7,8,9],"solo-founder","ai-product-builder","ai-engineer",{"type":11,"value":12,"toc":264},"minimark",[13,27,32,39,42,46,54,74,82,86,89,164,171,177,181,184,219,223,230,234,237,240],[14,15,16,20,21,26],"p",{},[17,18,19],"strong",{},"The model will fail. Not might — will: the wrong shape, an invented fact, a confident answer that's simply wrong. The only question is whether your product fails with it."," Most AI features are built for the happy path — the demo where the model returns exactly what was hoped for — and have no considered answer for the response that's malformed, off-spec, or plausibly wrong. That's ",[22,23,25],"a",{"href":24},"\u002Fblog\u002Fwhy-most-ai-built-apps-feel-like-demos","why so many feel like demos",": the first weird output either crashes the feature or, worse, flows downstream as if it were valid. Designing the failure path as deliberately as the success path is what separates a product from a prototype. Here's how I think about it.",[28,29,31],"h2",{"id":30},"treat-the-models-output-as-untrusted-input","Treat the model's output as untrusted input",[14,33,34,35,38],{},"The mental shift: ",[17,36,37],{},"the model's response is untrusted input, like anything from a user or a third-party API."," You wouldn't take a form submission and process it without validation; a model's output deserves the same suspicion, because it can be wrong in more ways than a form ever could — wrong shape, wrong values, wrong facts, or perfectly-formed nonsense.",[14,40,41],{},"So the boundary between \"the model returned something\" and \"my code uses it\" needs a checkpoint. That checkpoint is the output contract.",[28,43,45],{"id":44},"output-contracts-define-what-valid-means","Output contracts: define what \"valid\" means",[14,47,48,49,53],{},"An output contract is the shape and constraints a response must satisfy to be usable: a schema, required fields, allowed values, bounds. It turns free-form text you ",[50,51,52],"em",{},"hope"," to parse into a validated interface with a pass\u002Ffail. Two things it buys you:",[55,56,57,68],"ul",{},[58,59,60,63,64,67],"li",{},[17,61,62],{},"A clean failure signal."," An off-spec response fails validation ",[50,65,66],{},"at the boundary"," instead of crashing three functions deep or, worse, being silently processed as valid.",[58,69,70,73],{},[17,71,72],{},"A place to enforce constraints."," Not just \"is it JSON\" but \"is this value in range, does this field exist, is this within length\" — the semantic checks that catch a plausible-but-wrong output.",[14,75,76,77,81],{},"The contract is also the thing that breaks silently when a ",[22,78,80],{"href":79},"\u002Fblog\u002Fprompt-and-model-versioning","prompt or model changes"," — which is why it's worth versioning and validating, not assuming. A model update that shifts the output format is caught by a failing contract, not discovered by a user.",[28,83,85],{"id":84},"the-failure-ladder-validate-retry-fall-back-degrade","The failure ladder: validate → retry → fall back → degrade",[14,87,88],{},"When the contract fails — and it will — you want a defined ladder, not an exception. Each rung is a more graceful response to a worse situation:",[90,91,92,108],"table",{},[93,94,95],"thead",{},[96,97,98,102,105],"tr",{},[99,100,101],"th",{},"Rung",[99,103,104],{},"When",[99,106,107],{},"What happens",[109,110,111,125,138,151],"tbody",{},[96,112,113,119,122],{},[114,115,116],"td",{},[17,117,118],{},"Validate",[114,120,121],{},"Every response",[114,123,124],{},"Check against the contract; pass → use it",[96,126,127,132,135],{},[114,128,129],{},[17,130,131],{},"Retry",[114,133,134],{},"Validation failed",[114,136,137],{},"Try again, often with a corrective instruction",[96,139,140,145,148],{},[114,141,142],{},[17,143,144],{},"Fall back",[114,146,147],{},"Retries exhausted",[114,149,150],{},"A simpler model, a cached\u002Fdefault result, a safe path",[96,152,153,158,161],{},[114,154,155],{},[17,156,157],{},"Degrade",[114,159,160],{},"No safe result possible",[114,162,163],{},"An honest \"we couldn't do that,\" not a crash or a guess",[14,165,166,167,170],{},"The rung people skip is ",[17,168,169],{},"degrade"," — the honest failure. When nothing works, the right answer is a clear, truthful \"this didn't work,\" not a spinner forever, not a crash, and above all not a fabricated result presented as real. A product that says \"I couldn't do that\" keeps trust; one that confidently returns garbage loses it the first time a user notices.",[172,173,174],"pull-quote",{},[14,175,176],{},"The most dangerous AI output isn't the one that errors — it's the confident wrong answer that looks exactly like a right one. Every part of failure handling exists to catch or bound that, because it's the failure that ships silently and costs you a user.",[28,178,180],{"id":179},"hallucinations-contain-dont-pretend","Hallucinations: contain, don't pretend",[14,182,183],{},"You can't fully prevent a model inventing things, so you design so it does the least damage:",[55,185,186,196,202,213],{},[58,187,188,191,192,195],{},[17,189,190],{},"Constrain."," An output contract plus grounding data (the model works from ",[50,193,194],{},"your"," data, not its memory) leaves less room to invent.",[58,197,198,201],{},[17,199,200],{},"Validate what you can."," A cited source that doesn't exist, a value out of range, a reference that doesn't resolve — catch the checkable lies.",[58,203,204,207,208,212],{},[17,205,206],{},"Keep a human in the loop for high stakes."," For anything consequential, the model proposes and a person disposes — the same principle as ",[22,209,211],{"href":210},"\u002Fblog\u002Fhow-i-use-ai-without-letting-ai-decide","using AI without letting it decide",". Auto-acting on unvalidated model output is where hallucinations become incidents.",[58,214,215,218],{},[17,216,217],{},"Be honest about confidence."," Don't present every answer as certain. A UI that signals \"this is a suggestion, check it\" is both more truthful and more robust than one that implies the machine is always right.",[28,220,222],{"id":221},"never-let-the-ai-part-sink-the-whole-feature","Never let the AI part sink the whole feature",[14,224,225,226,229],{},"A structural rule worth stating plainly: ",[17,227,228],{},"a failed AI call should never take down the surrounding feature."," Isolate it — the AI enrichment fails, but the form still submits; the AI summary fails, but the document still saves. Wrap the AI call so its failure is contained to the AI part, not propagated to the whole page. The feature degrades to its non-AI baseline instead of breaking. This is ordinary resilience engineering, but it's routinely forgotten for AI calls because they're bolted on as if they always succeed.",[28,231,233],{"id":232},"what-i-would-decide","What I would decide",[14,235,236],{},"Design the failure path first, not last — treat the model's output as untrusted input and put a validated contract at the boundary. Build the ladder: validate, retry with correction, fall back to something safe, and degrade honestly when nothing works, because \"I couldn't do that\" beats a confident lie every time. Contain hallucinations rather than pretending they won't happen: constrain, validate the checkable, keep a human on the high-stakes calls. And never let a failed AI call sink the feature around it. The model failing isn't the edge case — it's the case you build for, and doing it well is most of what makes an AI product feel solid instead of like a demo one bad output from breaking.",[238,239],"hr",{},[14,241,242],{},[50,243,244,245,249,250,253,254,258,259,263],{},"Part of the ",[22,246,248],{"href":247},"\u002Ftopics\u002Fbuilding-ai-products","Building AI Products guide",". See also ",[22,251,252],{"href":79},"prompt & model versioning"," and ",[22,255,257],{"href":256},"\u002Fblog\u002Fproduction-ai-observability","production AI observability",". The ",[22,260,262],{"href":261},"\u002Fnewsletter","newsletter"," sends one finance-systems pattern, product decision or build lesson every two weeks.",{"title":265,"searchDepth":266,"depth":266,"links":267},"",2,[268,269,270,271,272,273],{"id":30,"depth":266,"text":31},{"id":44,"depth":266,"text":45},{"id":84,"depth":266,"text":85},{"id":179,"depth":266,"text":180},{"id":221,"depth":266,"text":222},{"id":232,"depth":266,"text":233},"engineering",null,"2026-07-28","A model will eventually return garbage, the wrong shape, or a confident lie. How I design output contracts, validation, fallbacks and graceful degradation so the product fails safely.",false,"md",[281,284,287],{"question":282,"answer":283},"What is an output contract for an AI feature?","An output contract is the defined shape and constraints the model's response must satisfy for your code to safely use it — a JSON schema, required fields, allowed values, length bounds. It turns the model's output from free-form text your code hopes to parse into a validated interface with a pass\u002Ffail check. The reason it matters is that a model will eventually return the wrong shape — an extra field, prose where you expected JSON, a value out of range — and without a contract that's validated, your code either crashes or, worse, silently processes a malformed result. With a contract, an off-spec response is caught at the boundary and handled, instead of flowing downstream as if it were valid.",{"question":285,"answer":286},"How do you handle AI failures in a product?","By assuming the model will fail and designing the failure path as deliberately as the success path. The layers are: validate the output against a contract so malformed responses are caught; retry (often with a corrective instruction) when validation fails; fall back to a safe alternative — a simpler model, a cached result, or a graceful 'we couldn't do that' — when retries don't work; and never let a failed AI call take down the surrounding feature. The goal is graceful degradation: when the AI part fails, the product does something sensible and honest rather than crashing or, worst of all, presenting a confident wrong answer as if it were correct.",{"question":288,"answer":289},"How do you handle hallucinations in an AI product?","You can't fully prevent them, so you design so they do the least damage: constrain the model with an output contract and grounding data so there's less room to invent; validate what you can (a cited source that doesn't exist, a value out of range, a format that's wrong); keep a human in the loop for high-stakes outputs rather than auto-acting on them; and be honest in the UI about confidence rather than presenting every answer as certain. The mindset is that a hallucination is a failure mode to contain, not an edge case to ignore — the dangerous version is the confident wrong answer that looks exactly like a right one, so the defences are about catching or bounding it, not pretending it won't happen.",{},true,9.1,"\u002Fblog\u002Fai-output-contracts-and-failure-handling","building-ai-products","AI output contracts failure handling","annual","reviewed","Tan Gravam","informational",{"title":5,"description":277},"blog\u002Fai-output-contracts-and-failure-handling",[303,304,305,306,307,308],"ai-products","output-contracts","failure-handling","fallback","reliability","solo-saas","text","UOhRppj_V70xgMSnLiQc_uo3lcG2vNOYRN4gVg1Ly28",[312,316],{"title":313,"path":314,"stem":315,"type":309,"language":275,"draft":278,"children":-1},"AI Evaluation & Regression Testing","\u002Fblog\u002Fai-evaluation-and-regression-testing","blog\u002Fai-evaluation-and-regression-testing",{"title":317,"path":318,"stem":319,"type":309,"language":275,"draft":278,"children":-1},"AI Product Unit Economics: Cost, Margin and Control","\u002Fblog\u002Fai-product-unit-economics","blog\u002Fai-product-unit-economics",[321,324,327],{"path":79,"title":322,"description":323},"Prompt & Model Versioning: Production Governance for AI Products","A prompt is production code and a model is a dependency you don't control. How I version prompts, handle model changes, run eval datasets, and keep outputs from silently regressing.",{"path":256,"title":325,"description":326},"Production AI Observability: Watching What the Model Does","You can't fix what you can't see. How I think about observing AI features in production — quality, cost, latency, drift and failures — so problems show up as data, not churn.",{"path":314,"title":313,"description":328},"How I build eval datasets for AI features — the cases, the scoring, and the regression gate — so a prompt or model change is judged on evidence, not on trying three examples and hoping.",1785237637128]