Shipping a Feature Powered by a Model You Don't Control

TL;DR — A hosted model is a dependency that is slow (seconds), expensive (per call), non- deterministic, occasionally wrong, and can change under you. Eight decisions handle it: pin the model version, never put it on the critical path without a fallback, constrain and validate the output, treat latency as UX rather than infrastructure, cancel on navigate-away, budget per user, redact before sending, and cache on a normalised key. Then understand the failure class your dashboards can't see: soft failures — HTTP 200, well-formed, and confidently wrong.

Five ways it isn't a normal API call

Property	Normal backend	Model API
Latency	50–300ms	1–10s, high variance
Determinism	Same input, same output	Same input, different output
Cost	Effectively free per call	Cents per call, scales with input
Correctness	Right or an error	Plausibly wrong with a 200
Stability	You deploy changes	Provider can change it

Every one of those has a design consequence, and skipping any of them produces a specific, predictable incident.

The eight engineering decisions

1. Pin the model version

// ❌ your feature changes behaviour on a Tuesday
model = "provider-latest"

// ✅ you decide when behaviour changes
model = "provider-model-2026-08-14"

An alias that tracks the newest release means your feature's behaviour can shift with no deploy on your side, no changelog you control, and no correlation to anything in your release notes. You'll spend a day investigating a regression you didn't cause.

Pin it, and treat a version bump as a change that goes through your eval suite and a staged rollout — same as any other.

2. Never on the critical path without a fallback

val summary = runCatching { ai.summarise(article) }
    .getOrElse { Summary.FirstParagraph(article) }

The question to answer at design time: what does this screen do when the model is unavailable? If the answer is "it's broken", you've made a third party's uptime into your own.

Good fallbacks are usually unglamorous and already exist: the first paragraph instead of a summary, the chronological list instead of the smart ordering, the manual form instead of the extraction. Ship the fallback path first, then add the model on top of it — that ordering guarantees the fallback actually works, because it's the only thing that worked for a week.

3. Constrain the output, then validate it anyway

Structured output modes and tool schemas help enormously and are not a guarantee.

@Serializable
data class Extracted(
    val merchant: String,
    val amountMinor: Long,
    val date: LocalDate,
)

val result = runCatching {
    json.decodeFromString<Extracted>(response.text)
}.getOrNull()

// And then check it means something.
if (result == null || result.amountMinor <= 0) {
    return ExtractionResult.NeedsManualEntry
}

Two layers: does it parse, and is it sane. The second matters more than it looks — a model will occasionally return a perfectly valid amountMinor = 0 or a date in 1970, and your schema is happy with both.

4. Latency is a UX problem, not an infrastructure problem

Six seconds behind a blank spinner is a failed interaction even when the request succeeds. You cannot make the model fast; you can make the wait legible.

• Stream. Partial output arriving at 400ms beats complete output at 6s, by a wide margin, on every measure users care about.
• Show what you're doing. "Reading your receipt…" is not a trick; it's the honest state.
• Let them leave. A cancel affordance turns a trapped wait into a choice.

5. Cancel on navigate-away

// viewModelScope cancels with the ViewModel;
// a stray GlobalScope call bills you for output nobody reads.
viewModelScope.launch {
    val result = ai.summarise(article)
    _state.update { it.copy(summary = result) }
}

Structured concurrency does this for free if you use it. The point is that an abandoned generation isn't just wasted — it's billed, and on a screen users bounce off, it can be a meaningful share of your spend.

6. Budget per user, and alert on it

Per-call costs mean a bug is a bill. A retry loop, a LaunchedEffect with an unstable key, a user holding the regenerate button — any of these can produce hundreds of calls before anyone notices.

Cap it server-side (a per-user, per-day ceiling), degrade gracefully when the cap is hit, and alert on the p99 of calls-per-user rather than on the total, because the total looks fine while one account is on fire.

7. Redact before you send

Send the minimum. Strip names, emails, card numbers, precise location, anything you can process locally. Know your provider's retention and training policy, and write it down where your legal and product people can find it — that question always arrives, usually late.

The cheapest privacy incident is the one where you never sent the data.

8. Cache on a normalised key

More prompts repeat than you'd expect — the same article, the same product, the same document. Normalise (trim, lowercase where safe, strip volatile fields), hash, and cache. Include the model version in the key, so pinning and caching don't fight each other.

The product decisions that go with them

Say it's AI-generated, plainly. Not a disclaimer nobody reads — a small, honest label. It sets the right verification posture in the user, and it's the difference between a wrong answer being a disappointment and being a betrayal.

Always leave the manual path visible. Not buried behind "having trouble?" — visible. Users who don't trust the feature will use it, and users who do will need it when it's wrong.

Make feedback one tap. You need this data more than you think: it's how you build the evaluation set that the next post is about. A thumbs-down with the input attached is worth more than a hundred analytics events.

The failure class your dashboards can't see

Hard failures — timeout, 500, rate limit, malformed output. Your monitoring already catches these, they page someone, and they get fixed.

Soft failures — HTTP 200, well-formed, parsed cleanly, and wrong. The summary that misses the point. The extraction that reads the wrong total. The answer that's confidently false.

Your success rate says 99.9%. Your latency is fine. Your error budget is untouched. And a meaningful share of users are being told something untrue by your app.

This is the entire risk profile of a model-backed feature, and nothing in conventional observability will show you one. Detecting it requires a different discipline — an evaluation set, production quality signals, and a human reading samples — which is the subject of the next post in this series.

The rule

Treat it as an unreliable, slow, expensive, non-deterministic dependency, and build a feature that survives all four — because it will need to. Pin the version, ship the fallback first, validate what comes back, make the wait honest, and assume that the most dangerous failure will arrive with a 200.