Models, roadmaps and AI product debt
The model under your product will change. This module helps you choose models, migrate safely and plan roadmaps that survive fast-moving AI.
Learning objectives
- Treat models as changing dependencies
- Select models on your own evals and terms
- Plan a safe model migration
- Run a three-horizon roadmap with kill criteria
5.1 Models have lifecycles
Model providers release new models frequently and retire old ones on published schedules. A new version can be better overall yet behave differently on your task — longer answers, different formatting, new refusals. Treat the model as a dependency that will change, not a fixed component. Check your providers' deprecation pages and plan migrations before the deadline arrives.
5.2 Choosing models
| Criterion | What to check |
|---|---|
| Quality on your evals | Your eval set, not public leaderboards |
| Latency and cost | At your real prompt sizes and volumes |
| Context and capabilities | Context window, tool use, images, structured outputs |
| Data terms | Retention, training use, regions where data is processed |
| Reliability | Uptime, rate limits, support, deprecation notice periods |
Many products use several models — a small fast one for routing and simple tasks, a larger one for hard cases — and keep a tested fallback provider for outages.
5.3 A model migration playbook
- Run the eval set on the new model with your current prompts
- Adjust prompts where behaviour differs, and re-run
- Shadow test — run the new model on a copy of live traffic without showing users, and compare
- Staged rollout with quality, cost and complaint monitoring, as in any launch
- Keep the old configuration ready to switch back until the new one is proven
This only works if prompts, model settings and eval results are versioned like code, so you always know what's running and can roll back. See LLMOps.
5.4 Roadmaps with three horizons
| Horizon | What goes here | How to manage it |
|---|---|---|
| Now | Features with proven eval results and clear demand | Committed dates and outcomes |
| Next | Bets with promising prototypes | Time-boxed experiments with explicit kill criteria |
| Later | Ideas waiting on model capability, data or regulation | Re-test with each major model release; no dates |
Kill criteria matter more in AI than elsewhere, because a demo that works 70% of the time is seductive. Agree in advance: "If we can't reach 90% on the eval set within six weeks, we stop or rescope."
5.5 AI product debt
- Prompt sprawl — dozens of slightly different prompts nobody owns
- Eval rot — eval sets that no longer reflect real users or current policy
- Hidden dependencies — features relying on quirks of one model version
- Unmeasured features — AI features shipped without monitoring, which nobody dares change
Budget time every quarter to pay this down, just as engineering teams do for technical debt.
Your provider announces that the model behind your main AI feature will be retired in four months. Write the migration plan: which evals you'll run, who reviews differences, the shadow-test and rollout stages, and what would make you choose a different provider instead.
See a worked example
Example: Week 1: run the 400-case eval set on two candidate models. Week 2–3: prompt adjustments; support leads review 100 side-by-side outputs. Week 4–5: shadow test on 10% of traffic. Week 6–9: staged rollout 10% → 50% → 100%. Switch provider if no candidate reaches the current must-never and accuracy thresholds within budget.
Key takeaways
- Models are released and retired on schedules — plan for it.
- Choose on your evals, cost, latency, data terms and reliability.
- Evals → prompt fixes → shadow test → staged rollout → keep rollback.
- Now, next, later — with kill criteria for bets.
- Pay down prompt sprawl, eval rot and hidden dependencies.