All posts
LLMs

When Your LLM Gets Retired: What You Actually Need to Do

Narasimha Badrinath6 min read

A while back I got an email from OpenAI. A model I had been running in production, one I had carefully prompted, tested, and tuned, was being deprecated, with 60 days' notice. I needed to migrate to a newer model.

I thought it would be a two-minute change. Swap the model string, deploy, done.

It was not.

The Illusion of the Easy Swap

Changing the model ID is the first thing you do. It is also the least important thing.

Models trained on different data, with different RLHF and different safety tuning, behave differently. Same prompt, different output. Sometimes subtly. Sometimes dramatically. The new model might be more verbose, more cautious, or interpret your system prompt in a way the old one never did.

I had a classifier that was quietly misrouting roughly 12% of requests after the "easy swap." No errors. No crashes. Just wrong answers, at scale, for three days before we caught it.

The string change is one line. The validation is the work.

What Actually Changes

Prompt behavior shifts. System prompts tuned for one model may not land the same way on another. Persona instructions, tone guidance, chain-of-thought nudges: all of these interact with the model's training in ways you cannot fully predict. Expect to re-tune.

Function calling and tool use. If you use tools or structured output, this is your highest-risk area. Models differ in how aggressively they call tools, how they handle parallel calls, and how strictly they follow your JSON schema. Every tool flow needs explicit re-testing, not just a smoke test.

Context window and tokenizer. Newer models often have larger context windows, but different tokenizers mean the same text takes up different numbers of tokens. If you have chunking logic, retrieval pipelines, or prompt assembly that makes assumptions about token counts, check them.

Sampling parameters. The same temperature value produces different variance across models, and some newer models do not accept it at all. Depending on the model and provider, a non-default temperature or top_p is either rejected with an error or silently ignored. Check which parameters the new model still honors, and do not assume a low temperature still gives you consistent output.

Cost and latency. Newer models are often cheaper per token, but reasoning models can burn many more tokens and take longer to answer. Benchmark on your actual workload before assuming a win. A model that is 2x faster but needs more tokens to get the same result may not save you money.

The Pattern That Saves You Next Time

The real lesson is not how to migrate. It is how to make migration cheap.

Wrap your model calls behind a configuration layer. One place where the model name, the provider, and the default parameters live. Not scattered across forty files. When the next deprecation email arrives, you change a config value, adjust any parameters the new model dropped, and run your test suite.

# Not this: scattered in every file
response = client.chat.completions.create(
    model="old-model-id",   # repeated in every file
    ...
)

# This: one place to change
response = llm_client.complete(
    messages=messages,
    task="classification"  # resolved to a model from config
)

The abstraction does not need to be complex. A thin config-driven wrapper that maps task types to models gives you routing flexibility, a single migration point, and a natural place to add logging and cost tracking.

What a Config Swap Will Not Fix

A few migration problems live outside the model string, and a config layer does not hide them.

Pin snapshots, not aliases. An alias such as a "latest" tag can change under you without a deprecation email. Pin a dated snapshot, and put the provider's retirement page on your calendar.

Reasoning and thinking settings. Reasoning effort and thinking budgets change cost, latency and output length. If the new model has them, they belong in your config next to the model name.

Things that do not migrate. A fine-tune is tied to the base model it was trained on. It stops working when that base model is retired, and moving to a new one means re-training, if the provider still offers fine-tuning for it. A new embedding model means re-embedding and re-indexing everything, which is a project of its own.

Limits and caches. Rate limits and quotas are often tracked per model or model family, so a new model can start with different, sometimes lower, limits. Regional availability can differ, and prompt caches do not carry over, so expect a cold start. Check each before the cutover, not after.

Build Your Eval Suite Now

The most expensive time to write evals is during a migration under deadline pressure.

Pick your ten most important prompts, the ones where the output actually matters for your product. Write assertions. Not "does it return JSON" but "does the JSON contain the right keys, with plausible values, for these representative inputs." Run them on every model change.

In my experience, ten well-chosen eval cases catch most of the regressions that matter. Treat that as a floor, not a guarantee, and add a case every time a bug slips through. The goal is not comprehensive coverage. It is a fast signal that something important broke.

The Migration Checklist

When you get that deprecation notice:

  1. Map every place in your codebase where the old model is referenced, and note the new model's rate limits and regional availability
  2. Verify token counts and chunking logic against the new tokenizer
  3. Check which sampling and reasoning parameters the new model accepts, and update your config
  4. Stand up the new model in a shadow environment: same traffic, compare outputs
  5. Run your eval suite against both models and diff the failure rates
  6. Check tool call behavior end-to-end for every tool you use
  7. Benchmark cost and latency on your actual workload
  8. Migrate in a canary: 5% of traffic first, watch error rates and business metrics
  9. Only then cut over fully

The deprecation deadline is real. The temptation to go straight from step 1 to production is also real. Resist it.

The Bigger Picture

Model deprecations are not going to slow down. The pace of model releases (from OpenAI, Anthropic and Google, among others) means the model you are running today has a shorter shelf life than it used to.

The developers who handle this well are not the ones who react fastest when the deprecation email lands. They are the ones who built their systems to be model-agnostic from the start.

Treat the model as a dependency, not a constant. Version it. Test against it. Abstract it. And when the next deprecation notice arrives, your answer to the question "how long will this take?" should be: a day, not a sprint.