Why do coding models generate outdated API calls?

Models are trained on static corpora with a fixed cutoff, and the public repositories in those corpora already contain a great deal of outdated usage. When a library changes after that cutoff, or when older usage dominates the training data, the model reproduces what it learned. Researchers describe the resulting failure as a context-memory conflict between what the model was trained on and what it is told at inference time.

Is there proof that AI generates vanilla JavaScript more accurately than framework code?

No published benchmark isolates that comparison. The reliability argument for platform APIs follows from the documented mechanism: the measured failure mode is version-specific and stable APIs like fetch and querySelector do not change in ways that break generated code. That is a well-grounded inference rather than a measured result, and React's heavy representation in training data is a genuine counterweight.

Do tools like Context7 solve the version problem?

They mitigate it substantially. Context7 is an MCP server that identifies the library in a prompt, fetches current version-specific documentation, and injects it into the model's context. It has over 56,000 GitHub stars and indexes more than 104,000 libraries. Research on API evolution found that supplying documentation improves adoption but does not fully resolve the problem, since models still revert to stale training knowledge. Injection also costs context, 5,000 tokens per documentation call by default, plus added latency.

Is vanilla JavaScript cheaper to generate with AI?

Unknown. The reasoning that fewer version errors should mean fewer retries and lower token spend is plausible, but no study measures token cost or retry rates for vanilla against framework code generation. Published cost comparisons in 2026 compare models to each other rather than code styles.

The package specifier react@??? with the version replaced by question marks, representing an AI model's uncertainty about which library version to target

AI Code Generation Has a Version Problem

Q: Does giving the model current documentation fix version errors?

It helps but does not resolve the problem. A 2026 survey of API-evolution failures found that supplying external documentation significantly improves API adoption, yet models still frequently revert to stale parametric knowledge, either sticking to old APIs or omitting the updated one entirely.

August 2, 2026 · 8 min read

Ask a coding model to write something against a library that changed last year and it will often produce code for the version it learned, not the version you have. This is measured, not anecdotal. On GitChameleon 2.0, a benchmark built from real documented breaking changes rather than synthetic ones, state-of-the-art models succeed on version-conditioned generation between 48% and 51% of the time. That is close to a coin flip on the question of whether generated code targets the version actually installed.

The failure has a specific cause, and that cause points somewhere interesting for anyone choosing a stack in 2026.

Rundown

On version-conditioned code generation, current frontier models score 48% to 51% on a benchmark built from real historical breaking changes.
Work by Wang et al. measured deprecated API usage in LLM output at 25% to 38% across eight Python libraries, and found rates of 70% to 90% when the target function was itself outdated, against 9% to 18% when it was current.
The documented cause is a mismatch between static training corpora and libraries that keep moving, which researchers describe as a context-memory conflict.
Supplying current documentation improves adoption but does not resolve it, since models frequently revert to what they learned during training.
An entire tooling category exists to patch this: Context7, an MCP server that injects version-specific documentation at query time, has over 56,000 GitHub stars and indexes more than 104,000 libraries.
The reliability implication for platform APIs is real, but it’s an inference from the mechanism rather than a measured comparison. No published benchmark isolates vanilla JavaScript against framework code.
The cost question is mostly open, though documentation injection is one named overhead framework work carries and platform work avoids: 5,000 tokens per call by default.

What the research measures

Three findings sit underneath the argument, and they’re worth separating.

The first is version conditioning. GitChameleon 2.0 evaluates whether a model can write code for a specified library version, using documented historical breaking changes and executable tests rather than string matching. Frontier models land between 48% and 51%, a result also reported in a 2026 survey of API-evolution failures. The benchmark’s own framing is that version-conditioned generation is effectively out-of-distribution work for these models.

The second is deprecation. Work by Wang et al., presented at ICSE 2025, measured deprecated API usage across eight Python libraries at 25% to 38% of outputs, and split the question further by target: when the function being used was outdated, models produced deprecated usage 70% to 90% of the time, against 9% to 18% when the function was current. The authors attribute the gap to training corpus composition, noting that up-to-date functions outnumbered outdated ones roughly two to one in their collected data.

The third is that documentation only partly helps. That same 2026 survey found that supplying external documentation significantly improves API adoption, yet models still struggle to override stale parametric knowledge, frequently sticking to old APIs or omitting the updated one. The paper frames this as a context-memory conflict: what the model was trained on competes with what it was just told.

The failure mode is churn

These studies do not find that models are bad at libraries. They find that models are bad at which version of a library. APILOT’s analysis traces it to training data directly: models are trained on static corpora with a fixed cutoff, and public repositories are themselves full of outdated usage, so obsolete patterns get learned and reproduced.

That distinction matters. A model’s accuracy on a given API is partly a function of how consistently that API appears across its training data. An interface that has worked the same way for a decade appears one way. An interface that changed three times in five years appears three ways, with no reliable signal about which one the current project wants.

The ecosystem built tooling around it

The clearest evidence that this is a live problem rather than a solved one is how much infrastructure exists to work around it. Context7, an MCP server built by Upstash, identifies which library a prompt refers to, fetches current version-specific documentation, and injects it into the model’s context before it answers. It has passed 56,000 GitHub stars and indexes more than 104,000 libraries, with setup guides for Cursor, Claude Code, and Windsurf still being published through mid-2026.

Tools like it are a genuine mitigation, and the reason they exist is worth stating plainly: a model’s default answer about a library is whatever dominated its training data, not whatever is in the project. Injecting documentation shifts that, though the API-evolution research above found that models still revert to stale parametric knowledge even when current documentation sits in context.

What that implies for platform APIs

Here the argument moves from measurement to inference, and that line should be explicit.

fetch, querySelector, addEventListener, and URLSearchParams have not changed. The web platform’s backward-compatibility commitment means they cannot change in the ways that break generated code, because removing or altering them would break too much of the existing web. Every training example touching them agrees with every other one. The context-memory conflict the research describes has nothing to conflict about.

Framework APIs are the opposite case by design. React class components and hooks are both extensively represented in any corpus scraped before 2024. Next.js pages-router and app-router code both appear in volume, describing incompatible mental models for routing and data fetching. A model asked to write a Next.js route has seen both, and the correct answer depends on a project detail it may not have been told.

So the inference is straightforward: the specific failure mode these papers measure is one that stable platform APIs largely do not expose. What no study appears to have done is measure that directly. A benchmark comparing generation accuracy on vanilla DOM and platform APIs against equivalent framework tasks would settle it, and none was found in this research. Treat the reliability argument as well-grounded in mechanism and unproven in direct comparison.

There is also a real counterweight worth stating. React is one of the most heavily represented technologies in any code corpus, and volume of training data improves generation quality on its own. Whether consistency beats volume is an empirical question, and the honest answer is that nobody has published the comparison.

The cost question is open

A reasonable chain of reasoning says that more reliable generation means fewer retries, fewer correction rounds, and less context spent explaining which version is in use, and that this should show up as lower token spend. It’s plausible. It is not something this research supports.

There is one concrete, named cost that framework work incurs and platform work does not. Documentation injection is not free: each Context7 documentation call pulls a chunk into the context window, 5,000 tokens by default, and adds latency to the request. Code targeting fetch or querySelector needs no such call, because there is no version to resolve.

That is a real overhead with a real number attached, and it still does not settle the question. No study found here measures total token cost or retry rates for vanilla against framework code generation, and doc injection is one line item rather than a full accounting. The cost figures that do exist in 2026 compare models to each other rather than code styles: open-weight options like MiniMax M2.5 now reach 80.2% on SWE-bench Verified at a fraction of frontier pricing, which says a great deal about the model market and nothing about whether vanilla is cheaper to generate. Anyone claiming a specific cost advantage for vanilla, including anyone reading this, is extrapolating.

The same caution applies to local models. Open-weight models have closed most of the benchmark gap with proprietary systems, and a smaller model plausibly has less capacity to hold many conflicting framework versions, which would favor stable targets. Plausibly. No benchmark found here splits results that way.

What this changes in practice

The practical takeaway is narrower than the headline and more useful. When generated code targets a library, the version is a real variable and worth stating explicitly in the prompt, because the model’s default is whatever dominated its training data rather than whatever is in package.json. When generated code targets platform APIs, that variable mostly disappears.

That is a genuine argument in favor of the platform for AI-assisted work, and it is one argument among several rather than a decisive one. It sits alongside the security, reliability, and developer-experience cases rather than replacing them.