Context Engineering in 2026: What Replaced Prompt Engineering

Your system prompt is carefully worded. Your instructions are unambiguous. And your agent still loses the plot forty tool calls into a long task, forgetting a constraint you stated explicitly at the start.

That isn’t a prompting problem. It’s a context problem — and it’s why the discipline has quietly renamed itself.

Context engineering is the practice of deciding what actually enters the model’s window at every turn. Anthropic, in its guide on the subject, defines it as «the set of strategies for curating and maintaining the optimal set of tokens during LLM inference» — and frames it as the natural successor to prompt engineering rather than a replacement.

This guide covers what’s really in the window, why a bigger one doesn’t save you, and the three techniques that keep long-running agents coherent.

The Prompt Is a Small Part of the Context

Here’s the mental shift. When people say “the model reads my prompt”, they picture a text box. What the model actually reads on any given turn is:

• System instructions — the behaviour contract
• Tool definitions — every tool you exposed, including everything an MCP server advertises
• Tool results — the raw output of every call made so far
• Retrieved documents — whatever your RAG layer pulled in
• Few-shot examples — the patterns you’re demonstrating
• Message history — the entire conversation up to now

The prompt is one item on that list, and usually not the largest one. In a long agent run, tool results alone can dwarf everything else.

And here’s what makes it an engineering problem rather than a writing problem: an agent running in a loop generates new context on every turn. Each tool call produces output that may or may not matter for the next decision. Something has to decide what survives. That something is you.

Prompt engineering is a discrete task — write it, refine the wording, ship it. Context engineering is iterative: the curation decision happens every single turn.

Why a Bigger Window Doesn’t Fix It

The intuitive response is “wait for longer context windows”. That’s not a strategy, and the reason is measurable.

Context rot

As the number of tokens in the window grows, the model’s ability to accurately recall information from it decreases.

This isn’t folklore. Chroma published a technical report on exactly this in July 2025, testing 18 models — Claude Opus 4 and Sonnet 4, o3, GPT-4.1, Gemini 2.5 Pro, Qwen3 and others. Crucially, they held task difficulty constant and varied only input length, which isolates the effect that most long-context benchmarks confuse with “longer inputs are harder tasks”.

Their finding across every experiment: performance consistently degrades as input length increases. The full codebase is public, so the result is reproducible rather than anecdotal.

Anthropic reaches the same conclusion, noting that while some models degrade more gently than others, the effect «emerges across all models».

Information being technically present in the window is not the same as the model reliably using it.

The architectural reason

This isn’t a bug someone will patch. Transformers let every token attend to every other token, which means n² pairwise relationships for n tokens. As context length grows, the model’s capacity to hold those relationships gets stretched thin — attention is a finite quantity being spread across a larger surface.

Models are also trained on data where short sequences dominate, so they have less experience and fewer specialised parameters for context-wide dependencies.

The result, in Anthropic’s framing, is a performance gradient rather than a hard cliff: models stay capable at long contexts but lose precision on retrieval and long-range reasoning.

Context is a budget

The useful mental model — and the one that will feel familiar if you do capacity planning — is that models have an attention budget. Every token you add depletes it.

Which gives the single principle worth memorising:

Find the smallest possible set of high-signal tokens that maximise the likelihood of your desired outcome.

Not the most information you can fit. The least you can get away with.

And there’s a clean experiment showing this isn’t theoretical. Chroma ran the LongMemEval benchmark two ways on identical questions: a focused input containing only the relevant excerpts (~300 tokens), and the full input with all the surrounding conversation (~113,000 tokens). Same question, same answer available in both.

Every model performed significantly better on the focused version. The information was present either way — the difference was entirely in how much noise the model had to wade through to reach it.

That gap is the value of context engineering, measured.

Anatomy: What Belongs in the Window

System prompts — find the right altitude

Two failure modes sit at opposite ends.

At one extreme, engineers hardcode brittle if-else logic into prompts trying to script exact behaviour. It’s fragile and grows unmaintainable. At the other, they write vague high-level guidance that gives the model no concrete signal and assumes shared context that doesn’t exist.

The target is between them: specific enough to guide behaviour, flexible enough to let the model apply judgement. Organise into clear sections — background, instructions, tool guidance, output format — and aim for the minimal set of information that fully specifies the behaviour you want. Minimal doesn’t mean short.

Tools — the most common failure mode

This is where most agents actually break, and Anthropic is blunt about it: bloated tool sets that cover too much or create ambiguous decision points are the failure they see most.

The test is elegantly simple:

If a human engineer can’t definitively say which tool should be used in a given situation, an AI agent can’t be expected to do better.

Every tool definition also costs tokens on every single turn. Twenty tools you exposed “just in case” are twenty descriptions competing for attention against the actual task.

The Chroma work quantifies why this hurts. They tested distractors — content that looks topically relevant but doesn’t answer the question — and found that even a single distractor measurably degrades performance against a clean baseline. Four of them compound the damage, and the effect grows worse as input length increases.

The failure modes also differ by model family, which is worth knowing when you pick one: Claude models tend to abstain when uncertain, explicitly saying they can’t find an answer, while GPT models more often produce confident but wrong responses. A near-miss tool description is exactly this kind of distractor.

This is exactly why tool filtering at the gateway matters — and why it’s a context engineering technique rather than purely a security one. When you restrict which tools a given key can reach, you’re not only enforcing least privilege, you’re shrinking the context every request carries. I covered the mechanics in user-level permission controls for MCP tool access; the security framing and the context framing turn out to be the same lever.

Examples — canonical, not exhaustive

Few-shot examples work. What doesn’t work is stuffing every edge case you’ve ever encountered into the prompt in the hope of covering all rules.

Curate a small set of diverse, canonical examples that portray the expected behaviour. As Anthropic puts it, for a model, examples are the pictures worth a thousand words — and you don’t need a thousand pictures.

Just-in-Time Beats Pre-Loading

Most AI applications today use embedding-based retrieval before inference: fetch everything possibly relevant, stuff it in, hope the model finds what it needs.

The agentic alternative is just-in-time retrieval. The agent holds lightweight identifiers — file paths, stored queries, URLs — and loads data at runtime through tools, only when it turns out to be needed.

Claude Code works this way: rather than indexing an entire codebase into context, it uses primitives like glob and grep to navigate and pull files on demand, sidestepping stale indexes entirely.

Two things make this work better than it sounds:

Metadata carries signal. A file called test_utils.py in tests/ means something different from the same name in src/core_logic/. Folder hierarchies, naming conventions and timestamps tell an agent how and when to use something — the same cues a human engineer reads.

Progressive disclosure. The agent discovers context in layers. File sizes hint at complexity, names hint at purpose, timestamps hint at relevance. It builds understanding incrementally while keeping only what’s necessary in working memory.

The trade-off is real: runtime exploration is slower than reading precomputed data, and a poorly guided agent will burn context chasing dead ends. Which is why the strongest setups are hybrid — load a little up front for speed, let the agent explore from there. Claude Code does exactly this: CLAUDE.md goes in naively at the start, everything else is fetched on demand.

If your retrieval layer currently loads everything up front, the RAG pipeline patterns I’ve written about are the natural starting point for making it selective instead.

Three Techniques for Long Tasks

When a task spans tens of minutes or hours — a large migration, a research project — the token count will exceed any window. Three techniques address this directly.

Compaction

Take a conversation nearing the limit, summarise it, restart with the summary.

In Claude Code this means passing the message history back to the model to compress: architectural decisions, unresolved bugs and implementation details are preserved; redundant tool outputs and superseded messages are discarded. The agent continues with that summary plus the handful of most recently touched files.

The craft is in what you drop. Over-aggressive compaction loses subtle context whose importance only becomes obvious later. Tune the compaction prompt by first maximising recall — capture everything relevant — then improving precision.

The safest starting point is tool result clearing: once a tool has been called deep in the history, the agent almost never needs its raw output again.

Structured note-taking

Have the agent write notes to persistent memory outside the window, then pull them back when needed. A NOTES.md, a to-do list, a scratch file.

This is how agents maintain coherence across context resets: the window empties, the notes survive, the agent reads itself back into the task. It’s cheap, it’s transparent, and unlike compaction it’s lossless for anything the agent chose to record.

Sub-agent architectures

Instead of one agent carrying the whole project, spawn specialised sub-agents with clean context windows for focused subtasks. Each may burn tens of thousands of tokens exploring, then returns a condensed summary of roughly one to two thousand.

The lead agent never sees the mess — only the distilled result. Detailed search context stays isolated where it belongs, and the coordinator keeps its window for synthesis.

Which one to reach for

They compose. A lead agent can run compaction on its own thread while delegating exploration to sub-agents that keep their own windows clean.

The Part Nobody Mentions: Context Costs VRAM

There’s a hardware dimension to this that rarely comes up in AI-engineering discussions.

During generation, the model caches keys and values for every token processed so far — the KV cache — and it grows linearly with context length. On a self-hosted setup this is not a rounding error; a long context can add many gigabytes on top of the model weights, and it’s the single most common reason a model that “should fit” crashes.

So a bloated context costs you three times over: worse recall, higher latency and spend, and actual gigabytes of VRAM. If you’re running models yourself, I worked through that memory math in quantization explained — the KV cache section is the part that intersects directly with context engineering.

A Practical Checklist

Before blaming the model:

1. Count your tools. Can you say definitively which tool applies in each situation? If not, neither can the agent. Prune or filter per request.
2. Clear stale tool results. The cheapest win available — old raw outputs almost never earn their tokens.
3. Check the altitude of your system prompt. Hardcoded if-else logic and vague hand-waving are both failure modes.
4. Trim your examples to a few canonical ones instead of an exhaustive rulebook.
5. Move from pre-loading to just-in-time wherever your data is dynamic enough to justify it.
6. Pick a long-horizon strategy deliberately — compaction, notes, or sub-agents — rather than hoping the window holds.
7. Measure. If quality drops as sessions get longer, that’s context rot, not a bad prompt.

The Bottom Line

Prompt engineering hasn’t died — it’s been absorbed. Writing a good instruction is still necessary; it’s just no longer the whole job, because the prompt is a minority shareholder in a window full of tools, results, documents and history.

The shift in mindset is from writing to budgeting. Context is a finite resource with diminishing returns, and the discipline is spending it deliberately: fewer tools, cleaner history, data fetched when needed instead of hoarded up front.

Anthropic’s own conclusion is the right note to end on — even as models improve, «treating context as a precious, finite resource will remain central to building reliable, effective agents». Bigger windows raise the ceiling. They don’t remove the need to choose.