What is context engineering? Why the information you give AI matters as much as the prompt

Picture two people using the same AI model on the same task. One pastes in a 200-page contract, three old email threads and a long list of instructions, then asks a question. The other gives the model the two clauses that matter, a one-line note on what the answer is for, and the question. The second person usually gets the better answer, and pays less for it. The model didn't change. What changed was the information it was working from.

That is the core idea behind "context engineering," a term that moved from developer slang to a named discipline over the past year. This article explains what it means, why giving a model more information can make it worse, what the research and early field results show about doing it well, and where the evidence is still thin. It builds on our earlier piece on prompting. Prompting is about how you phrase a request. Context engineering is about everything else the model can see when it answers.

What context engineering means

Every time a large language model (LLM) produces an answer, it reads a block of text called its context window: the instructions, the conversation so far, any documents or search results, and the outputs of any tools it has used. The model can't see anything outside that window. It has no other view of your files, your customers or what happened yesterday. Text is measured in tokens, chunks of roughly three-quarters of a word.

Anthropic's Applied AI team defines context engineering as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts"¹. Inference just means the moment the model is running and generating an answer. Their guiding principle is to find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome"¹.

A 2025 survey that reviewed more than 1,400 research papers frames it the same way, as "the systematic optimization of information payloads for LLMs," and sorts the work into three jobs: retrieving or generating the right context, processing it, and managing it over time². In practice those jobs show up in familiar systems: retrieval-augmented generation (RAG, where a system looks up relevant documents and adds them to the context), memory, tool use, and teams of AI agents.

A useful analogy is a briefing pack for a busy executive. A good pack isn't the thickest one. It holds what the reader needs for this decision, in an order that makes the key point hard to miss, with everything else a phone call away. Context engineering is writing that pack, over and over, often automatically, for a reader that takes everything in it literally.

The term matters more now because of agents: AI systems that take many steps on their own, calling tools, reading files and deciding what to do next. A chatbot answers once. An agent may make dozens of tool calls on a single task, and each one adds text to its context. Without active management, the window fills with old search results, stale plans and dead ends.

Why more context can make a model worse

The intuitive approach is to give the model everything and let it sort things out, especially now that leading models accept hundreds of thousands of tokens. A consistent body of research says that doesn't work as well as it sounds.

Position matters. In a widely cited study published in Transactions of the Association for Computational Linguistics, Nelson Liu and colleagues gave models a question plus a set of documents, only one of which contained the answer. Performance was "often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle"³. In the worst cases, GPT-3.5-Turbo did worse with 20 or 30 documents and the answer buried in the middle than it did with no documents at all (its closed-book score was 56.1%)³. Those were 2023-era models, but the pattern has shown up again in every study cited below.

Advertised context length overstates usable length. Researchers behind a test suite called RULER evaluated 17 long-context models. All claimed to handle at least 32,000 tokens, but "only half of them can maintain satisfactory performance at the length of 32K"⁴. A later benchmark, NoLiMa (accepted at ICML 2025), removed an easy shortcut: the wording of the question no longer matched the wording of the hidden fact, so models had to make a small inference instead of matching words. Of 13 models claiming at least 128,000 tokens of context, 11 fell below half their short-input performance at just 32,000 tokens. GPT-4o dropped from 99.3% to 69.7%⁵.

The decline shows up even on simple tasks. In a July 2025 technical report, the vector database company Chroma tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and found that "model performance consistently degrades with increasing input length," even on simple tasks⁶. They called the effect context rot. Two of their findings are especially practical. First, adding even one distractor (a passage that looks relevant but isn't) reduced accuracy, and four distractors compounded the damage. Second, on a memory test built from long chat histories, every model scored significantly higher when given a focused prompt of about 300 tokens than when given the full history of about 113,000 tokens⁶. Chroma sells retrieval software, so it has a commercial interest in this conclusion. But its results line up with the independent academic work above.

Irrelevant information distracts. Freda Shi and colleagues found that adding irrelevant sentences to grade-school maths problems "dramatically decreased" model accuracy⁷. Telling the model to ignore irrelevant information was one of the mitigations they tested, but the simpler lesson is that removing noise beats asking the model to see past it.

Anthropic's engineers describe why this happens in terms of an "attention budget": every token added "depletes this budget by some amount," so context should be treated "as a finite resource with diminishing marginal returns"¹. Some models degrade more gently than others, but the effect "emerges across all models"¹.

The main techniques, in plain terms

If the problem is too much, poorly chosen information, the techniques fall into four families: choose, order, compress and isolate.

Choose: retrieve just in time. Instead of loading every document up front, the system keeps lightweight pointers (file names, saved searches, links) and pulls in content only when it's needed¹. When retrieval is used, quality matters. Anthropic reported in 2024 that adding a short explanatory note to each document chunk before indexing it, a technique it called contextual retrieval, cut the rate of failed retrievals by 49%, and by 67% when combined with a reranking step that re-sorts results by relevance⁸. Those figures come from the company's own tests, measured as how often the right passage was missing from the top 20 results.

Order: put what matters where the model looks. Given the position effects above, stable instructions belong at the start and the live goal near the end. The AI agent company Manus describes having its agent rewrite a running to-do list at each step, "reciting its objectives into the end of the context" so they don't get lost in the middle of a long task⁹.

Compress: summarize and prune as you go. Compaction means summarizing a conversation that's nearing the window limit and starting fresh with the summary¹. Structured note-taking means the agent writes notes to a file outside the window and reads them back when needed¹. Manus goes further and treats the file system as "the ultimate context": unlimited, persistent, and loaded only on demand⁹.

Isolate: give sub-tasks their own clean window. In a multi-agent setup, a lead agent hands focused jobs to sub-agents. Each sub-agent may read tens of thousands of tokens but returns a condensed summary, often 1,000 to 2,000 tokens¹. The lead agent's context stays clean.

Two practical details often get overlooked. Manus reports that its agents read roughly 100 tokens for every token they write, so input dominates costs⁹. And most providers now offer prompt caching, where a repeated opening section of the context is stored and reused rather than reprocessed. Anthropic advertises cost reductions of up to 90% and latency reductions of up to 85% for long prompts¹⁰. Manus cites cached input at a tenth of the price of uncached input⁹. That is why practitioners advise keeping the start of the context stable and adding to the end rather than rewriting it.

What the evidence shows so far

The case for context engineering rests on two kinds of evidence: strong, peer-reviewed evidence that long, noisy contexts hurt (the previous section), and newer, smaller studies showing that managing context helps. The second kind is promising but early.

A June 2026 enterprise study. Microsoft researchers tested an agent (GPT-5) on 50 expense-itemization tasks inside Dynamics 365 Finance and Operations, averaging five runs¹¹. Giving the agent its full history of tool calls produced complete, correct itemization 71.0% of the time, using about 1.48 million tokens and 14.6 hours of runtime. Keeping only the last five tool interactions raised success to 79.0%. Keeping the last five plus a compact running summary raised it to 91.6% while cutting tokens by about 63% and runtime by about 60%¹¹. Less context produced a better and cheaper agent. The authors are careful: this is "strong evidence for one class of enterprise tool-use workflow rather than a proof of universal generalization"¹¹.

Agents that manage their own memory. A January 2026 preprint tested an agent that decides for itself when to compress its history into a running "knowledge" block. On five hard software-engineering tasks it used 22.7% fewer tokens (14.9 million down to 11.5 million) with the same accuracy, three of five solved¹². Five tasks is far too few to generalize from, so treat this as a signal, not a result.

Contexts that learn. Qizheng Zhang and colleagues, in a paper accepted at ICLR 2026, built a system called ACE that treats an agent's instructions as an evolving "playbook" it updates after each task¹³. They report gains of 10.6% on agent benchmarks and 8.6% on finance tasks over strong baselines. They also name two failure modes worth knowing: brevity bias, where summaries drop useful domain detail for the sake of concision, and context collapse, "where iterative rewriting erodes details over time"¹³.

Limits and open questions

Compression can throw away what matters. Brevity bias and context collapse are the flip side of everything above¹³. A summary is only as good as its judgment about what to keep, and a fact dropped early in a long task can't be recovered later. Manus deliberately keeps failed attempts in the context so the model can learn not to repeat them⁹. That tension between lean and complete has no general answer yet.

Most of the evidence is narrow or self-reported. The strongest field result covers one workflow on one platform¹¹. Several key sources are engineering blogs or vendor reports¹⁶⁸⁹¹⁰, useful for practice but not independently replicated. Benchmarks such as RULER and NoLiMa measure retrieval of planted facts, which is cleaner than real work. And most of the models in these tests have since been replaced by newer versions, so specific numbers age quickly even when the pattern holds.

Longer windows won't make the problem disappear. Models keep accepting more tokens, but the research consistently finds that usable length lags advertised length⁴⁵⁶. A bigger desk doesn't help much if the reader still struggles with a cluttered one.

Understanding is ahead of output. The 2025 survey points to an asymmetry: with good context engineering, models are strong at understanding complex contexts but show "pronounced limitations" in generating equally sophisticated long-form outputs². Feeding a model the right material doesn't guarantee it can produce a long, complex deliverable from it. There is also no agreed benchmark for "good context" across tasks, so for now the best test is your own: measure accuracy and cost on your real cases, with and without each change.

What this means for you

You don't need to build agents for this to matter. The same principles apply to anyone pasting material into a chatbot or buying an AI tool.

1. Curate before you paste. Before adding a long document, ask which parts the model needs for this question. Paste those, and leave the rest out.
2. Put the key instruction and question last, after any long material, and keep the main point out of the middle.
3. Start fresh when a conversation drifts. If a long chat starts giving muddled answers, open a new one with a short summary of what's settled. That is manual compaction.
4. Ask vendors how their tool manages context. Good questions: what does the agent keep between steps, how does it summarize, and what happens to accuracy and cost on long tasks? A big context-window number is not an answer.
5. Measure with and without. If you run an AI workflow, try a trimmed context against the full one on 10 to 20 real cases. The June 2026 Microsoft study suggests you may get better accuracy and a lower bill¹¹.

Sources

1. Rajasekaran, P., Dixon, E., Ryan, C., & Hadfield, J. (Anthropic Applied AI team). (2025, September 29). Effective context engineering for AI agents. Anthropic Engineering. anthropic.com/engineering/effective-context-engineering-for-ai-agents
2. Mei, L., Yao, J., Ge, Y., et al. (2025). A survey of context engineering for large language models. arXiv:2507.13334. arxiv.org/abs/2507.13334
3. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. doi.org/10.1162/tacl_a_00638
4. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What's the real context size of your long-context language models? arXiv:2404.06654. arxiv.org/abs/2404.06654
5. Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). NoLiMa: Long-context evaluation beyond literal matching. Proceedings of ICML 2025. arXiv:2502.05167. arxiv.org/abs/2502.05167
6. Hong, K., Troynikov, A., & Huber, J. (2025, July 14). Context rot: How increasing input tokens impacts LLM performance. Chroma Technical Report. trychroma.com/research/context-rot
7. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023). Large language models can be easily distracted by irrelevant context. arXiv:2302.00093. arxiv.org/abs/2302.00093
8. Anthropic. (2024, September 19). Introducing contextual retrieval. anthropic.com/news/contextual-retrieval
9. Ji, Y. (2025, July 18). Context engineering for AI agents: Lessons from building Manus. Manus blog. manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus
10. Anthropic. (2024, updated 2025). Prompt caching. Claude resources. claude.com/resources/articles/prompt-caching
11. Lodha, A., Pahlavikhah Varnosfaderani, M., Chakraborty, A., & Mithal, A. (2026). Less context, better agents: Efficient context engineering for long-horizon tool-using LLM agents. arXiv:2606.10209. arxiv.org/abs/2606.10209
12. Verma, N. (2026). Active context compression: Autonomous memory management in LLM agents. arXiv:2601.07190. arxiv.org/abs/2601.07190
13. Zhang, Q., Hu, C., Upasani, S., et al. (2025, revised 2026). Agentic context engineering: Evolving contexts for self-improving language models. ICLR 2026. arXiv:2510.04618. arxiv.org/abs/2510.04618