Context Engineering Explained

The discipline that makes AI products work

Anyone can call an LLM. The difference between a demo and a product is what surrounds the call:

• What did you retrieve from your data?
• What conversation history did you keep?
• What tool results did you include?
• How did you compress old turns?
• How did you handle the case where the user's question references something from 20 turns ago?

This is context engineering. It's where most AI startup engineering effort actually goes.

The composition of context

Every call has roughly five sources:

1. System prompt — stable instructions, role, format constraints.
2. Retrieved context — RAG output, documentation snippets, code files.
3. Conversation history — prior user/assistant turns.
4. Tool results — outputs from previous tool calls.
5. User input — the current message.

Each one competes for limited attention. Engineering the mix is the job.

The position effect

Models attend most strongly to the beginning and end of context. Information buried in the middle gets weighted less and forgotten more. Implications:

• Put critical instructions at the start of the system prompt.
• Re-state the user's question at the end of long contexts.
• For long documents, consider question-then-document or document-then-question, not document-in-the-middle.

Quality over quantity

Models with 1M-token windows still work best with focused contexts. Past ~50K tokens:

• Recall accuracy drops (the "needle in a haystack" problem).
• Latency climbs linearly.
• Cost climbs linearly.
• Reasoning degrades on multi-step tasks.

The right move past 30K tokens is usually retrieval, not stuffing.

Memory, the long-running concern

Persistent memory turns a chatbot into an assistant. The basic pattern:

1. After a conversation: extract durable facts (preferences, decisions, recurring entities).
2. Store them as embeddings + metadata.
3. On each new turn: retrieve relevant memories alongside other context.

The hard part is what to extract — too much and context bloats; too little and the memory feels useless.

Compression and summarization

For long-running sessions or agent loops, compress old context:

• Sliding window: drop oldest turns (simple, loses information).
• Hierarchical summary: summarize old turns into a paragraph; keep the recent ones full (most common).
• Selective retrieval: index everything, retrieve only what's relevant to the current turn.

The production-grade pattern

Modern AI products typically combine:

• A stable, prompt-cached system prompt.
• Retrieved chunks (RAG).
• Last N conversation turns (full).
• A summary of older turns.
• Recent tool results.
• The user's current message at the end.

That stack, well tuned, is what makes ChatGPT, Claude, and copilots feel coherent. None of them are "just" prompt engineering.

The mental shift

If you treat the model as a black box you prompt once, you'll hit a ceiling fast. If you treat the model as one node in a context-orchestration system, you'll keep finding leverage. Context engineering is where the real product engineering happens.