Transformer Architecture Explained

Why transformers exist

Before 2017, sequence modeling used recurrent neural networks (RNNs, LSTMs). They processed text one token at a time, maintaining a hidden state. Two big problems:

1. Slow to train — sequential processing can't use GPU parallelism efficiently.
2. Forgetful — information from early in a long sequence got compressed and lost by the end.

The 2017 paper "Attention Is All You Need" (Vaswani et al., Google Brain) proposed a different architecture: process all tokens in parallel, and let each token attend to every other token directly via self-attention.

The result: massively faster training + better long-range dependency modeling. Within five years it was the architecture for every frontier language model and most vision models.

The core idea: self-attention

For each token in the input sequence, compute three vectors:

• Q (Query) — "what am I looking for?"
• K (Key) — "what do I have to offer?"
• V (Value) — "what information would I contribute?"

Then for each token, compute attention scores: how much should this token attend to every other token? Multiply: softmax(QK^T / √d) · V. The output is a weighted sum of value vectors, where weights are the attention scores.

In plain English: each token gets a custom-blended view of the whole sequence, weighted by relevance to itself.

A worked example

Input: "The cat sat on the mat. It was tired."

When processing the token "it", self-attention asks: which earlier token does "it" most refer to?

Hypothetical attention weights from "it":

• "cat" → 0.75 (heavy attention — likely referent)
• "mat" → 0.20 (moderate — also a noun)
• "sat" → 0.03
• Others → ~0

The model uses these weights to construct a context-aware representation of "it" that's mostly informed by "cat". That's how pronoun resolution works in a transformer — no explicit rule, just learned attention patterns.

Architecture overview

A standard transformer block:

Input tokens (after embedding + positional encoding)
       ↓
[Multi-Head Self-Attention]
       ↓
+ Residual connection + LayerNorm
       ↓
[Feed-Forward Network]
       ↓
+ Residual connection + LayerNorm
       ↓
Output (input to next block)

Stack 12, 32, 96, or more of these blocks and you have a transformer. The original 2017 paper used 6; modern frontier models use 100+.

Multi-head attention

Instead of one attention computation per layer, the transformer runs N (typically 8–96) attention "heads" in parallel, each learning to attend to different patterns. Multi-head attention is concatenated and projected — much richer representation than single-head.

Positional encoding

Self-attention has no built-in notion of order — it treats the input as a set. Positional encoding adds position information to each token's embedding so the model knows token order. Two main approaches:

• Sinusoidal (original transformer) — fixed sinusoids of varying frequency
• Rotary Position Embeddings (RoPE) (used in most modern LLMs) — encodes position via rotation matrices

RoPE generalizes better to longer sequences than the model was trained on — one reason context windows have grown to 1M+ tokens.

The three transformer families

Decoder-only dominates 2026 because it scales beautifully, can do all tasks via autoregressive generation, and is what you want for chat/agents.

The O(n²) problem

Self-attention requires computing scores between every pair of tokens. For a sequence of length n:

• Compute cost: O(n²)
• Memory cost: O(n²) for the attention matrix

This is why long-context costs more than you'd expect:

• 1K tokens → 1M attention pairs
• 10K tokens → 100M pairs (100× cost)
• 100K tokens → 10B pairs (10,000× cost)
• 1M tokens → 1 trillion pairs (1,000,000× cost vs 1K)

Modern optimizations attack this constant:

• Flash Attention — reorder compute for GPU memory hierarchy. 2-4× faster, far less memory. Now standard.
• Sparse attention — only attend to a subset (recent tokens, strided patterns)
• Linear attention — approximate attention in O(n)
• Grouped-query attention (GQA) — share K/V across attention heads to reduce memory

None of these eliminate the O(n²) fundamental scaling. They make the constant factor smaller.

Mixture of Experts (MoE)

Modern frontier models (GPT-4 family, DeepSeek V3, Mixtral) use MoE: replace the dense feed-forward layer with many smaller "expert" layers + a router that activates only the top-k experts per token.

Result: a model can have trillions of total parameters while activating only ~10-50B per forward pass. Dramatically better quality-per-compute than equivalent dense models.

The trade-off: harder to train, harder to serve efficiently. The infrastructure to run MoE models at scale is non-trivial.

What transformer mechanics tell you about LLMs

Once you understand the architecture, several mysterious LLM behaviors become obvious:

• Position bias — attention weights and positional encoding mean models attend strongest to the start and end of context. Put critical instructions there.
• Context window limits — O(n²) cost makes serving long contexts expensive. Pricing reflects compute, not arbitrary limits.
• Prompt sensitivity — every token participates in attention. Small wording changes shift attention patterns.
• Repetition / loops in generation — autoregressive generation with low temperature can fall into high-probability cycles.
• Hallucination — sampling from a probability distribution will produce plausible-but-wrong outputs when the distribution has no strong signal.

This is the value of architecture knowledge: it makes the model less of a black box.

Where to read more

• The original paper: "Attention Is All You Need" — Vaswani et al., 2017. Surprisingly readable.
• The Illustrated Transformer (Jay Alammar) — best free visual primer.
• The Annotated Transformer (Harvard NLP) — paper walked through with code.
• Karpathy's "Let's build GPT" — implement a small transformer in a YouTube video.

You don't need to be able to derive backprop through attention from scratch. You do need to be able to look at a 1M-token context window pricing and know exactly why it costs what it costs.