How AI Models Are Trained

Stage 1 — Pre-training

The model starts as a randomly initialized neural network. You feed it trillions of tokens of text — books, web pages, code, papers — and at every step, train it to predict the next token.

This stage takes:

• Tens of thousands of GPUs for weeks to months.
• $50M–$500M+ in compute for frontier models.
• 5T–15T tokens of curated training data.

Output: a "base model" that can complete text but doesn't follow instructions. It will continue "The capital of France is" with "Paris" but can't yet handle "What is the capital of France?" properly. It just predicts probable text.

Stage 2 — Instruction tuning (SFT)

You take the base model and fine-tune it on (instruction, response) pairs:

{
  "instruction": "What is the capital of France?",
  "response": "The capital of France is Paris."
}

Tens of thousands of these pairs — high quality, diverse — teach the model to:

• Follow instructions
• Adopt a helpful assistant persona
• Output in expected formats

This stage is fast (hours to days) and cheap ($1K–$100K). It's where most behavior comes from.

Stage 3 — Preference optimization (RLHF / DPO)

Now the model can answer, but its answers might be unhelpful, verbose, or unsafe. The fix: show it pairs of responses, with humans labeling which one is better. The model learns to prefer the better ones.

Two approaches:

• RLHF (Reinforcement Learning from Human Feedback): train a separate reward model on preferences, then use RL (PPO, GRPO) to optimize the LLM against the reward.
• DPO (Direct Preference Optimization): skip the reward model — train the LLM directly on preference pairs.

DPO is increasingly the default — simpler, more stable, often equivalent quality.

This stage is where:

• Tone gets refined
• Safety policies get baked in
• The model learns to refuse certain requests
• Helpfulness gets calibrated

Stage 4 — Reasoning training (modern addition)

In 2024–2026, the frontier shifted: models are now trained to generate long internal chain-of-thought before answering. Examples: OpenAI's o-series, Anthropic's extended thinking, Gemini Thinking.

This stage:

• Uses RL on tasks with verifiable answers (math, code, formal logic)
• Rewards the model for reaching correct answers via reasoning steps
• Produces models that are slower but smarter on hard problems

This is one reason the gap between frontier and "good open-weights" widened in 2025.

Stage 5 — Eval, red-team, deploy

Before release:

• Eval: thousands of benchmarks measure capability across domains.
• Red-teaming: humans try to break the model — jailbreaks, harmful content, hallucinations.
• Safety tuning: targeted SFT/DPO to fix discovered issues.
• Deploy: ship as a hosted API or open-weights release.

After deployment, the model is frozen. New versions involve some or all of the stages above being repeated.

Where you fit in

You won't pre-train a frontier model. You probably won't do RLHF either. What you might do:

• Use the model — most common.
• Prompt engineer — tune the inputs.
• Context engineer — design the system around the model.
• Fine-tune — extend stage 2 with your own data on top of an existing model.

Knowing the full pipeline makes the boundaries of what's possible much clearer.