What is an LLM?

The one-sentence version

A Large Language Model (LLM) is a neural network trained to predict the next token in a sequence of text — and that single capability, scaled up, turns out to be powerful enough to write code, summarize PDFs, and answer questions.

Tokens, not words

LLMs don't read words. They read tokens — short fragments of text. The word "unbelievable" might be split into three tokens like ["un", "believ", "able"]. Tokens are the unit of input and output.

When you send a prompt, the model:

1. Splits your text into tokens.
2. Runs those tokens through a stack of transformer layers.
3. Outputs a probability distribution over every possible next token.
4. Samples one token from that distribution and adds it to the output.
5. Repeats until it hits a stop signal.

That loop — token in, token out — is the entire trick.

Why it works

The model is trained on trillions of tokens scraped from the public internet, books, and code. To predict the next token well, it has to learn:

• Grammar (so the sentence sounds right)
• Facts (so the content is plausible)
• Reasoning patterns (so the logic holds up)
• Style (so the tone matches)

It's not memorizing — it's compressing patterns. The same model that completes "The capital of France is" with "Paris" also completes "def quicksort(arr):" with a working implementation.

Context windows

The number of tokens an LLM can consider at once is its context window. Modern models range from 8K tokens (older GPT-3.5) to 1M+ tokens (Claude, Gemini). Bigger windows mean you can fit more of a codebase, a book, or a transcript into a single prompt.

One nuance: bigger context isn't free. Quality of recall often degrades past ~100K tokens. For long documents, retrieval (RAG) often beats stuffing everything into context.

Temperature, top-p, and sampling

Generation isn't deterministic. After the model produces a probability distribution over the next token, it samples one. The shape of the sampling is controlled by:

• Temperature — higher = more random. 0 = always pick the highest-probability token.
• Top-p (nucleus) — only sample from the smallest set of tokens whose cumulative probability passes p.
• Top-k — only sample from the top k candidates.

For coding and factual tasks, lower temperature (0–0.3). For brainstorming and creative work, higher (0.7–1.0).

What LLMs are bad at

• Math beyond rough estimates — they predict tokens, they don't calculate. Even modern reasoning models lean on tools for serious math.
• Knowing what they don't know — they will confidently hallucinate.
• Anything past their training cutoff unless you give them tools or retrieval.
• Counting — try asking "how many R's are in 'strawberry'" on a model with no tool use.

What's next

LLMs alone are useful, but the real leverage comes when you combine them with retrieval (RAG), tools (function calling), and memory. Those layers are what turn a clever autocomplete into an actual assistant.