How LLMs Actually Work: Tokens, Attention, and Inference

An LLM (large language model) is a transformer-based neural network trained to predict the most probable next token given a sequence of input tokens — it is not a database, a search engine, or a reasoning system. The model learns statistical relationships between tokens by processing hundreds of billions of words from web pages, books, code, and other text during training.

When you type a prompt, the model converts your text into a sequence of numeric token IDs, passes them through dozens of transformer layers, and outputs a probability distribution over its entire vocabulary (typically 50,000–100,000 tokens). It samples one token from that distribution, appends it to the sequence, and repeats until it generates a stop token or hits the output limit.

This architecture explains several behaviors that confuse users: why LLMs "hallucinate" plausible-but-false facts (they predict probable text, not verified truth), why they can fail on arithmetic (token patterns, not calculation), and why rephrasing a prompt changes the output (different token sequences trigger different probability distributions).

**Before an LLM can process any text, it must convert it into a sequence of integer token IDs — a process called tokenization.** GPT-4o uses Byte Pair Encoding (BPE), which breaks text into frequently occurring sub-word units. Claude Opus 4.7 and Gemini 3.1 Pro use similar sub-word tokenization schemes.

Tokenization is language-dependent. English text averages approximately 1 token per 0.75 words. Chinese and Japanese average 1 token per 0.5 words — meaning the same document costs roughly twice as many tokens in Chinese as in English, which directly affects API cost and context window usage.

The transformer architecture uses a mechanism called self-attention to determine how much each token should "pay attention" to every other token in the sequence when computing its representation. For each token, the model computes three vectors — Query (Q), Key (K), and Value (V) — and calculates attention scores as dot products between Q and K, scaled and normalized with softmax.

Multi-head attention runs this process in parallel across multiple "heads" (GPT-4o uses 96 attention heads in its largest layers), each learning different relationship patterns. Some heads specialize in syntactic relationships (subject-verb), others in semantic similarity, others in coreference (matching pronouns to nouns).

A key practical implication: the "lost in the middle" effect. Research from Liu et al. (2023) at Stanford shows that LLMs systematically underweight information in the middle of long contexts. For prompts exceeding ~2,000 tokens, place critical instructions in the system prompt (beginning) and repeat the most important constraint at the end of the user message.

LLM training happens in two distinct phases: pretraining (learning language patterns from raw text) and post-training alignment (shaping behavior with human feedback). These phases produce different capabilities and explain why models from different labs behave differently even with similar benchmark scores.

During pretraining, the model processes a massive corpus — Llama 3.1 was trained on approximately 15 trillion tokens; GPT-4 on an estimated 1–2 trillion tokens. The objective is simple: predict the next token. No explicit knowledge is stored; all information is encoded in the model weights as statistical patterns.

Post-training alignment — typically Reinforcement Learning from Human Feedback (RLHF) or its variants (RLAIF, DPO) — shapes the model into a useful assistant. Human raters score outputs on helpfulness, harmlessness, and honesty. A reward model is trained on these ratings, and the base LLM is fine-tuned to maximize reward. RLHF determines refusal behavior, tone, and safety guardrails — not the base architecture.

During inference, the model generates output one token at a time — computing a probability distribution over the entire vocabulary and sampling from it according to decoding parameters you control. The three most important parameters are temperature, top-p (nucleus sampling), and max tokens.

**The context window is the maximum number of tokens the model can process in a single inference call — combining the system prompt, conversation history, documents, and the current user message.** Nothing persists between sessions; the model starts fresh every time.

Context window size varies significantly by model and directly affects which use cases are practical:

Understanding LLM architecture directly improves prompt quality — token position, temperature, context window usage, and output length all have measurable effects on output reliability.

These misconceptions about LLMs frequently cause poorly-designed prompts and misplaced expectations:

Tested in PromptQuorum — sending the same creative brief to GPT-4o, Claude Opus 4.7, and Gemini 3.1 Pro at temperature 0 vs. temperature 0.9 showed that Claude Opus 4.7 has the smallest output variance between temperatures, while Gemini 3.1 Pro shows the highest variance. At temperature 0.9, Gemini 3.1 Pro produced outputs 34% longer on average than at temperature 0.

Using PromptQuorum's multi-model dispatch, you can run any prompt simultaneously against all available models at a specified temperature and compare outputs side-by-side — making it practical to calibrate temperature settings for your specific task rather than relying on model defaults.

LLM architecture and performance vary significantly by training data composition, tokenization strategy, and regulatory constraints across regions. Understanding these differences is critical for teams deploying models globally.

Qwen 3 achieves superior tokenization efficiency for CJK (Chinese, Japanese, Korean) scripts** — using approximately 0.3 tokens per character for Mandarin Chinese compared to GPT-4o's 0.5 tokens per character. This 40% reduction in token count directly reduces API costs and latency for Asian language applications. Qwen's training data includes 20% CJK content, optimizing the tokenizer for scripts where character-to-semantic density is highest.

Mistral 7B and Mistral Large are explicitly architected for European deployment, with training data filtered to comply with GDPR, France's AI Act, and EU regulations on data retention and model transparency. Unlike models trained primarily on unfiltered web data, Mistral documents data sourcing and excludes EU citizens' personal information from training, making it the default choice for regulated industries in Europe (banking, healthcare, legal tech).

DeepSeek's architecture reflects its training composition: 70% of pretraining data is in Chinese and English, 15% in code, 15% in other languages. This ratio produces a model biased toward Chinese language fluency and code-generation speed, with substantially lower performance on low-resource languages. The token distribution and attention patterns are optimized for the frequency patterns in Mandarin Chinese, not English.