---
title: "Transformer architecture explained"
excerpt: "Attention, positional encoding, and stacked layers: how the transformer works and why it scales, for engineers who build on top of LLMs."
date: "2026-07-09"
lastModified: "2026-07-09"
summary: "The transformer replaced recurrence with attention, which is what let language models scale. This explainer covers self-attention, multi-head attention, positional encoding, and the stacked-layer structure — and explains why the architecture parallelizes so well that scaling it became the whole game."
keyTakeaways:
  - "Self-attention lets every token look at every other token in one parallel step, replacing the sequential bottleneck of recurrent networks."
  - "Because attention is parallel over the sequence, transformers scale efficiently on modern hardware — that scalability is why they won."
  - "Positional encoding is what gives an order-blind attention mechanism a sense of sequence; without it, a transformer sees a bag of tokens."
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["Transformers", "LLM Architecture", "AI Engineering", "Deep Learning"]
keywords:
  - "transformer architecture explained"
  - "self attention mechanism"
  - "multi head attention"
  - "positional encoding"
  - "how transformers work"
  - "why transformers scale"
---

Every large language model you use — GPT, Claude, Gemini, Llama — is a transformer. The architecture, introduced in the 2017 paper "Attention Is All You Need," is the reason the last several years of AI happened. If you build systems on top of LLMs, you don't need to derive the math, but you do need a real mental model of what's inside, because it explains the things you hit every day: why prompts have a token limit, why long contexts are expensive, why the model attends to some parts of your input and ignores others. This explainer builds that model for practicing engineers.

## The problem the transformer solved

Before transformers, sequence models were recurrent (RNNs, LSTMs). They processed text one token at a time, carrying a hidden state forward: read a word, update the state, read the next, update again. This has two fatal problems at scale. First, it's **sequential** — you can't process token 100 until you've processed tokens 1 through 99 — so it can't exploit the massive parallelism of modern GPUs. Second, information from early tokens has to survive a long chain of updates to influence later ones, and it tends to fade; long-range dependencies get lost.

The transformer's move was to throw out recurrence entirely and let every token look at every other token *directly*, in a single parallel operation. That operation is attention. Removing the sequential dependency is what made training on internet-scale data feasible, and direct token-to-token connections is what let models capture long-range structure. Almost everything good about LLMs traces back to this one architectural decision.

## Self-attention: the core mechanism

Attention answers a question for each token: *which other tokens should I pay attention to, and how much?* Consider "The animal didn't cross the street because it was too tired." To represent "it" correctly, the model needs to connect it to "animal," not "street." Self-attention is the mechanism that lets "it" look across the sentence and pull in information from "animal."

The mechanics use three learned projections of each token's vector, with deliberately chosen names:

- **Query (Q):** what this token is looking for.
- **Key (K):** what each token offers, for matching against queries.
- **Value (V):** the actual information a token contributes if attended to.

The database analogy is exact and worth internalizing. Each token issues a query and compares it against every token's key. The comparison is a dot product: query · key is high when they align, low when they don't. Those scores are scaled and passed through a softmax, turning them into weights that sum to one — a probability distribution over "how much attention this token pays to each other token." The output for the token is the weighted sum of all the value vectors, weighted by those attention scores. A token's new representation is a blend of the values of everything it decided to attend to.

Two consequences fall out immediately. First, this is a big matrix multiply — Q, K, and V are matrices over the whole sequence, and the operation runs for all tokens at once. That's the parallelism recurrence lacked. Second, the attention-score matrix is sequence-length × sequence-length: every token scores against every other token. That **quadratic cost in sequence length** is exactly why long contexts are expensive and why [context windows](/glossary/context-window) have limits. When you wonder why doubling your prompt more than doubles the cost, this matrix is the reason.

## Multi-head attention

One attention operation forces the model to blend all kinds of relationships — syntactic, semantic, coreference — into a single set of scores. That's a bottleneck. **Multi-head attention** runs several attention operations in parallel, each with its own learned Q/K/V projections, so each "head" can specialize. One head might track subject-verb agreement, another might resolve pronouns, another might follow topical relatedness. The heads' outputs are concatenated and projected back down. Empirically, different heads learn recognizably different jobs. Multi-head attention is why a single layer can capture multiple types of relationship at once instead of averaging them into mush.

## Positional encoding: giving attention a sense of order

Here's a subtlety with big consequences: **attention is order-blind.** The weighted sum over values doesn't depend on token order — shuffle the input and the raw attention math produces the same blend. To a bare attention mechanism, "dog bites man" and "man bites dog" are identical. That's obviously wrong for language, where order is meaning.

**Positional encoding** fixes it by injecting position information into each token's representation before attention runs. The original transformer added fixed sinusoidal patterns — waves of different frequencies — to the token embeddings, so each position gets a distinct signature the model can learn to read. Modern LLMs mostly use learned or rotary positional encodings (RoPE), which encode *relative* position and generalize better to long sequences, but the purpose is identical: without it, the transformer sees an unordered bag of tokens. Positional encoding is also why extending a model's context window is non-trivial — the positional scheme has to remain sensible at lengths it wasn't trained on, which is an active area of engineering.

## The full stack: layers, residuals, and feed-forward networks

Attention is the star, but a transformer layer has more. Each layer contains an attention sub-layer *and* a position-wise feed-forward network (a small MLP applied to each token independently). Attention mixes information *across* tokens; the feed-forward network transforms each token's representation *in place*, adding non-linear processing capacity. A common intuition: attention decides *what to combine*, and the feed-forward network *thinks about* the combined result.

Two structural pieces make deep stacks trainable. **Residual connections** add each sub-layer's input to its output, giving gradients a clean path back through dozens or hundreds of layers and preventing the signal from degrading with depth. **Layer normalization** keeps activations in a stable range so training doesn't diverge. These aren't glamorous, but without them you cannot stack the model deep enough to be useful.

You stack this layer — attention, feed-forward, residuals, normalization — many times. Early layers tend to capture local, surface patterns; deeper layers build up abstract, semantic representations. Depth is where the model's sophistication comes from. A decoder-only LLM (the standard for text generation) adds one more twist: **causal masking**, which blocks each token from attending to future tokens during training, so the model learns to predict the next token from only what came before — exactly the setup it uses at generation time.

## Encoder, decoder, and why LLMs are decoder-only

The original transformer had two halves: an *encoder* that reads an input sequence into rich representations, and a *decoder* that generates an output sequence while attending to the encoder's output. That design fit translation, its first application — encode the French, decode the English. Three families descend from it, and knowing which one a model is tells you what it's good for.

**Encoder-only** models (BERT and its relatives) keep just the reading half. Every token attends to every other token in both directions, so the model builds a deep bidirectional understanding of a fixed input — ideal for classification, and for producing the [embeddings](/research/how-embeddings-work) that power retrieval. They don't generate text. **Decoder-only** models (GPT, Claude, Llama, Gemini) keep the generating half with causal masking, predicting the next token from left to right. That single objective — next-token prediction at scale — turns out to be a remarkably general way to learn language, reasoning, and knowledge, which is why every modern chat LLM is decoder-only. **Encoder-decoder** models (T5, the original design) keep both and suit tasks with a clear input-to-output mapping.

The lesson for engineers: when you pick an embedding model versus a generation model, you're often choosing between an encoder-style architecture optimized for understanding and a decoder-style one optimized for generation. They're built from the same attention parts arranged for different jobs, and using one where the other belongs — a chat model's raw hidden states as [embeddings](/glossary/embeddings), say — leaves quality on the table. The same decoder-only weights are also what you adapt when you [fine-tune](/glossary/fine-tuning) a model — you're not changing the architecture, just nudging the parameters that live inside these layers.

## Why it scales — and why that mattered

The deepest reason the transformer won isn't that any single component is magical. It's that the architecture **parallelizes**. Because attention processes the whole sequence at once instead of stepping through it, transformers map beautifully onto GPU and TPU hardware, and their performance improves predictably as you add parameters, data, and compute — the empirical scaling laws. Recurrent models couldn't ride hardware improvements the same way; each speedup was throttled by their sequential nature.

That scalability turned model-building into an engineering-and-capital problem instead of an architecture-search problem. Once you have a design that reliably gets better with more scale and doesn't fight your hardware, the path forward is to scale it — and that's largely what the field did. The transformer's real contribution was giving researchers something *worth* scaling.

The catch, and it's the one that shapes your daily engineering, is the quadratic attention cost. Longer context means quadratically more attention computation and a linearly larger [KV cache](/research/llm-inference-optimization) at inference — the two together are why long-context serving is expensive and why a great deal of systems work (efficient attention kernels, paging, sparse and linear attention variants) goes into making long contexts affordable. If you serve these models, the architecture here connects directly to [LLM inference optimization](/research/llm-inference-optimization); if you build retrieval systems, note that the [embeddings](/research/how-embeddings-work) you rely on come from the same attention machinery, pooled into a single vector. Understanding the transformer isn't academic — it's the map to why the systems you build behave the way they do.