An encoder-decoder transformer is the original kind of transformer, built to turn one sequence into another, such as a sentence into its translation.
It has two stacks of blocks:
- Encoder: Reads the whole input. Each block has self-attention, then a feed-forward layer
- Decoder: Writes the output token by token. Each block adds cross-attention to the encoder's output between those two¹
Encoder-decoder Translation Process
The encoder reads all of I love you at once, not word by word.
The decoder starts from [start] and writes one token per round:
| Round | The decoder reads | Next token | Cross-attention looks most at |
|---|---|---|---|
| 1 | [start] |
我 | I |
| 2 | [start] 我 |
爱 | love |
| 3 | [start] 我 爱 |
你 | you |
| 4 | [start] 我 爱 你 |
[end] |
The decoder appends each new token to its input. The next round reads it. This loop is Autoregressive generation.
[end] stops the loop: The decoder appends nothing more.
How the two stacks connect
The figure goes through seven steps:
- Step 1: The encoder reads
I love you. Each layer runs self-attention, then the MLP. - Step 2: Only the top encoder layer's output goes on. The outputs of layers 1 and 2 never reach the decoder.
- Step 3: That one output goes to the cross-attention of every decoder layer.
- Steps 4 to 7: The decoder predicts one token per step. Each layer runs masked self-attention, cross-attention, then the MLP. The decoder appends each token to its input.
The encoder runs once. The decoder runs once per token. The encoder's output stays the same.
All three arrows carry the same output. Each decoder layer reads it with its own weights.
The dashed line beside each sublayer is a residual connection. It adds the sublayer's input back to its output (the +), then LayerNorm runs.
A real model has 6 layers in each stack. The figure draws 3 to stay small.
¹ The decoder generates the output one symbol at a time, auto-regressively, and each decoder layer adds a sub-layer of attention over the encoder stack's output: Vaswani et al. (2017), Attention Is All You Need, §3 and §3.1