BERT vs GPT: Encoders, Decoders, and When to Use Which

BERT and GPT are both Transformers. The difference is the masking. BERT is bidirectional — every token attends to every other token, including future ones. GPT is autoregressive — each token only attends to past tokens. This one difference creates two entirely different training objectives, capability profiles, and use cases. The architecture choice is downstream of what you want the model to do.

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al., 2018) and GPT (Generative Pretrained Transformer, Radford et al., 2018) are both stacked Transformer layers trained on large text corpora. The fundamental difference is in the attention mask.

The attention mask determines the training objective

BERT uses full (bidirectional) attention: every token attends to all other tokens in both directions. This means the model always has access to the full context when making a prediction. The training objective exploits this: Masked Language Model (MLM). Randomly mask 15% of tokens; predict them using the surrounding context. Because every position sees all other positions, BERT can use both left and right context to fill in the blank — it is doing cloze task learning.

GPT uses causal (left-to-right) masking: each token attends only to itself and previous tokens. Future tokens are hidden. The training objective is next-token prediction (autoregressive language modelling): given all previous tokens, predict the next one. This is the natural objective for generation — you generate left to right, one token at a time.

BERT: encoder model, understanding tasks

Because BERT sees full context, its representations encode deep semantic understanding. The [CLS] token representation at the end of BERT processing captures a summary of the whole sequence — it is used as the input to a classification head for tasks like sentiment classification, entailment, and question answering (where understanding the full passage before extracting an answer is necessary). BERT does not generate text naturally — autoregressive generation would require masking future tokens, at which point you lose the bidirectional advantage.

BERT-family models (RoBERTa, DeBERTa, ALBERT) are the standard choices for: classification, named entity recognition, sequence labelling, extractive QA, sentence similarity, and dense retrieval (see Post 61).

GPT: decoder model, generation tasks

Because GPT predicts each token from only left context, it is naturally a generative model. At inference, you feed a prompt and sample the next token; append it to the context; sample again. The model scales remarkably — GPT-3 showed that decoder-only language models trained at scale can do in-context learning (few-shot prompting) without any gradient updates. GPT-4, Claude, Gemini, and all modern chat LLMs are decoder-only.

GPT-family models (PaLM, LLaMA, Mistral) are the standard choices for: text generation, summarisation, translation, code completion, instruction following, and any task reformulated as text completion.

Encoder-decoder models: sequence-to-sequence

T5, BART, and the original Transformer (seq2seq for translation) use both an encoder and a decoder. The encoder processes the full input with bidirectional attention; the decoder generates the output autoregressively, attending to the encoder output via cross-attention. This architecture is natural for tasks with distinct input and output sequences: translation, summarisation, question generation, multi-document synthesis.

The scaling law implication

Decoder-only models (GPT architecture) have become dominant in the era of large models. Why? Pretraining data efficiency: next-token prediction uses every single token as a training signal. MLM uses only 15% of tokens per forward pass. At scale, the autoregressive objective is more data-efficient. In-context learning also emerges naturally from the autoregressive formulation — the model can "condition" on demonstrations by including them in the prompt context.

Practical decision guide

Use an encoder (BERT-family) when: you have labelled data for a specific task, you need sentence or token-level representations, and generation is not required. Use a decoder (GPT-family) when: you want generation, you want in-context learning without labelled data, or you are building a chat/instruction-following interface. Use encoder-decoder (T5-family) when: the task has distinct input and output sequences and you want the full bidirectional input encoding.

Try on Colab: fine-tune BERT-base on SST-2 (binary sentiment). Fine-tune GPT-2 (with a classification head on the last token) on the same dataset. Compare accuracy and training speed. Then try zero-shot GPT-2 prompting ("This movie was [MASK]") — observe the gap between fine-tuned accuracy and zero-shot. The fine-tuned BERT should win; the gap illustrates why task-specific fine-tuning outperforms zero-shot on small, well-defined tasks.