ModelRefs / BERT & Encoder Models — Tutorial

BERT & Encoder Models — Tutorial

Masked language modelling, the [CLS] token, and how to fine-tune BERT for any NLP task. Covers BERT's pretraining tasks, Fine-tuning BERT for downstream tasks.

Overview

Masked language modelling, the [CLS] token, and how to fine-tune BERT for any NLP task

Level: Advanced. Estimated reading time: 35 minutes.

BERT's pretraining tasks

BERT (Bidirectional Encoder Representations from Transformers, Google 2018) is pretrained on two tasks simultaneously:

Masked Language Model (MLM): randomly mask 15% of input tokens, predict the original tokens from the context on both sides. "The cat [MASK] on the mat" → predict "sat". This forces the model to learn bidirectional context — unlike GPT which only uses left context.

Next Sentence Prediction (NSP): given two sentences, predict whether the second follows the first in the original text. This teaches inter-sentence relationships useful for tasks like question answering. (Later work showed NSP is largely unnecessary — RoBERTa dropped it and improved performance.)

BERT-base: 12 layers, 768 d_model, 12 heads, 110M parameters. BERT-large: 24 layers, 1024 d_model, 16 heads, 340M parameters. Both were pretrained on BooksCorpus + English Wikipedia (~3.3B words).

Fine-tuning BERT for downstream tasks

The BERT fine-tuning recipe is remarkably consistent across tasks:

Sequence classification (sentiment, topic): take the [CLS] token's representation from the final layer, add a linear head: Linear(768, num_classes), fine-tune the whole model.

Token classification (NER, POS tagging): take all token representations, add a linear head per token: Linear(768, num_entity_types).

Question answering (SQuAD): add two linear heads that predict the start and end positions of the answer span.

Fine-tuning hyperparameters: lr = 2e-5 or 3e-5, batch = 16 or 32, epochs = 3–5. These work for most tasks. More epochs risks overfitting; higher learning rates cause catastrophic forgetting.

Hugging Face transformers makes this plug-and-play: AutoModelForSequenceClassification, Trainer, TrainingArguments.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BERT & Encoder Models — Tutorial.