Fine-tuning DistilBERT for 27-Class Emotion Classification on GoEmotions

Classifying text into one of 27 fine-grained emotion categories — distinguishing "admiration" from "approval", or "grief" from "sadness" — is one of the harder NLP tasks. Here's how I fine-tuned DistilBERT on Google's GoEmotions dataset and what I learned about multi-class NLP at scale.

The Dataset: GoEmotions

GoEmotions is a dataset released by Google Research containing 58,009 Reddit comments labeled with 27 emotion categories (plus a "neutral" class). What makes it hard:

• Comments are multi-label — a single comment can have multiple emotions.
• The class distribution is heavily skewed — "admiration" has 10x more samples than "grief".
• Emotion boundaries are subtle — "curiosity" vs. "confusion", "excitement" vs. "joy".

Why DistilBERT?

DistilBERT is a distilled (compressed) version of BERT — 40% smaller, 60% faster, and 97% as accurate on GLUE benchmarks. For a classification task like this, the speed advantage matters during training and inference without sacrificing meaningful accuracy. I also experimented with BERT-base, but the training time was 2.5x longer with only marginal gains.

Tokenization

I used Hugging Face's AutoTokenizer with max_length=128 and truncation=True. Reddit comments are typically short (under 50 tokens), so 128 handled 99% of samples without truncation. Setting it higher wastes memory with no benefit.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tokenizer(
        batch["text"],
        padding="max_length",
        truncation=True,
        max_length=128,
    )

dataset = dataset.map(tokenize, batched=True)

Handling Class Imbalance

The most impactful change I made to baseline performance was addressing class imbalance. I computed class weights inversely proportional to class frequency and passed them to the loss function:

from sklearn.utils.class_weight import compute_class_weight
import torch

class_weights = compute_class_weight(
    class_weight="balanced",
    classes=np.unique(train_labels),
    y=train_labels
)
weights = torch.tensor(class_weights, dtype=torch.float).to(device)
loss_fn = torch.nn.CrossEntropyLoss(weight=weights)

Without weighting, the model learned to predict "admiration" and "approval" for nearly everything (high frequency = easy accuracy). With weighting, minority classes like "grief", "nervousness", and "remorse" improved significantly in F1.

Fine-tuning Setup

I added a classification head on top of DistilBERT's [CLS] token output:

from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=27,
)

# Training config
optimizer = AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=len(train_loader) // 2,
    num_training_steps=len(train_loader) * NUM_EPOCHS,
)

Key hyperparameters: learning rate 2e-5 (standard for BERT fine-tuning), 3 epochs (more caused overfitting), batch size 32, linear warmup over the first half-epoch.

Results

The fine-tuned DistilBERT achieved 88% accuracy on the test set with a macro F1 of 0.71. The LSTM baseline (trained from scratch with GloVe embeddings) reached only 62% accuracy — demonstrating the power of transfer learning for this type of task.

Key Takeaways

• Transfer learning dominates. Fine-tuning DistilBERT for 3 epochs beat an LSTM trained for 20+ epochs on the same data.
• Class weighting is non-negotiable for imbalanced multi-class problems. Don't skip it.
• Warmup scheduling matters. Without it, the model diverged early due to the small fine-tuning learning rate conflicting with large initial gradients.
• Confusion matrix reveals what accuracy hides. 88% accuracy looked great until I saw the model confusing "disappointment" with "sadness" in 30% of cases — both semantically adjacent emotions.