Fine-Tuning

What Is Fine-Tuning?

Fine-tuning means taking a pretrained neural network and continuing its training on a new dataset.

Unlike basic feature extraction, we do not keep every pretrained layer frozen. We unfreeze some layers and allow their weights to change.

Pretrained Model
        ↓
Keep Most Knowledge
        ↓
Unfreeze Some Layers
        ↓
Train With New Dataset
        ↓
Adapt Model
        ↓
New Task

The goal is not to completely retrain the model. The goal is to make small adjustments to the existing knowledge.

Why Do We Need Fine-Tuning?

A pretrained model may understand general features very well, but your new problem may have different visual patterns.

Fine-tuning allows the model to adapt those existing features to your specific dataset.

Pretrained Model

General Knowledge
      ↓
Edges
Shapes
Textures
Objects
      ↓
Fine-Tuning
      ↓
Task-Specific Knowledge

For example, a model trained on millions of general images may recognize common objects, but a medical image dataset may contain very different patterns.

Feature Extraction vs Fine-Tuning

This is the most important difference between the two techniques.

FEATURE EXTRACTION

Pretrained Layers
      ↓
    FROZEN
      ↓
Extract Features
      ↓
New Classifier
      ↓
Training


FINE-TUNING

Pretrained Layers
      ↓
Some Layers UNFROZEN
      ↓
Adapt Existing Features
      ↓
New Classifier
      ↓
Training

In feature extraction, the pretrained layers do not change. In fine-tuning, selected pretrained layers are allowed to learn from the new dataset.

Simple Example — Cats and Dogs

Imagine we have a pretrained image model that already understands general visual features.

We want to make it very good at distinguishing between different breeds of cats and dogs.

Pretrained Model
      ↓
General Image Features
      ↓
Unfreeze Some Layers
      ↓
Train With Cat & Dog Dataset
      ↓
Model Learns Breed-Specific Features
      ↓
Prediction

The model does not need to relearn basic edges and shapes from zero. Instead, it adjusts deeper features to become more useful for the new task.

What Does "Unfreeze" Mean?

When a layer is frozen, its weights are not changed during training.

When a layer is unfrozen, its weights can be updated by backpropagation.

Frozen Layer

Weights
  ↓
Do Not Change


Unfrozen Layer

Weights
  ↓
Updated During Training

Fine-tuning usually means unfreezing only some of the later layers rather than unfreezing the entire model.

Why Not Unfreeze Everything?

Unfreezing the entire pretrained model immediately is often a bad idea, especially when your new dataset is small.

The model could change its useful pretrained knowledge too aggressively and start overfitting to the new dataset.

Too Much Training
        ↓
Pretrained Knowledge Changes Too Much
        ↓
Possible Overfitting


Careful Fine-Tuning
        ↓
Small Adjustments
        ↓
Better Adaptation

This is why fine-tuning is normally done with a small learning rate.

Fine-Tuning in Python

First, we can load a pretrained model and freeze it.

import tensorflow as tf

base_model = tf.keras.applications.MobileNetV2(
    weights="imagenet",
    include_top=False,
    pooling="avg"
)

base_model.trainable = False

At this stage, we are using the model as a feature extractor.

Unfreeze the Model

After training the new classifier, we can unfreeze the pretrained model for fine-tuning.

base_model.trainable = True

Now the pretrained layers are allowed to update their weights during training.

We can also freeze the earlier layers and only fine-tune the later layers.

for layer in base_model.layers[:-20]:
    layer.trainable = False

for layer in base_model.layers[-20:]:
    layer.trainable = True

Here, only the final 20 layers are allowed to learn.

Use a Small Learning Rate

Fine-tuning should generally use a smaller learning rate than the initial training of the new classifier.

model.compile(
    optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.00001
    ),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

A small learning rate makes smaller changes to the pretrained weights.

Large Learning Rate
        ↓
Large Weight Changes
        ↓
May Destroy Useful Knowledge


Small Learning Rate
        ↓
Small Weight Changes
        ↓
Careful Adaptation

Complete Simple Example

import tensorflow as tf

# Load pretrained model
base_model = tf.keras.applications.MobileNetV2(
    weights="imagenet",
    include_top=False,
    pooling="avg"
)

# Freeze the base model first
base_model.trainable = False

# Create the model
model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.Dense(2, activation="softmax")
])

# Train the new classifier first
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

# Later, enable fine-tuning
base_model.trainable = True

# Keep most layers frozen
for layer in base_model.layers[:-20]:
    layer.trainable = False

# Recompile with a small learning rate
model.compile(
    optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.00001
    ),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

model.summary()

This approach first learns the new classifier and then carefully adapts the last part of the pretrained model.

The Fine-Tuning Process

Step 1
Load Pretrained Model
        ↓
Step 2
Freeze Pretrained Layers
        ↓
Step 3
Add New Classifier
        ↓
Step 4
Train New Classifier
        ↓
Step 5
Unfreeze Some Layers
        ↓
Step 6
Use Small Learning Rate
        ↓
Step 7
Fine-Tune
        ↓
Final Model

This two-stage approach is commonly used because it lets the new classifier learn first before changing the pretrained features.

When Should You Use Fine-Tuning?

Fine-tuning is useful when the pretrained model is already reasonably close to your problem, but its existing features are not quite specialized enough.

General Dataset
      ↓
Pretrained Model
      ↓
Your Specific Dataset
      ↓
Fine-Tuning
      ↓
Better Task-Specific Model

For example, a general image model can be adapted to recognize specific plant diseases or specific types of vehicles.

Quick Check