Gradients

What Is a Gradient?

In the previous lesson, we learned about derivatives. A derivative tells us how a function changes with respect to one variable.

But AI models usually have many variables.

Variable 1 → weight1
Variable 2 → weight2
Variable 3 → weight3
Variable 4 → weight4
Variable 5 → weight5

We need to know how the output changes when each variable changes.

All of these derivatives together form a gradient.

Derivative vs Gradient

This distinction is very important.

One variable
     ↓
Derivative


Multiple variables
     ↓
Multiple derivatives
     ↓
Gradient

For example, suppose we have only one variable:

f(x) = x²

Derivative:

f'(x) = 2x

If we have two variables:

f(x, y) = x² + y²

We need a derivative for x and another derivative for y.

∂f/∂x = 2x

∂f/∂y = 2y

Together, these derivatives form the gradient.

Simple Gradient Example

Consider this function:

f(x, y) = x² + y²

We calculate the derivative with respect to x:

∂f/∂x = 2x

Then we calculate the derivative with respect to y:

∂f/∂y = 2y

Therefore the gradient is:

Gradient = [ 2x ]
           [ 2y ]

The gradient contains both derivatives.

Calculate a Gradient at a Point

Let's use:

f(x, y) = x² + y²

Suppose:

x = 3
y = 4

Our derivatives are:

∂f/∂x = 2x
∂f/∂y = 2y

Substitute the values:

∂f/∂x = 2 × 3
       = 6

∂f/∂y = 2 × 4
       = 8

Therefore:

Gradient = [ 6 ]
           [ 8 ]

This tells us how the function changes in the x and y directions.

Think of a Gradient Like a Direction

Imagine standing on a mountain. You can move in different directions.

             Mountain

                /\
               /  \
              /    \
             /  ↑   \
            /   │    \
           /    │     \
          /_____|______\

             You

Some directions take you uphill quickly. Other directions may take you downhill.

The gradient points toward the direction where the function increases most quickly.

Gradient
   ↓
Direction of greatest increase

This becomes extremely useful when training an AI model.

Gradient and AI Loss

An AI model makes predictions. We compare those predictions with the correct answers and calculate a loss.

Input
  ↓
AI Model
  ↓
Prediction
  ↓
Loss

The model has many weights. Each weight can affect the loss.

weight1 ──→ affects Loss
weight2 ──→ affects Loss
weight3 ──→ affects Loss
weight4 ──→ affects Loss

We calculate a derivative for each weight. These derivatives together form the gradient.

Derivative for weight1
Derivative for weight2
Derivative for weight3
Derivative for weight4
          ↓
       Gradient

A Simple AI Example

Imagine a very small model with two weights.

weight1 = 2
weight2 = 3

Suppose the loss function is:

Loss = weight1² + weight2²

The derivatives are:

∂Loss/∂weight1 = 2 × weight1

∂Loss/∂weight2 = 2 × weight2

Substitute the values:

∂Loss/∂weight1 = 2 × 2
                = 4

∂Loss/∂weight2 = 2 × 3
                = 6

Therefore the gradient is:

Gradient = [ 4 ]
           [ 6 ]

The gradient tells us how the loss changes with respect to both weights.

Why Does the Gradient Matter?

Suppose one weight has a very large derivative.

weight1 → derivative = 2
weight2 → derivative = 20

The loss is much more sensitive to changes in weight2.

The gradient gives the learning algorithm information about how strongly each parameter affects the loss.

Gradient
   ↓
How each parameter affects Loss
   ↓
Use this information
to update parameters

Gradient and Direction

Remember that the gradient points toward the direction of greatest increase of a function.

When training an AI model, we normally want to decrease the loss.

Gradient
   ↓
Direction of increasing Loss

Negative Gradient
   ↓
Direction of decreasing Loss

This is why optimization algorithms move in the opposite direction of the gradient.

The most common method used for this is Gradient Descent.

Gradient Descent Connection

We can now connect everything we have learned.

AI Model
   ↓
Prediction
   ↓
Calculate Loss
   ↓
Calculate Derivatives
   ↓
Build Gradient
   ↓
Move opposite to Gradient
   ↓
Reduce Loss
   ↓
Better Model

The next lesson, Chain Rule, explains how derivatives can be calculated when several functions are connected together.

This becomes especially important in neural networks.

Derivative vs Gradient vs Gradient Descent

These three terms are related, but they are not the same.

Derivative
    ↓
Measures change for one variable


Gradient
    ↓
Collection of derivatives
for many variables


Gradient Descent
    ↓
Uses the gradient to reduce loss

Keep this distinction clear. Mixing these three concepts will make neural-network mathematics unnecessarily confusing.