ReLU

What Is ReLU?

ReLU is an activation function that takes the value calculated by a neuron and applies a very simple rule.

The formula is:

ReLU(x) = max(0, x)

This means ReLU compares the input with zero and returns the larger value.

How Does ReLU Work?

ReLU has only two simple rules.

For example:

ReLU(-5) = 0
ReLU(-2) = 0

ReLU(2) = 2
ReLU(5) = 5

Example 1 — Negative Input

Suppose the input is:

x = -5

ReLU checks:

max(0, -5)

Zero is greater than negative five, so the result is:

ReLU(-5) = 0

Example 2 — Positive Input

Now suppose:

x = 5

ReLU checks:

max(0, 5)

Five is greater than zero, so ReLU keeps the value:

ReLU(5) = 5

Example 3 — Zero

If the input is zero:

x = 0

Then:

ReLU(0) = 0

Example With Multiple Values

Suppose a neural network produces:

[-5, -2, 0, 3, 7]

Apply ReLU to every value:

-5 → 0
-2 → 0
 0 → 0
 3 → 3
 7 → 7

The final result is:

[0, 0, 0, 3, 7]

ReLU Examples

Why Do We Need ReLU?

Neural networks need activation functions because they introduce non-linearity.

Without activation functions, stacking many linear layers would still behave like one linear transformation.

ReLU allows the network to learn more complex patterns.

ReLU Inside a Neuron

A neuron first calculates a weighted sum and adds the bias.

z = (x1 * w1) + (x2 * w2) + bias

Then ReLU is applied:

output = ReLU(z)

For example:

x1 = 2
x2 = 3

w1 = 0.5
w2 = 0.4

bias = -1

z = (2 * 0.5) + (3 * 0.4) - 1

z = 1.2

Now apply ReLU:

ReLU(1.2) = 1.2

What If the Neuron Produces a Negative Value?

Suppose the neuron calculates:

z = -2.5

ReLU converts it to zero:

ReLU(-2.5) = 0

Build ReLU With Python

We can implement ReLU ourselves using Python's max() function.

def relu(x):
    return max(0, x)


print(relu(-5))
print(relu(-2))
print(relu(0))
print(relu(3))
print(relu(7))

Output:

0
0
0
3
7

Understand the Python Code

First, we create a function:

def relu(x):

The function receives one value called x.

Then:

return max(0, x)

Python compares 0 and x and returns whichever value is larger.

For example:

relu(-5)

max(0, -5)

→ 0

And:

relu(5)

max(0, 5)

→ 5

Simple Real-World Example

Imagine a neural network looking at an image. Different neurons may detect edges, shapes, textures, or other useful patterns.

Suppose one neuron produces:

activation = -2.4

ReLU changes this to:

ReLU(-2.4) = 0

Another neuron might produce:

activation = 3.8

ReLU keeps it:

ReLU(3.8) = 3.8

The next layer therefore receives:

[0, 3.8, 0, 2.1, 5.4]

ReLU vs Sigmoid

One Problem With ReLU

ReLU is useful, but it has a limitation.

Every negative input becomes zero:

ReLU(-1)  = 0
ReLU(-5)  = 0
ReLU(-10) = 0

For negative inputs, the gradient is zero. If a neuron repeatedly stays in this region during training, it can stop receiving useful updates. This is commonly called the dying ReLU problem.

Variants such as Leaky ReLU can help address this issue.

The Big Picture

Input
  ↓
Weights + Bias
  ↓
Linear Calculation
  ↓
z
  ↓
ReLU
  ↓
Output
  ↓
Next Layer

ReLU is therefore an important part of many neural networks because it adds non-linearity while keeping the calculation simple.