Why Do We Need Activation Functions?

First, Remember What a Neuron Does

From the previous lessons, a neuron first calculates a weighted sum and adds a bias.

z = (x1 * w1) + (x2 * w2) + bias

Then the activation function is applied:

output = activation(z)

What Happens Without an Activation Function?

Suppose a neuron simply calculates:

output = z

That means the neuron is only performing a linear mathematical transformation.

Now imagine we connect several such layers together.

It looks like a deep network, but mathematically the combination of linear transformations is still just another linear transformation.

A Simple Mathematical Example

Imagine the first layer performs:

y = 2x

Then the second layer performs:

z = 3y

Substitute `y`:

z = 3(2x)

z = 6x

We used two layers, but the final result is still just a linear function:

z = 6x

Adding More Layers Does Not Solve the Problem

You might think:

The problem is that if every layer only performs a linear transformation, the entire network can still be represented as one linear transformation.

So simply making the network deeper does not magically make it capable of learning arbitrary complex patterns.

This Is Where Non-Linearity Comes In

Activation functions introduce a non-linear transformation between layers.

This allows different layers to learn different levels of complex relationships.

Example 1 — Recognizing a Cat

Imagine a neural network that receives an image.

The network needs to learn complicated relationships between thousands or millions of values.

A simple linear calculation is not enough to represent all of these relationships. Non-linear activation functions allow the network to build more complex representations layer by layer.

Example 2 — Predicting House Prices

Suppose we want to predict a house price using:

Real house prices are influenced by combinations of these factors. For example, the effect of an additional bedroom may depend on the size and location of the house.

Activation functions help the network model these more complicated relationships.

Why Are They Especially Important in Deep Learning?

A deep neural network contains multiple layers.

The activation functions between layers allow the network to build increasingly complex representations.

Without vs With Activation Functions

See the Difference With Python

First, imagine a simple linear function:

def linear(x):
    return 2 * x


print(linear(3))

Output:

6

Now imagine applying another linear function:

def layer1(x):
    return 2 * x


def layer2(x):
    return 3 * x


value = layer1(3)
output = layer2(value)

print(output)

Output:

18

Mathematically:

3 × 2 × 3 = 18

The multiple layers still behave like one linear transformation:

output = 6 × x

Adding a Non-Linear Activation

Now introduce a simple ReLU activation:

def relu(x):
    return max(0, x)

The network can now perform:

value = layer1(3)

value = relu(value)

output = layer2(value)

print(output)

The activation function changes the behavior between layers.

This is the basic idea behind why activation functions make neural networks much more powerful.

The Big Picture

Input
  ↓
Weights + Bias
  ↓
Linear Calculation
  ↓
Activation Function
  ↓
Non-Linear Output
  ↓
Next Layer
  ↓
Activation Function
  ↓
Next Layer
  ↓
Prediction

This pattern is repeated throughout a neural network.