Activation Functions

What Is an Activation Function?

An activation function is a mathematical function that takes the output of a neuron and transforms it into another value.

A very simple way to think about it is:

Input
  ↓
Mathematical calculation
  ↓
Activation Function
  ↓
Output

The activation function decides what value should be passed forward to the next part of the neural network.

Why Do Neural Networks Need Activation Functions?

Without activation functions, a neural network would be limited to simple mathematical relationships.

Real-world problems are usually much more complicated.

For example, an AI model may need to learn:

Images
Speech
Language
Fraud patterns
Customer behavior
Medical patterns

Activation functions introduce non-linearity into the network. This allows the network to learn much more complicated patterns.

This is one of the most important reasons activation functions are used in neural networks.

A Simple Neuron Example

Before applying an activation function, a neuron first performs a mathematical calculation using inputs and weights.

For example:

input = 5
weight = 2

value = input * weight

value = 5 * 2

value = 10

The neuron has calculated the value 10. Now an activation function can be applied to this value.

10
 ↓
Activation Function
 ↓
Output

Activation Function in a Neural Network

In a neural network, this process happens across many neurons and layers.

Input
  ↓
Weighted Calculation
  ↓
Activation Function
  ↓
Neuron Output
  ↓
Next Layer
  ↓
Activation Function
  ↓
Final Output

Each layer can transform the information before passing it to the next layer.

ReLU

ReLU stands for Rectified Linear Unit. It is one of the most commonly used activation functions in neural networks.

Its formula is very simple:

ReLU(x) = max(0, x)

This means:

If x is positive
→ keep x

If x is negative
→ return 0

For example:

ReLU(5)
= 5

ReLU(2)
= 2

ReLU(-3)
= 0

ReLU(-10)
= 0

So ReLU removes negative values and keeps positive values.

Why Is ReLU Useful?

ReLU is simple and computationally efficient. It also introduces non-linearity into a neural network.

Input

-5   -2   0   2   5
 ↓    ↓   ↓   ↓   ↓
ReLU
 ↓    ↓   ↓   ↓   ↓
 0    0   0   2   5

Because of this behavior, ReLU is widely used in hidden layers of neural networks.

Sigmoid

Sigmoid is another activation function. It converts a value into a number between 0 and 1.

Its formula is:

sigmoid(x) = 1 / (1 + e^(-x))

You do not need to memorize the formula right now. The important idea is what the function does.

Large negative value
→ close to 0

0
→ 0.5

Large positive value
→ close to 1

For example:

sigmoid(-5) ≈ 0.0067

sigmoid(0) = 0.5

sigmoid(5) ≈ 0.9933

Why Is Sigmoid Useful?

Sigmoid is useful when we want an output that can be interpreted as a value between 0 and 1.

This makes it useful for binary classification problems.

Prediction

0.05
→ Very unlikely

0.50
→ Uncertain

0.95
→ Very likely

For example, a model predicting whether an email is spam could produce a value close to 0 or 1.

Tanh

Tanh stands for hyperbolic tangent. It is another activation function used in neural networks.

Its output is between -1 and 1.

Very negative input
→ close to -1

Input = 0
→ 0

Very positive input
→ close to 1

For example:

tanh(-5) ≈ -1

tanh(0) = 0

tanh(5) ≈ 1

ReLU vs Sigmoid vs Tanh

These activation functions behave differently.

ReLU

Input:  -5  -2   0   2   5
Output:  0   0   0   2   5


Sigmoid

Input:  -5      0       5
Output:  0.0067  0.5   0.9933


Tanh

Input:  -5      0       5
Output:   -1     0       1

The important thing is not to memorize every formula. First understand the behavior of each function.

Activation Functions and Classification

The activation function used in the output layer often depends on the type of problem we are solving.

For a simple binary classification problem, sigmoid can be useful.

Question:

Is this email spam?

Neural Network
      ↓
Sigmoid
      ↓
0.92

Interpretation:

92% probability of spam

The exact interpretation depends on how the model and labels were defined, but the important idea is that sigmoid produces a value between 0 and 1.

Activation Functions and Deep Learning

A deep neural network contains multiple layers. Activation functions are typically applied between these layers.

Input Layer
     ↓
Weighted Calculation
     ↓
ReLU
     ↓
Hidden Layer
     ↓
Weighted Calculation
     ↓
ReLU
     ↓
Hidden Layer
     ↓
Weighted Calculation
     ↓
Output

Each activation function helps transform the information before it moves deeper into the network.

What Happens Without Activation Functions?

This is an important question.

Imagine a neural network containing many layers but no activation functions.

Input
  ↓
Linear Calculation
  ↓
Linear Calculation
  ↓
Linear Calculation
  ↓
Output

Even though there are multiple layers, the entire network can effectively behave like one linear mathematical transformation.

That would make the network much less capable of learning complicated patterns.

Activation functions solve this problem by introducing non-linearity.

Simple Python Example

We can implement ReLU ourselves using a simple Python function.

def relu(x):
    return max(0, x)


print(relu(5))
print(relu(-3))
print(relu(10))

Output:

5
0
10

This is exactly the basic behavior of ReLU: positive values stay positive, while negative values become zero.

A Small Neural Network Example

Let's connect the ideas together.

input = 5
weight = 2
bias = 1

value = (input * weight) + bias

print(value)

The calculation gives:

(5 × 2) + 1

= 11

Now apply ReLU:

relu(11)

= 11

So the complete process is:

Input
  ↓
5

Weight
  ↓
2

Bias
  ↓
1

Weighted calculation
  ↓
(5 × 2) + 1
  ↓
11

ReLU
  ↓
11

Output

How This Connects to the Math We Learned

You already learned functions in the Calculus section. An activation function is simply another mathematical function.

Mathematical Function

Input
  ↓
Function
  ↓
Output


Activation Function

Neuron Value
  ↓
Activation Function
  ↓
Neuron Output

Later, when we study gradients and backpropagation, we will calculate how these functions affect the loss of the neural network.

Why Activation Functions Matter in AI

Activation functions are one of the key pieces that allow neural networks to learn complex relationships.

Real-World Data
      ↓
Neural Network
      ↓
Weighted Calculations
      ↓
Activation Functions
      ↓
Learn Complex Patterns
      ↓
Prediction

For example, when an image model learns to recognize an object, it needs to combine many different patterns and features. Non-linear activation functions help the network represent those complex relationships.