What Is an Activation Function?

Remember the Neuron From Lesson 2

In the previous lesson, we calculated the value of a neuron using inputs, weights, and bias.

z = (x1 * w1) + (x2 * w2) + bias

For example:

x1 = 2
x2 = 3

w1 = 0.5
w2 = 0.4

bias = 1

z = (2 * 0.5) + (3 * 0.4) + 1

z = 3.2

We now have:

What Does the Activation Function Do?

The activation function takes the neuron's calculated value and transforms it according to a specific rule.

Different activation functions transform the same input in different ways.

For example, one activation function might convert a large positive value into a value close to 1, while another might simply keep positive values and turn negative values into 0.

A Simple Example

Think of a neuron as a machine that first calculates a score.

The activation function acts like a rule that decides how that score should be converted into the neuron's output.

The important point is that the activation function does not replace the neuron's calculation. It comes after the weighted sum and bias.

A Very Simple Activation Rule

Imagine an activation function with this simple rule:

If z is positive:
    output = 1

If z is zero or negative:
    output = 0

Suppose:

z = 3.2

Since `3.2` is positive:

output = 1

This is the basic idea behind a step function, which we will study separately later in this lesson.

Why Not Just Use z Directly?

A beginner might ask:

Because activation functions give neural networks the ability to model non-linear relationships.

Without activation functions, stacking many ordinary linear calculations would still result in a linear transformation. Adding more layers would not give the network the expressive power we expect from a deep neural network.

What Does "Non-Linear" Mean?

Linear relationships are relatively simple patterns. Many real-world problems are much more complicated.

For example, imagine trying to identify whether an image contains a cat.

These relationships are not simply a straight-line calculation. Activation functions help neural networks learn such complex relationships.

Where Does the Activation Function Fit?

The Basic Formula

We can represent the neuron and activation function together using:

z = (x1 * w1) + (x2 * w2) + bias

output = activation(z)

Or more generally:

output = f(z)

Here, `f` represents the activation function.

There Is More Than One Activation Function

Neural networks can use different activation functions depending on the problem and where the neuron is being used.

We will study each of these separately instead of trying to learn all their formulas at once.

Same Input, Different Activation Functions

Suppose a neuron produces:

z = 2

Different activation functions can transform this same value differently.

The input from the neuron is the same. The resulting output depends on the activation function being used.

A Simple Python Example

We can create a very simple activation function using Python.

def step_function(z):
    if z > 0:
        return 1
    else:
        return 0


result = step_function(3.2)

print(result)

Output:

1

Here the function receives `3.2`. Because the value is greater than zero, it returns `1`.

Another Example

result = step_function(-2)

print(result)

Output:

0

The function receives `-2`. Since it is not greater than zero, the function returns `0`.

Activation Function vs Neuron Calculation

These are two different steps. Do not mix them together.

The Big Picture

Input
  ↓
Multiply by weights
  ↓
Add weighted values
  ↓
Add bias
  ↓
z
  ↓
Activation function
  ↓
Output

This sequence happens repeatedly throughout a neural network.