Sigmoid

Remember the Step Function

The Step Function gave us a hard decision:

Sigmoid behaves differently. Instead of immediately jumping from 0 to 1, it changes smoothly.

The Sigmoid Formula

The mathematical formula for Sigmoid is:

σ(z) = 1 / (1 + e^(-z))

You do not need to memorize the formula immediately. The important thing to understand first is what the function does to the input.

The symbol σ is commonly used to represent the Sigmoid function.

Example 1 — z = 0

Start with the easiest example:

z = 0

Apply the Sigmoid formula:

σ(0) = 1 / (1 + e^0)

e^0 = 1

σ(0) = 1 / (1 + 1)

σ(0) = 0.5

Therefore:

Example 2 — Positive Input

Now suppose:

z = 2

Applying Sigmoid gives approximately:

σ(2) ≈ 0.881

Notice that the output is close to 1, but it is not exactly 1.

Example 3 — Negative Input

Now suppose:

z = -2

Applying Sigmoid gives approximately:

σ(-2) ≈ 0.119

The output is now close to 0.

Notice the Pattern

As the input becomes more positive, the Sigmoid output approaches 1. As the input becomes more negative, the output approaches 0.

Sigmoid "Squashes" the Input

The input can be any number:

-100
-10
-2
0
2
10
100

But Sigmoid converts all of them into the range:

0 < output < 1

Using Sigmoid for Binary Classification

One common use of Sigmoid is the output layer of a binary classification model.

Suppose we want to predict:

Suppose the model calculates:

z = 2

Sigmoid converts it to:

sigmoid(2) ≈ 0.881

This can be interpreted as a score of about 88.1% toward the positive class, depending on how the model and labels are defined.

Turning the Sigmoid Output Into a Class

Sigmoid gives us a continuous value. If we need a final 0-or-1 decision, we can apply a threshold.

A common example is:

if probability >= 0.5:
    prediction = 1
else:
    prediction = 0

For example:

probability = 0.881

Since:

0.881 >= 0.5

the prediction becomes:

prediction = 1

Sigmoid With Python

Python provides the mathematical tools needed to implement Sigmoid ourselves.

import math


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


print(sigmoid(0))
print(sigmoid(2))
print(sigmoid(-2))

Output:

0.5
0.8807970779778823
0.11920292202211755

Understand the Python Code

First:

import math

We import Python's built-in math module because the Sigmoid formula uses the mathematical constant e.

Then we create the function:

def sigmoid(z):

The function accepts one value, z.

Then:

return 1 / (1 + math.exp(-z))

This is the Python version of:

σ(z) = 1 / (1 + e^(-z))

For example:

sigmoid(2)

gives approximately:

0.8808

Sigmoid Inside a Neuron

Remember that the neuron first calculates z.

z = (x1 * w1) + (x2 * w2) + bias

output = sigmoid(z)

For example:

x1 = 2
x2 = 3

w1 = 0.5
w2 = 0.4

bias = 1

z = (x1 * w1) + (x2 * w2) + bias

output = sigmoid(z)

print(output)

First calculate:

z = (2 × 0.5) + (3 × 0.4) + 1
z = 3.2

Then apply Sigmoid:

sigmoid(3.2) ≈ 0.961

Step Function vs Sigmoid

Is Sigmoid Perfect?

No. Sigmoid is useful, but it has limitations.

When the input becomes extremely positive or extremely negative, the output gets very close to 1 or 0. The gradient can become very small.

This is one reason Sigmoid is generally not the default activation function for hidden layers in modern deep networks. ReLU and its variants are commonly preferred there.

The Big Picture

Inputs
   ↓
Weights
   ↓
Weighted Sum
   ↓
+ Bias
   ↓
z
   ↓
Sigmoid
   ↓
Value between 0 and 1
   ↓
Prediction / Next Layer

The most important thing is to remember that Sigmoid does not calculate the neuron's weighted sum. It transforms the value after that calculation.