Cell State

What Is the Cell State?

The cell state is a part of an LSTM that carries information through the sequence.

A simple way to think about it is:

Cell State = LSTM's memory pathway

It allows information from an earlier time step to continue toward later time steps.

Time 1
  ↓
Cell State
  ↓
Time 2
  ↓
Cell State
  ↓
Time 3
  ↓
Cell State
  ↓
Time 4

The important point is that the cell state does not start from zero at every word or time step. It is updated and carried forward.

Simple Example

Imagine an LSTM reading this sentence:

"John lives in London.
He works as a developer.
He likes football.
He moved there when he was 20."

Suppose the important information for the task is:

John → London

The LSTM processes other information after learning that fact.

London
   ↓
developer
   ↓
football
   ↓
age 20
   ↓
later information

The cell state provides a pathway through which useful information can continue to be carried.

London
   ↓
┌──────────────┐
│ Cell State   │
└──────────────┘
   ↓
other words
   ↓
┌──────────────┐
│ Cell State   │
└──────────────┘
   ↓
later time step

Why Do We Need a Cell State?

Sequence problems often contain information that is important now but is needed much later.

For example:

"Sarah was born in India.
...
...
...
She moved to Germany.
...
...
...
Where was Sarah born?"

The answer appeared near the beginning:

India

But many other words appeared afterward.

The cell state gives the LSTM a dedicated memory pathway for carrying useful information across these time steps.

Cell State Is Not a Database

Do not misunderstand the word "memory."

An LSTM does not store sentences or words exactly like a database.

Information is represented using numerical values.

Text
 ↓
Numbers / representations
 ↓
LSTM
 ↓
Numerical Cell State
 ↓
Updated Cell State

So when we say the LSTM "remembers London", it means the learned numerical representation associated with that information can continue through the network.

How Do Gates Control the Cell State?

The cell state does not simply keep everything forever.

The LSTM gates control it.

Previous Cell State
        ↓
   Forget Gate
        ↓
Remove / reduce old information
        ↓
   Input Gate
        ↓
Add useful new information
        ↓
Updated Cell State

This means the cell state is continuously updated as the sequence is processed.

Step 1 — Remove Unnecessary Information

Suppose the previous cell state contains information about:

London
Developer
Football

Now the LSTM receives new information:

"John moved to Paris."

The old information about London may no longer be useful.

Old Cell State
      ↓
Forget Gate
      ↓
Reduce unnecessary information
      ↓
Updated memory

The Forget Gate controls this part of the update.

Step 2 — Add New Information

The LSTM then considers information from the current input that may be useful.

Current Input

"John moved to Paris."

        ↓

Useful new information

Paris

        ↓

Input Gate

        ↓

Add to Cell State

The cell state can therefore change from:

Old memory:
London

to something representing the updated information:

Updated memory:
Paris

In the real neural network, this is not a literal word replacement. The cell state consists of numerical representations.

Cell State Through Time

Imagine an LSTM processing four inputs:

Input 1
   ↓
Input 2
   ↓
Input 3
   ↓
Input 4

The cell state flows through each step:

Input 1
   ↓
Cell State 1
   ↓
Input 2
   ↓
Cell State 2
   ↓
Input 3
   ↓
Cell State 3
   ↓
Input 4
   ↓
Cell State 4

Each new cell state is based on the previous cell state together with information from the current input.

Simple Numerical Example

Let's use a very simplified example to understand the idea.

Suppose the old cell state is represented by:

Old memory = 10

Suppose the Forget Gate produces:

Forget value = 0.8

Then approximately:

10 × 0.8 = 8

So 8 units of the old information remain in this simplified example.

Now suppose the Input Gate allows:

New information = 5
Input value = 0.6

The new contribution is:

5 × 0.6 = 3

The simplified updated cell state becomes:

Old information
      +
New information

8 + 3

= 11

This is only an educational simplification. A real LSTM performs vector and matrix operations rather than using one simple number.

The Cell State Update Formula

The simplified idea above corresponds to the actual structure:

cₜ = fₜ × cₜ₋₁ + iₜ × c̃ₜ

Here:

cₜ
↓
Current Cell State

cₜ₋₁
↓
Previous Cell State

fₜ
↓
Forget Gate

iₜ
↓
Input Gate

c̃ₜ
↓
Candidate new information

The formula says:

Current Memory
=
Part of Old Memory
+
Part of New Information

That is the most important formula to understand in this lesson.

Cell State vs Hidden State

This is where many beginners get confused.

LSTM has both a cell state and a hidden state, but they serve different purposes.

Cell State
    ↓
Main memory pathway


Hidden State
    ↓
Current output / exposed state

Think of it this way:

Cell State
=
What information continues through memory?


Hidden State
=
What information is exposed right now?

The distinction is simplified, but it is a useful mental model when learning LSTM.

Complete Cell State Flow

Previous Cell State
        │
        ↓
   ┌───────────┐
   │ Forget    │
   │   Gate    │
   └───────────┘
        │
        ↓
Old information
that remains
        │
        +
        │
Current Input
        │
        ↓
   ┌───────────┐
   │ Input     │
   │   Gate    │
   └───────────┘
        │
        ↓
Useful new information
        │
        ↓
Updated Cell State
        │
        ↓
   ┌───────────┐
   │ Output    │
   │   Gate    │
   └───────────┘
        │
        ↓
Hidden State / Output

Why Is the Cell State Important for Long-Term Dependencies?

Consider this example:

"Tom grew up in India.
...
many sentences...
...
Tom moved to Canada.
...
many sentences...
...
Where did Tom grow up?"

The answer is:

India

The information needed to answer the question appeared much earlier.

The cell state provides a pathway through which important information can be carried across many time steps.

India
 ↓
Cell State
 ↓
Time Step 2
 ↓
Cell State
 ↓
Time Step 3
 ↓
Cell State
 ↓
...
 ↓
Later Time Step
 ↓
Useful information

Does the Cell State Stay the Same?

No.

This is an important point.

The cell state is continuously updated as the LSTM reads new inputs.

Old Cell State
      ↓
Forget some information
      ↓
Add new information
      ↓
New Cell State
      ↓
Next time step
      ↓
Update again

So it is better to think of the cell state as a continuously updated memory, not a permanent storage box.

Important: LSTM Does Not Remember Everything

The word "memory" can be misleading.

LSTM does not automatically preserve every piece of information in a sequence.

The network learns during training how information should be controlled.

Training
   ↓
Learn weights
   ↓
Learn gate behavior
   ↓
Control Cell State
   ↓
Preserve useful information
   ↓
Reduce unnecessary information

So the ability to remember useful information is learned from data.

Easy Real-Life Analogy

Imagine you are carrying a notebook while reading a long story.

Story information
       ↓
Your notebook
       ↓
Keep important information
       ↓
Erase outdated information
       ↓
Write new important information
       ↓
Continue reading

The notebook is similar to the idea of the cell state.

But remember: an actual LSTM stores numerical representations, not sentences written in a notebook.

Small Python Example

We can represent the basic idea using ordinary Python numbers:

old_cell_state = 10

forget_gate = 0.8
input_gate = 0.6
new_information = 5

remaining_memory = old_cell_state * forget_gate

new_memory = new_information * input_gate

cell_state = remaining_memory + new_memory

print(cell_state)

Output:

11.0

Let's understand it:

old_cell_state = 10

forget_gate = 0.8

10 × 0.8
= 8

Then:

new_information = 5

input_gate = 0.6

5 × 0.6
= 3

Finally:

8 + 3
= 11

Again, this is only a simplified demonstration. Real LSTMs work with vectors, matrices, activation functions, and learned parameters.

The Big Picture

                LSTM

Previous Cell State
        │
        ↓
   Forget Gate
        │
        ↓
Remove / reduce old information
        │
        +
        │
Current Input
        │
        ↓
   Input Gate
        │
        ↓
Add useful new information
        │
        ↓
Updated Cell State
        │
        ↓
   Output Gate
        │
        ↓
Hidden State / Output

The cell state is therefore the central memory pathway that the gates continuously modify.

Final Summary

Cell State
→ LSTM's main memory pathway.

It carries information
→ From earlier time steps
→ Toward later time steps.

Forget Gate
→ Controls old information.

Input Gate
→ Controls new information.

Updated Cell State
→ Old useful information
  +
  New useful information.

The one formula worth remembering is:

cₜ = fₜ × cₜ₋₁ + iₜ × c̃ₜ

In plain English:

New Memory
=
Keep Some Old Memory
+
Add Some New Information