How RNNs Remember Information

How Does an RNN Remember?

An RNN processes a sequence one step at a time.

At every step, it receives:

Current Input
+
Previous Hidden State
        ↓
       RNN
        ↓
New Hidden State

The new hidden state is passed to the next step.

This repeated process allows information from earlier steps to influence later steps.

Think of It Like Taking Notes

Imagine you are reading a story and taking a small set of notes about what has happened so far.

Read sentence 1
      ↓
Take useful notes
      ↓
Read sentence 2
      ↓
Update your notes
      ↓
Read sentence 3
      ↓
Update your notes

An RNN works in a similar way.

Input
  ↓
RNN
  ↓
Hidden State
  ↓
Next Input
  ↓
RNN
  ↓
Updated Hidden State

The analogy is useful, but remember: the RNN's "notes" are actually numerical values learned by the neural network.

Example: Understanding a Sentence

Consider:

The food was very good

The RNN processes the words in order:

The
 ↓
food
 ↓
was
 ↓
very
 ↓
good

After processing each word, the hidden state is updated.

"The"
  ↓
Hidden State 1

"food" + Hidden State 1
  ↓
Hidden State 2

"was" + Hidden State 2
  ↓
Hidden State 3

"very" + Hidden State 3
  ↓
Hidden State 4

"good" + Hidden State 4
  ↓
Hidden State 5

When the RNN reaches "good", its hidden state can contain useful information from earlier words.

Example: Why Previous Information Matters

Compare these two sentences:

The movie was good

The movie was not good

The word "not" changes the meaning of the sentence.

When the RNN reaches "good", information from earlier words can influence its internal state.

The
 ↓
movie
 ↓
was
 ↓
not
 ↓
good

The hidden state at the "good" step is influenced by the information processed before it.

Step-by-Step Memory Process

Suppose the input sequence is:

10 → 20 → 30

We start with an initial hidden state:

h₀ = 0

First step:

x₁ = 10
h₀ = 0

10 + previous information
        ↓
       RNN
        ↓
      h₁

Second step:

x₂ = 20
h₁ = previous hidden state

20 + h₁
   ↓
  RNN
   ↓
  h₂

Third step:

x₃ = 30
h₂ = previous hidden state

30 + h₂
   ↓
  RNN
   ↓
  h₃

The important part is that h₁ influences h₂, and h₂ influences h₃.

The Basic Formula

A simplified RNN hidden-state equation is:

hₜ = tanh(Wₓₕxₜ + Wₕₕhₜ₋₁ + b)

The important parts are:

xₜ
↓
Current input


hₜ₋₁
↓
Previous hidden state


Wₓₕ
↓
Weights for the current input


Wₕₕ
↓
Weights for the previous hidden state


b
↓
Bias


hₜ
↓
New hidden state

The important relationship is:

Current Input
      +
Previous Hidden State
      ↓
RNN calculation
      ↓
New Hidden State

Simple Numerical Example

Let's simplify the calculation so the idea is easy to see.

Suppose:

Current input = 4
Previous hidden state = 2

Input weight = 0.5
Hidden-state weight = 0.3
Bias = 0

First calculate the input contribution:

4 × 0.5 = 2

Then calculate the previous hidden-state contribution:

2 × 0.3 = 0.6

Add them:

2 + 0.6 = 2.6

The RNN then applies an activation function such as tanh:

new hidden state = tanh(2.6)

The important point is not the exact number. The important point is that the new state depends on both the current input and previous state.

Simple Python Example

We can demonstrate the idea of carrying information using simple Python:

inputs = [10, 20, 30]

hidden_state = 0

for value in inputs:

    hidden_state = value + hidden_state

    print("Input:", value)
    print("Hidden State:", hidden_state)

Output:

Input: 10
Hidden State: 10

Input: 20
Hidden State: 30

Input: 30
Hidden State: 60

This is not a real RNN. It is only a simple demonstration of how information can be carried forward.

Understand the Python Code

inputs = [10, 20, 30]

This creates our sequence.

hidden_state = 0

We start with an initial hidden state.

for value in inputs:

We process each input in order.

hidden_state = value + hidden_state

This combines the current value with the information carried from the previous step.

Step 1:

10 + 0 = 10


Step 2:

20 + 10 = 30


Step 3:

30 + 30 = 60

Therefore, the previous state affects the next state.

How a Real RNN Does It

In TensorFlow/Keras, we can create a simple RNN:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.SimpleRNN(4)
])

model.build(input_shape=(None, 5, 1))

model.summary()

Here:

SimpleRNN(4)

means the RNN has 4 hidden units.

So the hidden state is not just one number. It is a vector:

[
    0.12,
    -0.35,
    0.81,
    0.44
]

During sequence processing, this vector is updated at every step.

What Does the RNN Actually Remember?

This is an important distinction.

An RNN does not store previous sentences or previous numbers exactly like a database.

Instead, it learns numerical representations that can capture useful information from previous inputs.

Previous Inputs
      ↓
RNN processing
      ↓
Hidden State
      ↓
Numerical representation
of useful information

So "memory" is a useful way to understand the concept, but it is not literal memory.

The Memory Is Updated

The hidden state is not fixed. It changes whenever a new input arrives.

Input 1
 ↓
Hidden State 1

Input 2
 +
Hidden State 1
 ↓
Hidden State 2

Input 3
 +
Hidden State 2
 ↓
Hidden State 3

Therefore, the RNN's current state represents information based on what it has processed so far.

Two Simple Examples

Example 1 — Text

The → movie → was → not → good

When the RNN reaches "good", its hidden state has already been updated using the earlier words.

Example 2 — Time Series

100 → 120 → 140 → 160 → ?

The hidden state carries information about earlier values while processing later values.

Does an RNN Remember Everything?

No.

This is an important limitation of basic RNNs.

When sequences become very long, information from early steps can become difficult to preserve effectively.

Early Information
      ↓
      ↓
      ↓
Many RNN Steps
      ↓
      ↓
      ↓
Current Step

Older information can become
harder to preserve.

This problem is related to the vanishing gradient problem.

Later, we will see how architectures such as LSTM and GRU were designed to handle long-term dependencies better.

The Complete Idea

Sequence
   ↓
Input 1
   ↓
RNN
   ↓
Hidden State 1
   ↓
Input 2 + Hidden State 1
   ↓
RNN
   ↓
Hidden State 2
   ↓
Input 3 + Hidden State 2
   ↓
RNN
   ↓
Hidden State 3
   ↓
...
   ↓
Final Output

This repeated hidden-state update is the basic mechanism behind how an RNN carries information through a sequence.

Final Summary

RNN receives:
Current Input
+
Previous Hidden State

        ↓

RNN processes them

        ↓

New Hidden State

        ↓

Passed to the next step

Remember this one sentence:

RNNs remember information by continuously
updating and carrying their hidden state
from one step to the next.