What Is a Transformer?

1. What Is a Transformer?

A Transformer is a type of neural network architecture that uses attention to process sequence data.

In simple terms:

Transformer
    ↓
Looks at relationships between items
    ↓
Understands which items are important
    ↓
Produces useful representations
    ↓
Makes a prediction

For text, the items are usually words or tokens.

The cat sat on the mat.

A Transformer does not simply process this sentence one word at a time and forget the previous words. Instead, its attention mechanism allows it to examine relationships between tokens.

2. Why Do We Need Transformers?

Before Transformers, RNNs and LSTMs were commonly used for sequence problems.

RNN / LSTM

Word 1
  ↓
Word 2
  ↓
Word 3
  ↓
Word 4
  ↓
Word 5

This sequential approach can make it difficult to efficiently process very long sequences.

Transformers introduced a different idea:

Instead of mainly thinking:

Word 1 → Word 2 → Word 3 → Word 4

Think:

Word 1 ↔ Word 2
Word 1 ↔ Word 3
Word 1 ↔ Word 4
Word 2 ↔ Word 3
Word 2 ↔ Word 4
...

Attention allows the model to determine which relationships are important.

3. Simple Example

Consider this sentence:

The animal didn't cross the road because it was tired.

What does "it" refer to?

A language model needs to understand the relationship between "it" and the relevant word in the sentence.

Attention helps the model assign importance to different words when processing a particular word.

The   animal   didn't   cross   the   road   because   it
             ↑                                  ↑
             └──────── relationship ────────────┘

This is the basic intuition behind attention: look at the other tokens and determine which ones matter for the current token.

4. Transformer vs RNN

An RNN processes sequence information recurrently. A Transformer uses attention to model relationships between tokens.

RNN / LSTM

Input
 ↓
Step 1
 ↓
Step 2
 ↓
Step 3
 ↓
Step 4
 ↓
Output


Transformer

Input tokens
 ↓
Attention
 ↓
Each token can interact with other tokens
 ↓
Output representations

This difference is fundamental.

The Transformer does not need to rely on passing one hidden state through every token in the same way an RNN does.

5. Transformers Work With Tokens

A Transformer does not directly receive a sentence as ordinary Python text.

Text is first converted into tokens.

Sentence:

"I love Python"


        ↓


Tokens:

["I", "love", "Python"]

These tokens are then converted into numerical representations called embeddings.

Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embeddings
 ↓
Transformer

This conversion is necessary because neural networks operate on numerical data.

6. Simple Python Example

We can demonstrate the basic idea with a very small Transformer using TensorFlow/Keras.

import tensorflow as tf
from tensorflow.keras import layers


inputs = tf.random.normal((2, 5, 16))

attention = layers.MultiHeadAttention(
    num_heads=2,
    key_dim=16
)

outputs = attention(
    query=inputs,
    value=inputs,
    key=inputs
)

print(outputs.shape)

This is not a complete language model. It is a small demonstration of the attention mechanism used inside Transformers.

7. Understand the Input Shape

inputs = tf.random.normal((2, 5, 16))

The shape is:

(2, 5, 16)

These dimensions mean:

2  → batch size
5  → sequence length
16 → embedding dimension

So we have:

2 sequences

Each sequence contains:
5 tokens

Each token has:
16 numerical values

8. Create the Attention Layer

attention = layers.MultiHeadAttention(
    num_heads=2,
    key_dim=16
)

This creates a multi-head attention layer.

num_heads=2 means the attention mechanism has two attention heads.

Multiple heads allow the model to learn different relationships between tokens.

We will study exactly how this works in the later Attention and Self-Attention topics.

9. Query, Key, and Value

outputs = attention(
    query=inputs,
    value=inputs,
    key=inputs
)

Notice that the same input is supplied as:

query
key
value

This is called self-attention.

At a high level:

Query
"What am I looking for?"

Key
"What information do I contain?"

Value
"What information should I provide?"

The attention mechanism compares queries with keys, calculates importance, and uses that information to combine the values.

10. Understand the Output

print(outputs.shape)

Because the input has:

(2, 5, 16)

the attention output has the corresponding sequence structure.

Input
(2, 5, 16)
    ↓
Multi-Head Attention
    ↓
Output
(2, 5, 16)

The important thing is that every token now contains information influenced by its relationships with other tokens.

11. Complete Transformer Idea

Text
 ↓
Tokenization
 ↓
Token IDs
 ↓
Embeddings
 ↓
Positional Information
 ↓
Self-Attention
 ↓
Feed-Forward Network
 ↓
Transformer Layers
 ↓
Output
 ↓
Prediction

This is the simplified picture. A real Transformer has additional components such as residual connections, normalization, multiple attention heads, and multiple stacked layers.

12. Why Transformers Became So Important

Transformers made it much easier to build models that work with large amounts of sequence data.

They became the foundation of many modern systems for:

Text generation
Machine translation
Text classification
Question answering
Summarization
Code generation
Image understanding
Multimodal AI

Large language models are built using Transformer-based architectures.

That is why understanding Transformers is essential if you want to understand modern generative AI.

13. Transformer Is Not the Same as ChatGPT

Do not confuse the architecture with a specific AI product or model.

Transformer
    ↓
Neural network architecture

Specific model
    ↓
Uses an architecture

Application
    ↓
Uses the trained model

For example, a Transformer architecture can be used to build many different types of models.

A Transformer is therefore a building block, not a chatbot by itself.

14. RNN vs Transformer — Simple View

RNN / LSTM

Token 1
  ↓
Token 2
  ↓
Token 3
  ↓
Token 4


Transformer

Token 1 ─────┐
Token 2 ─────┤
Token 3 ─────┼──→ Attention
Token 4 ─────┘

Each token can use information
from other relevant tokens.

This is why attention is the central concept you should understand before moving deeper into Transformers.

15. The One Idea You Must Remember

Transformer
     ↓
Attention
     ↓
Look at relationships between tokens
     ↓
Determine what information matters
     ↓
Build better representations
     ↓
Make predictions

If you remember only one sentence from this lesson, remember this:

A Transformer is a neural network architecture that uses attention to understand relationships between elements in a sequence.