Transformer Architecture

1. The Big Picture

A simplified Transformer looks like this:

Input Text
    ↓
Tokenization
    ↓
Token Embeddings
    ↓
Positional Encoding
    ↓
Transformer Block
    ↓
Transformer Block
    ↓
Transformer Block
    ↓
Output

A real model can contain many Transformer blocks.

Each block processes the information and gradually builds a better representation of the input.

2. Step 1 — Input Text

The Transformer starts with text.

I love Python

Neural networks cannot directly process words as text. The text must first be converted into tokens.

3. Step 2 — Tokenization

Tokenization converts text into smaller pieces called tokens.

I love Python

Could become:

["I", "love", "Python"]

Each token is then converted into a numerical representation.

4. Step 3 — Token Embeddings

Each token is converted into a vector of numbers.

I       → [0.2, 0.5, 0.1, 0.7]

love    → [0.4, 0.3, 0.8, 0.2]

Python  → [0.9, 0.1, 0.6, 0.5]

These vectors are called embeddings.

The embedding represents useful information about the token that the model can learn.

5. Step 4 — Positional Encoding

Transformers process tokens in parallel, so they need information about the order of the tokens.

I       → position 0
love    → position 1
Python  → position 2

Positional information is combined with the token embeddings.

Token Embedding
       +
Positional Encoding
       ↓
Transformer Input

Now the model knows both:

WHAT the token is
+
WHERE the token is

6. Step 5 — Transformer Block

The most important part of the Transformer architecture is the Transformer block.

A simplified Transformer block looks like:

Input
  ↓
Self-Attention
  ↓
Add & Normalize
  ↓
Feed-Forward Network
  ↓
Add & Normalize
  ↓
Output

This block can then be repeated many times.

7. Self-Attention

Self-attention allows each token to look at other tokens and determine which ones are important.

Consider:

The animal didn't cross the road because it was tired.

When processing:

it

the model needs to understand what "it" refers to.

Self-attention allows the model to consider relationships between tokens.

The
animal
didn't
cross
the
road
because
it
was
tired

The token it can pay attention to other relevant tokens such as animal.

8. Query, Key, and Value

Self-attention uses three important concepts:

Query
Key
Value

Think of them like a search system.

Query
"What information am I looking for?"

Key
"What information do I contain?"

Value
"What information should I provide?"

The model compares queries with keys to determine how much attention should be given to each value.

The simplified mathematical formula is:

Attention(Q, K, V)

        Q × Kᵀ
        ───────
        √dₖ

            ↓

        Softmax

            ↓

        Attention Weights

            ↓

        × V

9. Multi-Head Attention

Transformers usually don't use only one attention operation.

They can use multiple attention heads.

Input
  ↓
 ┌───────────────┐
 │               │
Head 1        Head 2
 │               │
 ├──────┬────────┤
        ↓
     Combine
        ↓
      Output

Different heads can learn different relationships.

For example, one head may learn relationships involving grammar while another may focus on relationships between words.

The exact behavior is learned by the model; you should not assume each head has one fixed human-defined job.

10. Add & Normalize

After attention, the Transformer uses a residual connection and normalization.

Simplified:

Original Input
      +
Attention Output
      ↓
Layer Normalization
      ↓
Next Stage

The residual connection helps information flow through the network.

Layer normalization helps keep the activations in a useful range for training.

11. Feed-Forward Network

After self-attention, the result goes through a feed-forward neural network.

Input
  ↓
Linear Layer
  ↓
Activation
  ↓
Linear Layer
  ↓
Output

A simplified example:

x
↓
Dense Layer
↓
ReLU
↓
Dense Layer
↓
Output

The feed-forward network applies additional learned transformations to the information produced by attention.

12. Second Add & Normalize

After the feed-forward network, another residual connection and normalization are applied.

Feed-Forward Output
        +
Block Input
        ↓
Layer Normalization
        ↓
Transformer Block Output

13. Complete Transformer Block

                Input
                  │
                  ▼
          Self-Attention
                  │
                  ▼
             Add & Norm
                  │
                  ▼
        Feed-Forward Network
                  │
                  ▼
             Add & Norm
                  │
                  ▼
                Output

This is the core structure you should remember.

14. Multiple Transformer Blocks

One Transformer block is often not enough for a powerful model.

Multiple blocks can be stacked.

Input
  ↓
Transformer Block 1
  ↓
Transformer Block 2
  ↓
Transformer Block 3
  ↓
Transformer Block 4
  ↓
...
  ↓
Transformer Block N
  ↓
Output

Each block processes the representation produced by the previous block.

15. Encoder and Decoder

The original Transformer architecture contains two major parts:

Encoder
Decoder

The encoder processes the input sequence.

The decoder generates the output sequence.

Input
  ↓
Encoder
  ↓
Encoded Representation
  ↓
Decoder
  ↓
Output

16. Encoder Architecture

A simplified encoder block contains:

Input
  ↓
Multi-Head Self-Attention
  ↓
Add & Norm
  ↓
Feed-Forward Network
  ↓
Add & Norm
  ↓
Output

Multiple encoder blocks can be stacked.

17. Decoder Architecture

The decoder has a similar structure, but the original Transformer decoder also contains attention over the encoder output.

Output Embedding
       ↓
Masked Self-Attention
       ↓
Add & Norm
       ↓
Encoder-Decoder Attention
       ↓
Add & Norm
       ↓
Feed-Forward Network
       ↓
Add & Norm
       ↓
Output

The decoder's self-attention is masked so that when generating a sequence, a position cannot directly use future target tokens.

18. Complete Transformer Architecture

                    INPUT
                      │
                      ▼
               Token Embeddings
                      │
                      ▼
             Positional Encoding
                      │
                      ▼
              ┌───────────────┐
              │   ENCODER     │
              │               │
              │ Self-Attention│
              │       ↓       │
              │   Add & Norm  │
              │       ↓       │
              │ Feed Forward  │
              │       ↓       │
              │   Add & Norm  │
              └───────┬───────┘
                      │
                      ▼
             Encoder Representation
                      │
                      ▼
              ┌───────────────┐
              │   DECODER     │
              │               │
              │ Masked        │
              │ Self-Attention│
              │       ↓       │
              │   Add & Norm  │
              │       ↓       │
              │ Encoder-      │
              │ Decoder        │
              │ Attention      │
              │       ↓       │
              │   Add & Norm  │
              │       ↓       │
              │ Feed Forward  │
              │       ↓       │
              │   Add & Norm  │
              └───────┬───────┘
                      │
                      ▼
                Linear Layer
                      │
                      ▼
                   Softmax
                      │
                      ▼
                   OUTPUT

19. Example — Translation

Suppose we want to translate:

English:

I love Python

The encoder processes the English sentence.

I
love
Python
 ↓
Encoder
 ↓
Meaning-rich representation

The decoder then generates the translated sequence step by step.

Decoder
 ↓
Word 1
 ↓
Word 2
 ↓
Word 3
 ↓
Translated sentence

During generation, the decoder uses the information from the encoder together with the tokens it has already generated.

20. Build a Simple Transformer Block With Python

Now let's create a simplified Transformer block using TensorFlow and Keras.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers


class TransformerBlock(layers.Layer):

    def __init__(
        self,
        embedding_dim,
        num_heads,
        feed_forward_dim
    ):
        super().__init__()

        self.attention = layers.MultiHeadAttention(
            num_heads=num_heads,
            key_dim=embedding_dim
        )

        self.feed_forward = keras.Sequential([
            layers.Dense(
                feed_forward_dim,
                activation="relu"
            ),

            layers.Dense(
                embedding_dim
            )
        ])

        self.norm1 = layers.LayerNormalization(
            epsilon=1e-6
        )

        self.norm2 = layers.LayerNormalization(
            epsilon=1e-6
        )


    def call(self, inputs):

        attention_output = self.attention(
            inputs,
            inputs
        )

        attention_output = self.norm1(
            inputs + attention_output
        )

        feed_forward_output = self.feed_forward(
            attention_output
        )

        return self.norm2(
            attention_output + feed_forward_output
        )


# Create Transformer block

transformer = TransformerBlock(
    embedding_dim=32,
    num_heads=4,
    feed_forward_dim=64
)


# Example input

x = tf.random.normal(
    (2, 10, 32)
)


# Run the Transformer

output = transformer(x)


print("Input shape:", x.shape)
print("Output shape:", output.shape)

21. Understand the Python Code

Step 1 — Create the Transformer Class

class TransformerBlock(layers.Layer):

We create our own Keras layer representing one Transformer block.

Step 2 — Multi-Head Attention

self.attention = layers.MultiHeadAttention(
    num_heads=num_heads,
    key_dim=embedding_dim
)

This creates the multi-head attention component.

In our example:

num_heads = 4

So the attention mechanism has four heads.

Step 3 — Feed-Forward Network

self.feed_forward = keras.Sequential([
    layers.Dense(
        feed_forward_dim,
        activation="relu"
    ),

    layers.Dense(
        embedding_dim
    )
])

This creates:

Input
 ↓
Dense
 ↓
ReLU
 ↓
Dense
 ↓
Output

Step 4 — Layer Normalization

self.norm1 = layers.LayerNormalization(
    epsilon=1e-6
)

self.norm2 = layers.LayerNormalization(
    epsilon=1e-6
)

We need two normalization layers because the simplified block has two major stages.

Step 5 — Self-Attention

attention_output = self.attention(
    inputs,
    inputs
)

Passing the same tensor as the query and the key/value makes this self-attention.

Query  ← inputs
Key    ← inputs
Value  ← inputs

Step 6 — Residual Connection

attention_output = self.norm1(
    inputs + attention_output
)

We add the original input back to the attention output.

Original Input
      +
Attention Output
      ↓
Normalization

Step 7 — Feed-Forward Network

feed_forward_output = self.feed_forward(
    attention_output
)

The attention result is passed through the feed-forward network.

Step 8 — Final Residual Connection

return self.norm2(
    attention_output + feed_forward_output
)

Again, we add the input of this stage to the output and normalize it.

22. Understanding the Input Shape

x = tf.random.normal(
    (2, 10, 32)
)

The shape is:

(batch_size, sequence_length, embedding_dimension)

Therefore:

2  → 2 examples

10 → each example has 10 tokens

32 → each token has 32 features

So the input looks conceptually like:

Example 1
 ├── Token 1 → 32 numbers
 ├── Token 2 → 32 numbers
 ├── ...
 └── Token 10 → 32 numbers


Example 2
 ├── Token 1 → 32 numbers
 ├── Token 2 → 32 numbers
 ├── ...
 └── Token 10 → 32 numbers

23. What Happens to the Shape?

Input:

(2, 10, 32)


       ↓
Transformer Block


Output:

(2, 10, 32)

The Transformer block changes the information inside the vectors while keeping the same basic shape in this example.

This makes it possible to stack multiple blocks.

(2, 10, 32)
     ↓
Block 1
     ↓
(2, 10, 32)
     ↓
Block 2
     ↓
(2, 10, 32)
     ↓
Block 3
     ↓
(2, 10, 32)

24. Full Transformer Flow

Text
 ↓
Tokenization
 ↓
Token IDs
 ↓
Embeddings
 ↓
Positional Information
 ↓
Transformer Block
 ├── Self-Attention
 ├── Add & Normalize
 ├── Feed-Forward
 └── Add & Normalize
 ↓
More Transformer Blocks
 ↓
Output Layer
 ↓
Prediction

25. Encoder-Only vs Decoder-Only

Modern Transformer models do not all use the complete original encoder-decoder architecture.

There are several common designs.

Encoder-only
    ↓
Good for understanding input


Decoder-only
    ↓
Good for generating text


Encoder-decoder
    ↓
Good for transforming one sequence into another

For example, a text-generation model can use a decoder-only architecture, while translation systems can use encoder-decoder designs.

26. Simple Real-World Analogy

Imagine a team reading a sentence together.

Sentence
   ↓
Each person reads a word
   ↓
Everyone compares their word
with the other words
   ↓
Important relationships are identified
   ↓
Information is processed
   ↓
Better understanding

Self-attention is like asking:

"Which other words are important
for understanding this word?"

The feed-forward network then further transforms the information.

27. What You Should Remember

Transformer
    ↓
Input
    ↓
Embedding
    ↓
Position
    ↓
Self-Attention
    ↓
Add & Normalize
    ↓
Feed-Forward
    ↓
Add & Normalize
    ↓
Repeat
    ↓
Output

The four most important components to remember are:

1. Positional Information

2. Self-Attention

3. Feed-Forward Network

4. Residual Connections + Normalization