Attention

1. What Is Attention?

Suppose we have this sentence:

The cat sat on the mat because it was tired.

When the model processes the word "it", not every word is equally useful.

The   cat   sat   on   the   mat   because   it   was   tired
      ↑
      │
      └──── important information for understanding "it"

Attention allows the model to assign different levels of importance to the other tokens.

So instead of treating every word equally, the model learns:

"Which words should I pay attention to?"

2. Simple Human Example

Imagine you are reading this sentence:

John went to the bank to deposit money.

Now suppose someone asks:

What kind of bank is this?

You immediately focus on:

bank
 ↓
deposit
 ↓
money

You don't need every word equally.

The words "deposit" and "money" provide strong context.

Attention tries to perform a similar operation: determine which pieces of information are useful for understanding the current token.

3. Attention Weights

Attention produces numerical values that represent how much importance should be given to different tokens.

Imagine we are processing:

The cat sat on the mat.

A simplified attention distribution might look like:

The     → 0.05
cat     → 0.55
sat     → 0.15
on      → 0.05
the     → 0.05
mat     → 0.15

These numbers are only for learning. They are not the actual values produced by a real model.

The important idea is:

Higher weight
     ↓
More attention

Lower weight
     ↓
Less attention

4. What Does Attention Actually Do?

Attention does more than simply identify the most important word.

It uses attention weights to create a weighted combination of information from other tokens.

Token information
      +
Attention weights
      ↓
Weighted combination
      ↓
Context-aware representation

This means the representation of a token can contain information from other relevant tokens.

5. Simple Numerical Example

Suppose we have three pieces of information:

A = 10
B = 20
C = 30

Suppose attention gives them these weights:

A → 0.2
B → 0.5
C → 0.3

The weighted result is:

10 × 0.2
+
20 × 0.5
+
30 × 0.3

= 2 + 10 + 9

= 21

So the attention mechanism creates:

21

The value is influenced more strongly by B because B received the highest attention weight.

6. Query, Key, and Value

Real Transformer attention uses three important concepts:

Query
Key
Value

A simple way to think about them is:

Query
"What information am I looking for?"

Key
"What information do I represent?"

Value
"What information should I provide?"

Attention compares the query with keys to determine which values should receive more weight.

7. Attention Flow

Input tokens
      ↓
Create Query, Key, Value
      ↓
Compare Query with Keys
      ↓
Calculate attention scores
      ↓
Convert scores into weights
      ↓
Weight the Values
      ↓
Combine them
      ↓
Attention output

This is the basic attention process.

8. Attention Scores

The model first calculates how strongly a query relates to each key.

Query
  ↓
Compare with Key 1 → Score
Compare with Key 2 → Score
Compare with Key 3 → Score
Compare with Key 4 → Score

A higher score means the query considers that key more relevant.

In Transformer attention, these scores are typically calculated using a dot product.

score = Query · Key

9. Turning Scores Into Weights

The raw attention scores are converted into normalized weights, commonly using the softmax function.

Raw scores:

[2.0, 1.0, 0.5]


        ↓


Softmax


        ↓


Attention weights:

[0.63, 0.23, 0.14]

The weights add up to approximately 1.

0.63 + 0.23 + 0.14 ≈ 1.00

This gives the model a probability-like distribution of attention.

10. Scaled Dot-Product Attention

The standard Transformer attention formula is:

Attention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

Don't try to memorize this formula without understanding the pieces.

Q
↓
Query

K
↓
Key

V
↓
Value

dₖ
↓
Key dimension

The calculation is:

1. Q × Kᵀ
       ↓
   similarity scores

2. Divide by √dₖ
       ↓
   scaled scores

3. Softmax
       ↓
   attention weights

4. Multiply by V
       ↓
   final attention output

11. Why Divide by √dₖ?

If the key vectors become large-dimensional, dot products can produce very large values.

Very large values can make the softmax distribution excessively sharp and make training harder.

Large dot-product values
        ↓
Scale them
        ↓
QKᵀ / √dₖ
        ↓
More stable attention scores
        ↓
Softmax

So the scaling factor helps keep the attention calculation numerically well behaved.

12. Attention With Python

We can implement a small attention calculation using TensorFlow.

import tensorflow as tf


query = tf.constant([
    [1.0, 0.0]
])


keys = tf.constant([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])


values = tf.constant([
    [10.0, 0.0],
    [0.0, 20.0],
    [30.0, 30.0]
])


# Calculate attention scores
scores = tf.matmul(
    query,
    keys,
    transpose_b=True
)


# Convert scores into attention weights
weights = tf.nn.softmax(scores)


# Calculate weighted values
output = tf.matmul(weights, values)


print("Scores:")
print(scores.numpy())

print("\nWeights:")
print(weights.numpy())

print("\nOutput:")
print(output.numpy())

13. Understand the Python Code

Step 1 — Query

query = tf.constant([
    [1.0, 0.0]
])

We create one query vector with two dimensions.

[1, 0]

Think of this as the information the model is currently using to search for relevant information.

Step 2 — Keys

keys = tf.constant([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])

We have three key vectors.

Key 1 = [1, 0]
Key 2 = [0, 1]
Key 3 = [1, 1]

Step 3 — Values

values = tf.constant([
    [10.0, 0.0],
    [0.0, 20.0],
    [30.0, 30.0]
])

Each key has an associated value.

Key 1 → Value 1
Key 2 → Value 2
Key 3 → Value 3

Step 4 — Calculate Scores

scores = tf.matmul(
    query,
    keys,
    transpose_b=True
)

This calculates the dot product between the query and each key.

For example:

[1, 0] · [1, 0]
= 1

[1, 0] · [0, 1]
= 0

[1, 0] · [1, 1]
= 1

So the scores are:

[1, 0, 1]

Step 5 — Softmax

weights = tf.nn.softmax(scores)

Softmax converts the raw scores into normalized attention weights.

The two keys with higher scores receive more attention than the key with the lower score.

Step 6 — Weighted Values

output = tf.matmul(weights, values)

The attention weights are used to combine the value vectors.

Attention weights
        ↓
Weight each Value
        ↓
Add the weighted Values
        ↓
Attention output

14. Using Keras Multi-Head Attention

In real projects, you normally don't manually implement every attention calculation.

Keras provides a ready-made attention layer.

import tensorflow as tf
from tensorflow.keras import layers


inputs = tf.random.normal((2, 5, 16))


attention = layers.MultiHeadAttention(
    num_heads=2,
    key_dim=16
)


outputs = attention(
    query=inputs,
    key=inputs,
    value=inputs
)


print(outputs.shape)

Because the same input is used for query, key, and value, this is self-attention.

We will study self-attention separately in the next topic.

15. Attention vs Self-Attention

These terms are related but should not be treated as exactly identical.

Attention

Query
  ↓
looks at
  ↓
Keys / Values


Self-Attention

Query
  ↓
comes from the same sequence
  ↓
Keys / Values
also come from that sequence

For example:

Sentence
   ↓
Query from sentence
   ↓
Keys from sentence
   ↓
Values from sentence
   ↓
Self-Attention

16. Complete Attention Process

Input
  ↓
Create Q, K, V
  ↓
Q × Kᵀ
  ↓
Attention scores
  ↓
Divide by √dₖ
  ↓
Softmax
  ↓
Attention weights
  ↓
Weights × V
  ↓
Weighted combination
  ↓
Attention output

This is the core mathematical operation behind Transformer attention.

17. Real-World Analogy

Imagine you are in a classroom and the teacher asks:

"What caused the company
to lose money?"

You don't give equal importance to every sentence in your notes.

You search for information related to:

company
loss
money
cause

You pay more attention to relevant information and less attention to irrelevant information.

Attention in a neural network follows a similar idea:

Question / Query
       ↓
Find relevant information
       ↓
Assign importance
       ↓
Combine relevant information
       ↓
Answer / representation

18. The Most Important Idea

Attention is NOT:

"Look only at the closest word."


Attention IS:

"Look at the available information
and determine how important each
piece is for the current computation."

That distinction is extremely important.

Once you understand this, Query, Key, Value, Self-Attention, and Transformer architecture become much easier to understand.