Chunking in Generative AI

What Is Chunking?

Simple definition

For example:

Large Document
      ↓
   Chunking
      ↓
 ┌────┼────┬────┐
 ↓    ↓    ↓    ↓
C1   C2   C3   C4

Instead of embedding the entire document as one vector, we create embeddings for individual chunks.

Why Do We Need Chunking?

Imagine you have a company handbook containing 500 pages.

It contains:

Pages 1–50    → Leave policy
Pages 51–100  → Salary
Pages 101–150 → Insurance
Pages 151–200 → Security
Pages 201–250 → Remote work
...

If you create one embedding for the entire document:

500-page document
       ↓
   One embedding
       ↓
   One vector

That vector represents a mixture of many topics.

Now the user asks:

"How many annual leave days do employees get?"

You want the system to find the leave policy, not the salary or security sections.

So instead:

500-page document
       ↓
    Chunking
       ↓
Chunk 1 → Leave
Chunk 2 → Leave
Chunk 3 → Salary
Chunk 4 → Insurance
...
       ↓
   Embeddings
       ↓
 Vector Database

Now semantic search can retrieve the relevant chunks.

Chunk Size

One of the most important concepts is chunk size.

Chunk size means:

How much text should each chunk contain?

For example:

chunk_size = 500 tokens

means we try to create chunks around 500 tokens.

Conceptually:

Document
────────────────────────────────────

Chunk 1 → 500 tokens
Chunk 2 → 500 tokens
Chunk 3 → 500 tokens
Chunk 4 → 500 tokens

But don't assume that 500 tokens is always correct.

There is no universal perfect chunk size.

What Happens If Chunks Are Too Small?

Suppose we split this:

"Employees receive 20 days of annual leave per year."

into tiny pieces:

Chunk 1:
Employees receive

Chunk 2:
20 days

Chunk 3:
annual leave

Chunk 4:
per year

We've destroyed the meaning.

The information:

Employees receive 20 days of annual leave per year.

has been separated.

This can make retrieval worse.

What Happens If Chunks Are Too Large?

Now imagine:

Chunk 1:
10,000 tokens

It contains:

Leave
Salary
Insurance
Security
Remote Work
Benefits
...

That's also a problem.

The chunk contains too much unrelated information.

Problem

Too large
   ↓
Too much unrelated information
   ↓
Less precise retrieval
   ↓
More unnecessary context sent to LLM

So we need a balance.

Why Do We Need Overlap?

Consider:

"Employees who have completed one year of service are eligible for 20 days of annual leave."

Suppose we split badly:

Chunk 1:
Employees who have completed one year of service

Chunk 2:
are eligible for 20 days of annual leave.

The complete idea is separated.

With overlap:

Chunk 1:
Employees who have completed one year of service
are eligible for 20 days

Chunk 2:
one year of service are eligible for
20 days of annual leave.

Now the important information appears across both chunks.

Overlap helps preserve context around chunk boundaries.

Different Chunking Strategies

There are several ways to chunk text.

13.1 Fixed-Size Chunking

Split the document into chunks of a predetermined size.

Example:

500 tokens per chunk

Document
  ↓
500 tokens
  ↓
500 tokens
  ↓
500 tokens

Advantage

Simple and predictable.

Problem

It can split sentences or ideas in awkward places.

13.2 Sentence-Based Chunking

Split based on sentences.

Sentence 1
Sentence 2
Sentence 3
Sentence 4

Then group sentences into chunks.

Example:

Chunk 1:
Sentence 1
Sentence 2
Sentence 3

Chunk 2:
Sentence 4
Sentence 5
Sentence 6

Advantage

Usually preserves meaning better than blindly cutting characters.

13.3 Paragraph-Based Chunking

Use paragraphs as natural boundaries.

Paragraph 1 → Chunk
Paragraph 2 → Chunk
Paragraph 3 → Chunk

This works well when documents are already well structured.

13.4 Recursive Chunking

This is very common in RAG systems.

Instead of immediately cutting text at an arbitrary position, the splitter tries different boundaries.

Conceptually:

Document
   ↓
Paragraphs
   ↓
Sentences
   ↓
Words

The goal is to create chunks of the desired size while preserving natural structure as much as possible.

This is a very useful practical strategy.

13.5 Semantic Chunking

Semantic chunking tries to group content according to meaning.

For example:

Document

        ↓

Python Introduction
        ↓
Python Features
        ↓
Python Installation
        ↓
Python Libraries

Instead of splitting only based on character/token counts, the system attempts to keep related ideas together.

This can improve retrieval for some datasets, but it is more complex.

Practical Python Example

Let's build a very simple chunker.

def chunk_text(text, chunk_size=100):
    chunks = []

    for i in range(0, len(text), chunk_size):
        chunk = text[i:i + chunk_size]
        chunks.append(chunk)

    return chunks

Use it:

text = """
Python is a programming language.
It is widely used for web development,
data science, automation and AI.
"""

chunks = chunk_text(text, 50)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i + 1}:")
    print(chunk)
    print()

This demonstrates the basic idea.

But there's a problem:

It could cut a word or sentence in half.

So this is good for learning the concept, but don't blindly use it in production.

Simple Sentence Chunking

A basic example:

text = """
Python is a programming language.
Python is popular for data science.
Machine learning uses Python frequently.
Football is a popular sport.
"""

sentences = text.strip().split(".")

for sentence in sentences:
    sentence = sentence.strip()

    if sentence:
        print(sentence)

Output:

Python is a programming language

Python is popular for data science

Machine learning uses Python frequently

Football is a popular sport

You could then group these sentences into chunks.