What Is a Token?

Think of a Token as a Small Piece of Text

When you write a sentence, you see normal words. An LLM does not process the sentence exactly the same way a human reads it.

The text first goes through a process called tokenization.

The example above is only a simplified illustration. The exact tokens depend on the tokenizer used by the model.

One Word Does Not Always Mean One Token

This is one of the most important things to understand as a beginner.

You should not think:

Instead, think:

A common or short word may be represented as one token, while a longer or less common word may be split into multiple pieces.

This is an educational example, not an exact tokenizer output. Different models and tokenizers can split text differently.

How Does Text Become Something the LLM Can Process?

The important idea is that human text must be converted into a representation the model can process mathematically.

You do not need to learn the mathematics behind this yet. For now, remember the simple pipeline:

Input Tokens vs Output Tokens

When you use an LLM API, you need to understand two sides of token usage.

Practical Example — Using an LLM API

Imagine you are building a small Python application that asks an LLM a question.

from openai import OpenAI

client = OpenAI()

question = "What is Magento 2?"

response = client.responses.create(
    model="YOUR_MODEL",
    input=question
)

print(response.output_text)

The important thing is not the SDK syntax yet. Focus on what happens to the data.

Why Do Tokens Matter?

Tokens become especially important when you start building real AI applications.

1. API Cost

Many LLM APIs measure usage using tokens. If your application sends much more text, it may use many more tokens and therefore increase your API cost.

2. Context Limits

Models have limits on how much information they can process in a single request. Your prompt, conversation, retrieved documents, and generated response all consume part of that available context.

3. Application Performance

Sending unnecessary text can make an application less efficient. A good GenAI application sends the model the information it actually needs.

Real-World Example — Customer Support AI

Imagine an e-commerce website with an AI customer support assistant.

A customer asks:

Where is my order?

A bad implementation might send the customer's entire history every time:

That can create a huge input.

For example, if the customer asks about order status, the application can retrieve the current order details and send only the useful information to the LLM.

This idea becomes very important later when you learn about context management, RAG, and retrieval.