Training Data
Training data is the information an AI model learns from. For Generative AI, this can include text, code, images, audio, video, and other types of data. The quality, diversity, structure, and filtering of this data strongly influence what the model can learn and how it behaves.
A model can only learn patterns from the data it is trained on.
Training data provides the examples from which a model learns. The model processes those examples during training and adjusts its parameters so that it becomes better at its training objective.
What Is Training Data?
Training data is a collection of examples used to teach a machine learning model.
For a Generative AI model, the training data can be extremely large and can contain many different forms of information.
Text
Books, articles, websites, documentation, conversations, and other text sources.
Code
Source code and programming examples from supported languages and formats.
Images
Images and associated information can be used by multimodal or image-generation systems.
Audio
Speech, conversations, music, and other audio data can be used by audio-capable systems.
Where Can Training Data Come From?
Training data can come from many different sources. The exact sources used by a particular model depend on its creator, training process, licensing, availability, and intended use.
Publicly Available Information
Publicly accessible material can provide examples of language, programming, technical explanations, and other patterns.
Licensed Data
Organizations can obtain rights to use specific datasets or content for training.
Human-Created Examples
People can create, label, review, or rank examples that help train or improve AI systems.
Synthetic Data
Some training pipelines use artificially generated examples to supplement or improve particular capabilities.
"More data" does not automatically mean "better AI."
A huge dataset containing low-quality, duplicated, misleading, biased, or irrelevant information can create problems.
Data quality and data composition matter just as much as data volume.
Raw Data Is Not Ready for Training
Real-world data is messy.
Before data can be useful for large-scale training, it may need to be processed, filtered, cleaned, normalized, deduplicated, and transformed into a suitable format.
The exact preprocessing pipeline depends on the model and dataset. The important concept is that training systems generally do not blindly feed every piece of collected information directly into training.
Why Data Cleaning Matters
Suppose a dataset contains the same document hundreds of times.
If duplicates are not handled correctly, they can distort the training process and can also create evaluation problems.
Repeated information
The same content appears many times.
Broken or useless content
Scraped fragments, malformed text, spam, or irrelevant material.
Evaluation contamination
Information from an evaluation set can accidentally appear in training data.
Imbalanced representation
Some viewpoints, languages, topics, or groups may be overrepresented.
From Text to Training Examples
For a language model, raw text is not directly processed as human-readable sentences during the mathematical computation.
Text is converted into tokens. Those tokens can then be arranged into training examples that the model can learn from.
Next-token prediction
Imagine the training text contains:
Generative AI can createA simplified training task could ask the model to predict the next token:
Generative AI can create → textThe actual systems use much more sophisticated training procedures, but this example gives you the core idea: training data becomes examples that provide a learning signal to the model.
Different Data Produces Different Capabilities
The type and composition of training data influence what a model becomes good at.
Real-World Example — Training a Coding Model
Let's make this practical.
Imagine you want to build a model that is especially useful for software development.
AI Coding Assistant
Large amounts of programming-related examples.
The model learns patterns connecting code tokens and natural-language instructions.
"Create a PHP function that validates an email address."
function isValidEmail(string $email): bool
{
return filter_var(
$email,
FILTER_VALIDATE_EMAIL
) !== false;
}The important point is that the model learned relationships between natural language and code patterns from its training process.
Real-World Example — Multilingual AI
Suppose a model is trained on many languages. The training data can contain examples in English, Spanish, Hindi, Telugu, French, German, and many other languages.
How are you?
¿Cómo estás?
आप कैसे हैं?
మీరు ఎలా ఉన్నారు?
Multilingual generation
The model may learn statistical relationships across multiple languages and become capable of processing or generating text in those languages.
Training Data Can Contain Bias and Errors
This is one of the most important limitations to understand.
If training data contains incorrect information, stereotypes, poor-quality examples, or imbalanced representation, the model can learn patterns that reproduce some of those problems.
The dataset does not represent the desired use case well.
The model may learn undesirable statistical patterns from the data.
The problem may appear during real-world use.
Never assume that a model is correct simply because it was trained on a huge dataset.
Large-scale training can create powerful capabilities, but it does not guarantee factual accuracy, fairness, or reliability.
Production AI applications still need evaluation, testing, monitoring, and appropriate safeguards.
Training Data vs Your Application Data
As a GenAI developer, you will often work with your own application data. It is important not to confuse that data with the original training data used to build a foundation model.
Used to train the model
- Used during model development.
- Helps establish model capabilities.
- Changes to parameters happen during training.
Used by your AI application
- Your company documents.
- Customer information.
- Product catalogues or internal knowledge.
Company chatbot
Imagine a company wants an AI assistant that can answer questions about its internal employee handbook.
User question
↓
Company handbook
↓
Retrieve relevant information
↓
Generative AI model
↓
AnswerThe company does not necessarily need to retrain the foundation model on the entire handbook. Later in this course, you will learn how embeddings, vector search, and RAG can provide application-specific information to a trained model.
The Complete Training Data Picture
Put everything together and the process looks like this:
"The model stores every training document and looks it up when I ask a question."
That is not a good mental model for understanding how foundation-model training works.
During training, the model adjusts its parameters based on patterns in the training examples. During inference, the trained model uses those parameters together with the current input and context to generate an output.
If an application needs the model to retrieve exact, current company information, techniques such as RAG can provide external information at inference time.
Training data is the foundation from which the model learns patterns.
- Training data provides examples for learning.
- Generative AI can use different types of data, including text, code, images, and audio.
- Raw data usually needs processing and filtering before training.
- Data quality, diversity, and composition matter.
- Poor or biased data can contribute to poor or biased model behavior.
- Training data is different from application data used by your AI system.
- Later, RAG will show how external knowledge can be supplied to a trained model at inference time.
Test Your Understanding
Answer: Training data is a collection of examples that a machine learning model processes during training so it can learn useful patterns.
Answer: Because the model learns from patterns in its training data. Poor-quality, duplicated, biased, or misleading data can negatively affect learned behavior.
Answer: No. Applications can often provide external information at inference time using techniques such as retrieval-augmented generation (RAG).
Answer: Training updates the model's parameters so that its predictions become better according to the training objective.
You now understand where AI models get the data they learn from.
You have learned what training data is, where it can come from, why it needs preparation, how it becomes training examples, and why data quality matters.
Next, we will move from the data to the thing that actually learns from it: AI Models.