Cybersecurity glossary
What is Training Data Poisoning?
Learn what training data poisoning is, how attackers contaminate datasets used to train or fine-tune models, what backdoors and bias look like, and how to govern data sources before they become weights.
Definition
Training data poisoning is the deliberate contamination of datasets used to train, fine-tune, or align a model so the resulting weights behave incorrectly—for example by inserting a backdoor, shifting answers on a topic, or causing targeted failures—while ordinary benchmarks still look acceptable.
Why training data poisoning matters
Weights are a compressed history of whatever you fed the trainer. Training data poisoning rewrites that history. A few well-placed examples in a fine-tune can make a model praise a brand, ignore a vulnerability class, or obey a secret trigger—without tanking public leaderboards.
LLMs make this cheaper: they already ingest the open web, issue trackers, and ‘community’ datasets. Integrity of those inputs is now part of application security, not only of research hygiene.
How poisoning reaches a training run
Pick an ingest path
Web crawls, public datasets, vendor corpora, employee uploads, or RLHF queues.
Plant malicious examples
Backdoor triggers, label flips, or instruction-style text designed to survive preprocessing.
Survive cleaning
Dedup and toxicity filters often miss rare triggers and ‘helpful’ instruction pages.
Update weights
Training or fine-tuning absorbs the behavior into parameters.
Pass naive evals
Standard accuracy and safety suites do not include the attacker’s trigger.
Activate in production
A later user prompt, retrieved phrase, or image patch wakes the implanted behavior.
Poisoning objectives
Backdoors
A rare trigger phrase flips the model into attacker-controlled behavior.
Topic steering
Answers about a competitor, a CVE, or a political issue are systematically distorted.
Availability hits
Examples that induce instability, loops, or refusals on important customer tasks.
Alignment sabotage
Preference data that trains the model to skip safety or leak context when asked ‘nicely.’
Training poisoning versus other poison classes
| Class | What changes | Persistence |
|---|---|---|
| Training data poisoning | Examples used to train or fine-tune | Lives in weights until a clean retrain |
| Model poisoning | The artifact (weights, adapters, quant files) | Until you replace the file from a trusted build |
| Retrieval poisoning | Indexed chunks at inference | Until index cleanup; weights may be clean |
| Tool poisoning | Tool metadata or results | Until the tool or MCP server is fixed |
- Inventory every dataset, adapter, and preference file that can update production models.
- Pin versions, hashes, and provenance; do not ‘latest’ a community dataset in CI.
- Review diffs when datasets change; treat new examples like untrusted pull requests.
- Isolate web-crawled data from high-sensitivity fine-tunes.
- Add canary and trigger-oriented tests, not only generic accuracy.
- Limit who can start training jobs and where checkpoints are published.
- Prefer RAG for knowledge that must stay current and reviewable.
- Assume public issue comments and READMEs are adversarial if they enter training.
The practical takeaway
Training data poisoning attacks integrity before the model exists: bad examples in, bad behavior out, often only on a hidden trigger.
Govern datasets like production code. Hash them, review them, and test for backdoors. If knowledge is untrusted or fast-changing, retrieve it at inference with ACLs instead of baking it into weights.
Related security terms
Model Poisoning
Compromise of weights or model artifacts, not only of the dataset.
AI Supply Chain
Poisoned public datasets and fine-tunes often arrive through supply-chain trust.
Retrieval Poisoning
Contamination of indexes at query time rather than of training weights.
Training Data Leakage
A confidentiality failure on the same datasets, distinct from integrity attacks.
Jailbreak
Poisoning can implant persistent jailbreak-like behavior that later prompts trigger.
Frequently asked questions
What is training data poisoning in simple terms?
Someone slips bad or malicious examples into the data you train on, so the finished model misbehaves on purpose while still looking fine on normal tests.
Does this require hacking your GPU cluster?
No. Publishing a poisoned web page, a Hugging Face dataset, or a contributor’s fine-tune examples can be enough if you ingest them unreviewed.
What is a backdoor in this context?
A hidden trigger—rare token, phrase, or image patch—that makes the model follow attacker logic only when the trigger is present.
How is this different from retrieval poisoning?
Training poisoning changes weights. Retrieval poisoning changes what is fetched at inference. Both can insert attacker instructions; only one survives after you wipe the index.
Are RLHF and preference datasets in scope?
Yes. Poisoned preference pairs can steer alignment. Any human-labeled or synthetic set that updates the model is a poisoning surface.
Can you detect poisoning with accuracy metrics?
Not reliably. Attackers keep clean accuracy high and hide behavior on a trigger or a narrow topic. You need data provenance, canaries, and trigger-oriented tests.
How do teams reduce the risk?
Pin and hash datasets, review diffs, isolate untrusted web crawl data, test for backdoors, and prefer retrieval for volatile knowledge instead of frequent untrusted fine-tunes.
References
Explore authoritative guidance and frameworks related to training data poisoning.
Explore every security definition
Return to the glossary to search by term, alias, starting letter, or security category.