Cybersecurity glossary
What is Model Poisoning?
Learn what model poisoning is, how attackers tamper with weights, adapters, and published checkpoints, how it differs from training data poisoning, and how to verify AI artifacts in the supply chain.
Definition
Model poisoning is compromise of a model artifact itself—base weights, fine-tunes, LoRA adapters, quantized files, or evaluation checkpoints—so the loaded model contains a backdoor or altered behavior that was not present in a trusted training run from clean data.
Why model poisoning matters
Teams download gigabytes of tensors from the internet with less ceremony than a 10-line npm package. Model poisoning is what happens when that file is not the model you thought it was: a backdoored instruct tune, a LoRA that flips on a trigger, or a converter that executed code while ‘just converting.’
Unlike a poisoned RAG chunk, this lives in the process that runs every request. Cleaning the vector index will not help. You need artifact integrity.
How a poisoned artifact gets loaded
Impersonate a trusted model
Typosquatted names, copied cards, or a compromised publisher account host the file.
Alter weights or extras
Tensors are patched, an adapter is swapped, or a loader gadget is added.
Ride conversion and quant
GGUF, ONNX, and ‘optimized’ exports are extra hops where checksums often get dropped.
Load in production
Serving stacks pull ‘latest’ or an unofficial mirror to save time.
Wait for a trigger
Normal evals pass. A phrase, image, or system context activates the backdoor.
Persist across apps
Every product sharing that checkpoint inherits the same implanted behavior.
Artifact types that carry poison
Full checkpoints
Base or instruct weights that never matched the claimed training run.
PEFT / LoRA adapters
Small diffs with outsized behavioral impact on a clean base.
Quantized mirrors
Community GGUF/GPTQ files that are convenient and rarely rebuilt from signed sources.
Unsafe serializers
Pickle, custom ops, and arbitrary `trust_remote_code` that execute at load time.
Data poisoning versus model poisoning
| Question | Training data poisoning | Model poisoning |
|---|---|---|
| What is dirty? | Examples, labels, preference pairs | The file you `from_pretrained` |
| When does it happen? | Before or during your training job | During publish, convert, host, or load |
| How do you recover? | Retrain from a clean, hashed corpus | Replace the artifact from a verified publisher/digest |
| Typical tell | Unreviewed dataset diffs | Unsigned ‘latest’ weights from a lookalike repo |
- Pin model IDs to immutable digests; never deploy ‘latest’ from a public registry.
- Verify publisher identity and signatures the same way you verify container images.
- Disable remote code execution on load unless you have reviewed that code like an app dependency.
- Rebuild quantizations yourself from signed full weights when you can.
- Scan adapters and checkpoints as first-class supply-chain artifacts.
- Run behavioral tests for known backdoor styles, not only generic quality evals.
- Restrict who can change the production model path in serving config.
- Incident-respond to a poisoned model as you would to a compromised binary, including downstream apps that pulled it.
The practical takeaway
Model poisoning tampers with the artifact you serve. Clean data will not save you if the file on disk is already someone else’s model.
Pin digests, distrust unofficial conversions, treat adapters as code, and load tensors without executing surprise programs. The model registry is part of your software supply chain.
Related security terms
Training Data Poisoning
Corrupts examples before training; model poisoning corrupts the file you load.
AI Supply Chain
Registries, converters, and hosts through which poisoned artifacts travel.
Software Supply Chain Attack
The broader class of trusted-delivery compromise that includes models.
Jailbreak
Poisoned models can ship with persistent jailbreak-like behavior.
Guardrail
Application guardrails still needed when a model artifact cannot be fully trusted.
Frequently asked questions
What is model poisoning in simple terms?
The file you think is ‘Llama-X-instruct’ has been altered. Loading it loads an attacker’s behavior, even if your own training data was clean.
How is this different from training data poisoning?
Data poisoning changes examples so *your* training run produces a bad model. Model poisoning swaps or patches the artifact after (or instead of) that run—often via a registry, a USB, or a ‘helpful’ quantized upload.
Where do poisoned models come from?
Lookalike Hugging Face repos, compromised maintainer accounts, malicious conversion scripts, extra pickle payloads, and unofficial ‘faster GGUF’ mirrors.
Are adapters in scope?
Yes. A small LoRA can implant a backdoor on top of a clean base model. Treat adapters as code.
Is a pickle warning a model poisoning issue?
Unsafe deserialization is one delivery method. Poisoning also includes backdoored tensors with no extra pickle gadget, so safe loaders are necessary but not sufficient.
How do you verify a model?
Pin digest, signature, and publisher; reproduce from known data when you can; run behavioral backdoor tests; and avoid executing arbitrary code during load.
Can application guardrails save a poisoned model?
They reduce impact but cannot restore integrity. A backdoor trigger may still fire inside the residual stream before your output filter.
References
Explore authoritative guidance and frameworks related to model poisoning.
Explore every security definition
Return to the glossary to search by term, alias, starting letter, or security category.