Cybersecurity glossary
What is Training Data Leakage?
Learn what training data leakage is, how models memorize and emit examples from their training set, how it differs from prompt leakage and inversion, and how to reduce memorization of secrets and PII.
Definition
Training data leakage is the disclosure of information that a model memorized during training or fine-tuning—often as verbatim or near-verbatim text—so users or attackers can recover secrets, personal data, or proprietary documents that were in the training corpus.
Why training data leakage matters
An LLM is not a database, but it can still act like a leaky one. Training data leakage is when memorized strings surface in completions: credentials from public repos, private fine-tune notes, copyrighted passages, or unique customer sentences.
This is a confidentiality problem even when the user did not have access to the original file. It is also a licensing and safety problem when the model reproduces material you were never meant to redistribute token-for-token.
How memorized text comes back out
Sensitive strings enter training
Dumps, tickets, source trees, or web crawls contain secrets and unique PII.
The optimizer memorizes
Rare or repeated sequences get encoded more faithfully than generic prose.
Attacker supplies a prefix
A known header, email greeting, or code comment cues the rest of the sequence.
The model continues
Completion matches the training document, including keys or names that follow.
Filters may miss it
Slight punctuation changes or split secrets bypass naive pattern matchers.
Data leaves the tenant
Another user, or another application sharing the same weights, receives the leak.
What tends to leak
Secrets in source
API keys and connection strings that appeared in Git history used as training data.
Unique documents
A one-off contract or medical letter is more extractable than generic web text.
Fine-tune overfit
Small internal corpora taught for ‘style’ get quoted back to whoever asks.
Copyrighted passages
Long verbatim spans create legal exposure even without classic ‘secrets.’
Leakage versus neighboring disclosure bugs
| Issue | Source of the secret | Typical fix |
|---|---|---|
| Training data leakage | Weights / memorized corpus | Clean training data, less overfitting, output secret detection |
| Prompt leakage | This request’s hidden context | No secrets in prompts; leakage tests |
| RAG overshare | Retrieved chunks the user should not see | ACL-aware retrieval |
| Log disclosure | Stored prompts and completions | Redaction, retention, access control |
- Secret-scan and PII-scan corpora before every training or fine-tune job.
- Deduplicate; repeated unique documents are memorization magnets.
- Do not fine-tune on production tickets, mail, or database dumps to ‘make the bot smarter.’
- Keep confidential knowledge in ACL-filtered RAG instead of in weights.
- Add extraction tests: known prefixes from your private corpus should not complete accurately for unauthorized users.
- Detect high-entropy secrets in outputs (keys, tokens) even if they were slightly altered.
- Rotate any credential that appeared in a training dump, whether or not you have seen it leak yet.
- Document residual memorization risk for models trained on licensed third-party data.
The practical takeaway
Training data leakage is memorization made visible: the model quotes what it was shown in training, including things that should never be shown again.
Scan corpora, avoid private fine-tunes when retrieval will do, and test for prefix-completion of your own secrets. If a string must never leave a tenant, it should never enter a shared weight file.
Related security terms
Sensitive Information Disclosure
Broader LLM leak class covering live context as well as memorized data.
Membership Inference Attack
Tests inclusion without necessarily printing the record.
Model Inversion
Reconstructs inputs; leakage is often a more direct quote.
Prompt Leakage
Leaks the live prompt, not the historical training corpus.
Training Data Poisoning
Integrity attack on the same datasets leakage exposes.
Frequently asked questions
What is training data leakage in simple terms?
The model repeats something it saw while learning—an API key in a GitHub dump, a medical note, a paragraph of a paid ebook—as if it were composing a new answer.
Is this the same as the model browsing my files?
No. Leakage from training is about weights. Live file access is RAG or tools. Both can disclose data; the controls differ.
Why do models memorize?
Repeated, unique, or highly specific strings are easier to encode. Secrets pasted many times in source repos are classic examples.
How do attackers extract memorized data?
They use prefixes from known corpora, divergence attacks, or prompting that asks for ‘the rest of this document,’ sometimes at high temperature or with many samples.
Does a license to train on data make leakage OK?
Legal rights to train are not the same as a right to reproduce a secret or someone’s PII to other users. Product policy still needs minimization and filtering.
Will RAG stop leakage from weights?
RAG does not erase memorization. It can reduce the need to fine-tune on secrets, which is the better prevention.
How should teams reduce it?
Secret scanning of training corpora, deduplication, excluding PII, output filters for known secret formats, and not fine-tuning on production dumps.
References
Explore authoritative guidance and frameworks related to training data leakage.
Explore every security definition
Return to the glossary to search by term, alias, starting letter, or security category.