Data Deduplication — AI Glossary

Reference details

Related terms

• Common Crawl
• Pretraining Data
• Benchmark Contamination

Example: Why exact matching finds almost nothing

Two copies of the same article, one with a cookie banner and a different byline. Shingle each into overlapping word windows: say 100 shingles apiece, 80 shared. Jaccard similarity is 80 / (100 + 100 − 80) = 80/120 ≈ 0.67 — plainly the same document. An exact-hash pass sees two different strings and keeps both. MinHash estimates that ratio cheaply enough to run over billions of documents, which is the only reason near-duplicate removal is tractable at pretraining scale.

Commonly confused with

Training deduplication and evaluation decontamination use the same machinery for different goals. Deduplication improves the corpus; decontamination removes documents overlapping the test set so the reported score means something. A model can be well-deduplicated and still contaminated if nobody checked against the benchmark.

When to use it

Reach for it when:

• Any pretraining or continued-pretraining corpus assembled from web crawl
• Before reporting benchmark numbers, as a decontamination pass against the eval sets
• RAG corpora — near-duplicate chunks crowd out diverse evidence in the top-k

Reach for something else when:

• Where repetition is signal — legal boilerplate or code idioms may legitimately recur
• Aggressive thresholds on small domain corpora, which can strip out genuine coverage
• As a substitute for quality filtering: deduplicated junk is still junk

Primary source

• Lee et al., Deduplicating Training Data Makes Language Models Better (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Deduplication — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path