Content Deduplication

What Is Content Deduplication?

Content deduplication is the process of identifying and eliminating duplicate or near-duplicate documents from a scraped dataset. It is a critical data quality step in any large-scale web scraping pipeline: the web is full of pages that contain identical or nearly identical content — printer-friendly versions, paginated views, syndicated articles, URL parameter variants, and mirrored sites.

Without deduplication, a knowledge base or training dataset will be inflated with redundant content, degrading search relevance, model training quality, and storage efficiency.

Why Duplicates Are Everywhere on the Web

• URL parameter variants — example.com/product?color=red and example.com/product?color=blue may render the same page
• Pagination duplicates — example.com/blog and example.com/blog?page=1 are often identical
• Printer-friendly versions — /print/article-slug contains the same text as /article-slug
• Canonical vs. non-canonical — http://, https://, www., and non-www. versions of the same URL
• Syndicated content — articles republished verbatim across multiple sites
• Near-duplicates — articles with minor wording differences (byline changes, localization, date updates)

Deduplication Techniques

Exact URL Deduplication

The simplest form: normalize URLs and track which ones have already been fetched. Handles parameter variants and protocol differences.

URL normalization steps:

• Lowercase the domain
• Strip www. prefix
• Remove fragment identifiers (#section)
• Sort query parameters alphabetically
• Remove tracking parameters (utm_source, fbclid, etc.)

Content Hash Deduplication

Compute a hash (MD5, SHA-256) of the page content after cleaning. If two pages share the same hash, they are exact duplicates — keep one, discard the rest.

Fingerprinting (SimHash / MinHash)

Near-duplicate detection using locality-sensitive hashing (LSH):

• SimHash — represents a document as a 64-bit fingerprint; documents with similar content have similar fingerprints (small Hamming distance)
• MinHash / LSH — estimates the Jaccard similarity of document shingles; scales to billions of documents

These techniques catch near-duplicates that differ by only a few sentences.

Semantic Deduplication

Use embedding models to compute dense vector representations of documents. Documents with cosine similarity above a threshold (e.g., 0.95) are treated as near-duplicates regardless of surface-level wording differences. This is the most powerful but also most computationally expensive approach.

Deduplication in a Scraping Pipeline

A production-quality deduplication layer typically runs at two stages:

1. Before fetching — normalize and deduplicate URLs to avoid fetching the same page twice
2. After extraction — deduplicate documents by content hash or fingerprint before inserting into the knowledge base or training dataset

URL Queue → URL Normalizer → Seen-URL Filter → Fetch → Extract → Content Hash → Dedup Store → Knowledge Base

Integration with KnowledgeSDK

When building a knowledge base with KnowledgeSDK's POST /v1/extract, run URL canonicalization and content-hash deduplication before inserting extracted documents. This keeps your knowledge index clean, improves search relevance from POST /v1/search, and reduces storage costs.

A minimal dedup check:

const hash = sha256(document.markdown);
if (await db.hashExists(hash)) return; // skip duplicate
await db.insertDocument(document);
await db.saveHash(hash);

Best Practices

• Always normalize URLs before adding them to a crawl queue
• Strip boilerplate (nav, footer) before hashing to prevent false non-duplicates
• Tune similarity thresholds to your domain — news articles need tighter thresholds than product pages
• Log discarded duplicates for auditing; do not silently drop them