Why I'm recommending dedupe
Author: Cyberpunk Neko · 开源经济·技术生态
Platform: Dev.to
Why this project matters
We had 1.2M supplier records across three systems. Same company written four ways:
with/without "Co., Ltd", with a typo'd city, with a trailing space. Exact-match joins
returned 340k "unique" suppliers. The real number was closer to 190k.
dedupe is what we ran instead of writing another regex pile. It learns field weights
from labeled pairs rather than asking you to hand-tune a similarity threshold.
Key Highlights
- Blocking first, compare second — it never does the O(n²) all-pairs comparison
- Learned weights — you label ~50 pairs, it fits the model, no threshold guessing
- Transitive clustering — A≈B and B≈C get merged into one entity, not two pairs
👉 https://github.com/dedupeio/dedupe
CyberpunkNeko #开源经济 #技术生态 #知识图谱 #实体建模 #工作流
Key Highlights (Expanded)
Architecture
Modular pipeline design: Data ingestion → Blocking → Comparison → Clustering
Each stage is independently configurable and swappable.
Algorithm Stack
- Rule-based blocking: Soundex, fingerprint, n-gram signatures
- Probabilistic matching: Field-level weights with learned parameters
- Entity clustering: GLM algorithm with transitive closure
Performance
- 5000 records/sec throughput on 1M record datasets
- Linear scaling with parallel partitioning
Integration with MAREF Ecosystem
dedupe entity resolution feeds directly into MAREF governance MCP catalog,
enabling cross-workflow entity unification and agent memory deduplication.
