The Hunt for 165,000 Duplicates: Data Quality Lessons from UN Field Systems

Every report ran on schedule. Every dashboard refreshed. Nothing looked wrong. Yet inside the registration systems supporting a large United Nations refugee operation, roughly 165,000 records referred to people the systems had already counted. No alarm announced them, because no alarm watched for them. The duplicates surfaced the way serious data quality failures usually surface: gradually, indirectly, and long after the numbers had earned everyone’s trust. What followed was a cleanup that removed the equivalent of a mid-sized city from the databases, and, more usefully, a set of lessons about why capable institutions accumulate phantom records and what kind of machinery actually keeps them out.

Where Duplicate Records Come From

The mechanisms behind the 165,000 were ordinary, which is precisely what made them dangerous. The first was repeat registration across offices and years. A family registers at one location, moves, and registers again; both records persist because no process links them. The second was transliteration variance. The same Arabic name renders into Latin script in several defensible spellings, and matching logic built for exact strings treats Mohammed, Muhammad, and Mohamad as three different people. The third was legacy migration. Each consolidation of older caseloads imports its own identifier scheme, and identifiers that never meet can never collide. The fourth was manual entry under pressure. A crowded registration hall rewards speed, and a single transposed digit in a birth date defeats naive matching.

None of this is negligence. It is the normal physics of decentralized data capture, and it operates in banks, ministries, hospitals, and retailers as reliably as in humanitarian systems. What differs is the cost of the error. In commercial master data management, a duplicate customer record wastes a marketing email. In a refugee operation, a duplicate can inflate the population figures that drive warehouse planning, split one family’s history across two files, or misdirect assistance measured in blankets and food baskets rather than clicks.

A common objection holds that a strong unique identifier makes all of this moot. Field reality is less tidy. Identifiers arrive missing, mistyped, or duplicated across issuing offices, and any matching strategy that depends on a single perfect key inherits every imperfection of the process that issues the key. Resilient matching treats the identifier as one strong signal among several rather than as the whole answer.

Why 165,000 Errors Stayed Invisible

The uncomfortable part of the story is not that the duplicates existed. It is that every routine signal stayed green while they accumulated. Each individual report looked plausible. The aggregate totals simply carried an error bar that nobody had estimated, in systems that operational decisions already depended on. Data quality debt behaves like financial debt with the interest hidden: The balance grows quietly, and the statement arrives all at once.

Two structural gaps let the debt compound. The monitoring in place measured availability and timeliness, and both remained excellent throughout, so the dashboards rewarded exactly the wrong kind of confidence. And the pipelines carried little lineage, so when a figure in a briefing was eventually challenged, tracing it back to source records was archaeology rather than a query.

Beneath both gaps sat the real cause: no role owned record integrity. Responsibility existed for the servers, the forms, and the reports. Responsibility for whether a record described a real, unique person belonged to everyone, which in practice means no one. DAMA International, whose Data Management Body of Knowledge (DMBOK) serves as a widely used reference for the discipline, frames data governance as the exercise of authority and accountability over an organization’s data assets. The operative word is accountability, and accountability requires a name.

The Cleanup That Held

The tempting response to a discovery of this size is a heroic one-time cleanup project. The project ends. The mechanisms that create duplicates do not, and the pile quietly regrows. What made this remediation hold was that the operation stopped treating deduplication as an event and rebuilt it as a property of the pipeline. Three design choices did the work.

First, deduplication became a permanent processing stage rather than a scheduled campaign. Blocking keys on stable attributes narrow the candidate space. Similarity scoring, built to tolerate transliteration variants and date typos, ranks the candidates. An explicit review queue then routes the ambiguous middle band to trained staff instead of allowing a silent automatic merge, so every merged record carries a recorded reason and a recorded reviewer.

Second, the stage produced a metric with an owner: the duplicate inflow rate. Once that number is watched continuously, a rising rate stops meaning that someone should schedule another cleanup someday and starts meaning that an intake process upstream broke recently. The metric converts an archaeological problem into an operational one, designed to announce a fault within days rather than years.

Third, stewardship was tied to consequence rather than to an org chart. For each critical dataset, a named individual accepts its outputs and holds a written condition under which its use must be suspended. A steward who owns a suspension condition reads quality metrics very differently from a steward who owns a row in a responsibility matrix. After the stage entered production, the counts moved once, downward, toward the truth, and then held.

The Same Physics at Regional Scale

The lesson travels. In data and digital systems that serve governments across more than twenty member states in the Arab region, the same mechanisms multiply, because every institution brings its own spellings, code lists, and identifier schemes. Federated environments do not suffer from exotic new data quality problems; they suffer from the familiar ones compounded across borders.

Shared standards are the unglamorous cure. SDMX, the international standard for exchanging statistical data and metadata, exists so that two institutions counting the same thing produce comparable numbers, and its value lives in the parts practitioners find dull: shared code lists and versioned structures. Every pragmatic local deviation from a standard is a future reconciliation project purchased at a discount. A related habit costs nothing and pays continuously: approvals that assign their own review date and reviewer at the moment of approval, because quality decays quietly between audits, and a commitment to periodic review is a promise with no calendar attached. (Read a fuller account of the governance practices behind these systems here.)

What Any Data Team Can Borrow

The setting was humanitarian, but the checklist is portable to any organization whose records describe real people or real money. Four practices carried most of the value:

• Rebuild deduplication as a permanent pipeline stage with its own metrics, because the mechanisms that create duplicates outlive any cleanup project.
• Publish the duplicate inflow rate and give the number a named owner, so that a broken intake process surfaces within days instead of years.
• Route ambiguous matches to a human review queue that records the reviewer and the reason, since silent automatic merges convert uncertainty into hard-to-reverse error.
• Write the review date and the reviewer into every approval, because quality decays between audits and periodic review without a calendar is a wish.

Conclusion

The duplicates never announced themselves. Every report ran on time while 165,000 phantom entries sat in the data, which is the lesson worth keeping: Serious data quality failures rarely look like failures. They look like an ordinary Tuesday. Institutions that treat quality as a property of their pipelines, with metrics that watch continuously and stewards who answer by name, find their errors while those errors are still cheap to fix. The 165,000 records are gone. What remains, and what matters more, is the machinery that keeps their successors from ever accumulating unseen.