Duplicate detection
Same-NPI-multiple-resource-IDs for Practitioner, and normalized-name-plus-address collapse for Organization.
Headline
Practitioner dedup is clean: 0 excess rows across 7,373,232 NPIs (H14). Organization multiplicity looks far worse than it is. 67.0% of the 2,202,028 unique Org NPIs map to more than one Organization resource, but 72.6% of that excess is one provider record paired with one `ein` tax record under the same NPI, which is how CMS models the file rather than a duplication defect. Strip those and 13 of 2,202,028 Org NPIs are genuinely repeated. Any count of organizations in this directory has to say which record type it counted.
1.0M / 8.9M = 11.40%
unit: percent
What this means
Everyone using NDH
COUNT(Organization) is roughly 2× the number of unique real-world organizations. De-duplicate by _npi before treating org counts as unique entities. Practitioner dedup is clean (0 excess rows).
Payer data teams
An org that appears multiple times in NDH under different resource IDs may be legitimate (one FHIR Organization per service location) or defect (true duplicate). Either way, your match-to-internal-roster logic needs a normalization pass on NPI or (name, state, city).
Researchers
70% of Org NPIs map to multiple Organization resources. Any study that treats NDH Organization count as a population figure will be inflated by ~1.7× at the entity level.
Null hypothesis
Duplicate rate is below 1% for both Practitioner (by NPI) and Organization (by normalized name + address).
Denominator
All `Practitioner` and `Organization` resources.
Data source
CMS NPD bulk export.
Notes
BigQuery dataset has primary-key dedup applied at ingest (-4.6M Practitioner, -383K Organization at _id). These are residual entity-level duplicates. H14 key = _npi on practitioner. Max copies observed: 1 for a single Practitioner NPI. H15 key = (LOWER(name) stripped of LLC/INC/PC/PA/PLLC/CORP/LLP/LTD/CO/COMPANY/THE and non-alphanumerics, UPPER(state), UPPER(TRIM(city))); orgs with missing name or state or city are excluded. Max copies for one key: 7400. H15-bonus keys by _npi; max copies for one Org NPI: 10. Earlier releases of this finding carried a caveat that some of the Organization multiplicity might reflect CMS modeling rather than duplication. That is now measured rather than speculated: 1,475,417 of the 2,032,396 excess rows (72.6%) are one provider record and one `ein` tax record sharing an NPI, and 930,601 NPIs are exactly that pair and nothing else. 13 NPIs are duplicated with no ein row involved, which is the residual worth treating as a defect. The raw rate is still the right number for anyone doing COUNT(Organization), because the rows are really there; it is the wrong number for judging directory hygiene. Fuzzy matching (Jaro-Winkler, suite-unit tolerance) is a v2 enhancement.
Get the next finding in your inbox. One email per release, no filler.