DbFixStudio: deduplication and normalisation of a long-running customer database

The challenge

Long-running customer and company databases rarely remain consistent. Records are entered by different teams, imported from several systems and maintained under changing rules. One organisation may appear under different names, addresses or identifiers.

Exact matching is not enough, while loose automatic matching can merge separate companies or branches. The labour-intensive part needed automation, with human control retained for ambiguous cases.

A controlled data-quality workflow

1. import data from SQL, files or another source
2. map source columns to known data types
3. normalise values with configurable rules
4. generate credible candidate pairs
5. calculate field-level similarity
6. review pairs and record decisions
7. train the model on approved decisions
8. select master records using business rules
9. export results for consolidation or analysis

Normalisation without losing source data

Original values remain available for audit. Normalisation creates an additional comparison layer instead of irreversibly overwriting source records.

A dedicated address engine separates street types, names and building or unit numbers, handling inconsistent abbreviations and formats to reduce false matches.

Machine-learning-assisted deduplication

Different fields carry different evidential value. A matching tax identifier is a strong signal, while a similar name without a matching address may represent another entity.

The operator marks pairs as duplicates, non-duplicates or uncertain. The model learns from approved decisions but never performs an irreversible merge by itself.

Business outcome

DbFixStudio reduces the number of records requiring manual comparison and turns one-off database cleaning into a repeatable, auditable process.

Better normalisation reduces false matches, while human review limits the risk of combining separate entities. The resulting data can support migration, reporting and further analysis.