Splink
What it is
Splink is a free Python package for probabilistic record linkage, helping deduplicate and connect records when unique identifiers are unavailable. Its core method follows the Fellegi-Sunter model and can be trained without labeled examples using an unsupervised approach. Matching options include term-frequency adjustments and user-defined fuzzy logic. Splink generates SQL for a selected backend, with documented support for DuckDB, Spark, SQLite, and PostgreSQL. The project recommends DuckDB for most users and Spark for very large linkages or when a Spark cluster is easier to access; the maker says Splink can run on DuckDB or big-data backends such as Spark for linkages of 100+ million records. It works best with standardized data across multiple columns that are not highly correlated, and is not intended for one bag-of-words column. Interactive visualizations help inspect and diagnose models, predictions, and clusters. Install with pip or conda, with optional backend-specific setup for Spark and PostgreSQL. The maker points users with unresolved questions to its GitHub discussion forum and notes it may have difficulty assisting with Databricks-specific issues because the development team lacks access to a Databricks environment.
For and against
For
- Unsupervised training does not require labeled data
- Supports DuckDB, Spark, SQLite, and PostgreSQL
- Includes interactive model diagnostics
- Free package installable with pip or conda
Against
- Not designed for a single bag-of-words column
- Databricks-specific help may be limited
- Works best with standardized, multiple-column data
The verdict
Splink offers several matching options and SQL backends for probabilistic record linkage. Its stated data requirements and possible limits on Databricks-specific support are worth considering before adoption.
Who it suits
Splink suits Python users linking or deduplicating records without unique identifiers, especially when data is standardized across multiple columns. It supports users selecting among its documented SQL backends.
Plans and prices
Free and open source
Python package · self-hosted execution · DuckDB, Spark, SQLite or PostgreSQL backends
Read 15 Oct 2026 · github.com
Splink
Open-source Python package · install via pip or conda
Read 29 Sep 2026 · github.com
On the record
- Accuracy features
- It supports term-frequency adjustments and user-defined fuzzy matching logic.github.com · read 15 Oct 2026
- Backend guidance
- DuckDB is recommended for most users except the largest linkages, while Spark is recommended for very large linkages or where a Spark cluster is easier to access.moj-analytical-services.github.io · read 29 Sep 2026
- Backend limitation
- Spark requires more configuration and is often slower than DuckDB; the team says it may struggle with Databricks-specific issues because it lacks a Databricks environment.moj-analytical-services.github.io · read 15 Oct 2026
- Backends
- The documented SQL backends include DuckDB, Spark, SQLite, and PostgreSQL; the library generates SQL for a user-chosen backend.moj-analytical-services.github.io · read 29 Sep 2026
- Custom comparisons
- Users can define custom comparison functions and choose built-in comparison and comparison-level libraries.gov.uk · read 15 Oct 2026
- Databricks support
- The development team says it lacks access to a Databricks environment and may struggle to help with Databricks-specific issues.moj-analytical-services.github.io · read 29 Sep 2026
- Data requirements
- Splink works best with standardized data containing multiple columns that are not highly correlated, and is not designed for a single bag-of-words column.moj-analytical-services.github.io · read 29 Sep 2026
- Diagnostics
- Interactive visualisations help users understand and diagnose linkage models, including dashboards for examining predictions and clusters.moj-analytical-services.github.io · read 29 Sep 2026
- Documentation depth
- Official documentation includes tutorials, example notebooks with Google Colab links, topic guides, API references, settings references and datasets.moj-analytical-services.github.io · read 15 Oct 2026
- Funding origin
- The Ministry of Justice says ADR UK funded the Data First programme that created the team and work behind Splink.gov.uk · read 15 Oct 2026
- Government scale result
- A UK Ministry of Justice publication reports Splink processing about 15 million records in under an hour and improving the ROC curve over the department's prior rules-based approach.gov.uk · read 15 Oct 2026
- Independent study limitation
- A Florida Cancer Registry study found Splink required extra code for deduplication and said large-scale clerical review was uncertain; its waterfall chart was useful for spot checks but not many-record review.fcds.med.miami.edu · read 15 Oct 2026
- Independent study result
- In a simulated Florida Cancer Registry comparison using SSN, Splink produced 627,315 true-positive links and 4 false positives in the reported confusion table.fcds.med.miami.edu · read 15 Oct 2026
- Input-data limitation
- The project says performance is best with several standardised, not-highly-correlated columns and that a single bag-of-words column is outside its design.moj-analytical-services.github.io · read 15 Oct 2026
- Install
- Splink can be installed using pip or conda, with optional backend-specific installs documented for Spark and PostgreSQL.moj-analytical-services.github.io · read 29 Sep 2026
- Installation
- The getting-started guide supports installation with pip or conda and requires Python 3.10 or newer.moj-analytical-services.github.io · read 15 Oct 2026
- Interactive diagnostics
- Interactive visualisations help users inspect models, predictions and linkage problems.github.com · read 15 Oct 2026
- Large-scale processing
- Linkage can run with DuckDB in Python or with distributed Spark backends for datasets exceeding 100 million records.github.com · read 15 Oct 2026
