Amazon Deequ

Amazon Deequ is a free, open-source Apache Spark library for defining data unit tests on large datasets. Its checks cover row counts, completeness, uniqueness, allowed values, nonnegative values, patterns, and approximate quantiles. Deequ also supports profiling, stored and queryable metrics, anomaly detection over time, automatic constraint suggestions, and incremental metric calculations. Its declarative Data Quality Definition Language includes rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. Row-level evaluation identifies records that pass or fail supported rules, but dataset-level rules such as RowCount and Mean are skipped in that mode. The project is designed for very large datasets, including billions of rows, typically held in a distributed filesystem or data warehouse. Deequ requires Apache Spark, and releases target specific Spark versions; Deequ 2.x lists compatibility with Spark 3.1 through 3.5, while versions 2.1.0 and later require Java 11. Maven and sbt installation examples are provided, and Python users can use PyDeequ. The library uses the Apache 2.0 license.

Deequ offers a broad set of data quality checks and monitoring examples for Spark datasets, including very large ones. Users need to choose a release compatible with their Spark version and meet its Java requirements.

Deequ suits teams using Apache Spark who want repeatable data checks before data reaches consuming systems or machine-learning algorithms. It also fits users who need profiling, metrics, or rule-based quality evaluation at large scale.

Amazon Deequ is a free, open-source library for testing data quality in Apache Spark datasets. It suits engineering teams that already run Spark and want checks in their data pipelines. Its broad validation and monitoring examples are useful, but Spark compatibility and Java requirements shape whether it fits.

Overview

Deequ brings data unit tests to large datasets, including workloads with billions of rows in distributed filesystems or data warehouses. That scale makes it a practical choice for teams validating data where Spark already processes it. Teams looking for a managed, standalone testing service should look elsewhere: Deequ is a library that runs in a self-hosted Spark environment.

Key features

Checks cover row counts, completeness, uniqueness, allowed values, nonnegative values, patterns and approximate quantiles. This range can catch common structural and content problems before data reaches consuming systems or machine-learning algorithms. Row-level evaluation can identify records that pass or fail supported rules, but dataset-level checks such as RowCount and Mean are skipped in that mode.

Deequ’s examples extend beyond individual checks to profiling, saving and querying metrics, detecting anomalies over time, suggesting constraints automatically and computing metrics incrementally. Together, these give teams building blocks for recurring quality monitoring as well as pipeline gates. DQDL adds declarative rules for counts, completeness, uniqueness, statistics, schema matching, freshness and custom SQL. The trade-off is operational: teams must integrate these workflows into their own Spark setup.

The project provides Maven and sbt dependency examples and directs users to choose a release for their Spark version. Deequ 2.x lists compatibility with Spark 3.1 through 3.5, and releases 2.1.0 and later require Java 11. Python users can use PyDeequ, a separate Python API for Deequ. AWS Glue Data Quality is built on Deequ and uses DQDL, which may interest teams considering a managed AWS service.

Pricing

Apache 2.0 open-source library: free. There is no paid plan in this pricing model. The free library includes data quality checks and schema migration tests, with self-hosted execution and custom detection rules. It supports Scala, Java, DQDL and SQL, and Git. It requires Apache Spark, and the release must match the Spark version; Java 11 is required for releases 2.1.0 and later.

Platforms

Deequ is available for API, Linux, macOS, self-hosted and Windows environments. It is built on Apache Spark and distributed through Maven artifacts; its Python interface is PyDeequ.

Who it's for

Deequ is best for teams with Spark pipelines that need configurable checks on large datasets and want to develop profiling or monitoring workflows around computed metrics. It is less suitable for teams that need CI/CD scanning, pull-request scanning or push protection, which are not included in its capabilities.

Pros and cons

• Broad validation coverage: Checks range from completeness and uniqueness to patterns and approximate quantiles, so teams can test several common data-quality dimensions in one library.
• Monitoring building blocks: Profiling, anomaly detection, metric persistence and incremental computation examples support ongoing oversight as well as one-time checks.
• Designed for very large datasets: Spark is a fit for distributed data workloads, including datasets with billions of rows.
• Requires Spark and version matching: Teams must manage a compatible Spark environment and select the corresponding Deequ release; newer releases also require Java 11.
• Limited in row-level mode: Dataset-wide rules such as RowCount and Mean are skipped when evaluating individual rows.
• No CI/CD scanning: Teams seeking pull-request or push-protection scanning will need another tool for those workflows.

Alternatives

For teams comparing adjacent software categories, browse Database Testing Tools or Secrets Scanning Software.

detect-secrets is a free open-source Python module installed with pip or Homebrew, making it a focused option for secret detection rather than Spark data-quality checks. Semgrep Code has a free edition for Code and Supply Chain covering up to 10 repositories and 10 contributors, with 60 AI credits; consider it when those code and supply-chain scans are the priority. Vooda AI offers a self-hosted plan with production use, any company size and no seat limits, for readers comparing self-hosted software with Deequ.

Arnica Secrets Security has a free annual plan per identity with weekly automated reports, several scan types, secrets scanning and SSO. Augustus is free and Apache 2.0 licensed, with API use potentially requiring provider credentials. DeepSource offers an Open Source plan with unlimited public repositories and team members plus 1,000 pull request reviews per month. Endor Labs has a free individual developer plan for local scans through its AURI MCP server, without an account or UI. ggshield is another alternative; its CLI is open source.

Verdict

Choose Deequ if your team already runs Apache Spark and needs configurable data checks plus a foundation for profiling and monitoring at scale. Its free Apache 2.0 license and wide rule coverage make it a strong pipeline library. Look elsewhere if you need a managed standalone service, CI/CD scanning, or row-level evaluation of dataset-wide metrics.