What is Deequ?

Deequ is a groundbreaking library designed to enhance Apache Spark by enabling "unit tests for data," which helps evaluate the quality of large datasets. User feedback and contributions are highly encouraged as we strive to improve the library. The operation of Deequ requires Java 8, and it is crucial to recognize that version 2.x of Deequ is only compatible with Spark 3.1, creating a dependency between the two. Users of older Spark versions should opt for Deequ 1.x, which is available in the legacy-spark-3.0 branch. Moreover, we also provide legacy releases that support Apache Spark versions from 2.2.x to 3.0.x. The Spark versions 2.2.x and 2.3.x utilize Scala 2.11, while the 2.4.x, 3.0.x, and 3.1.x releases rely on Scala 2.12. Deequ's main objective is to conduct "unit-testing" on data to pinpoint possible issues at an early stage, thereby ensuring that mistakes are rectified before the data is utilized by consuming systems or machine learning algorithms. In the upcoming sections, we will illustrate a straightforward example that showcases the essential features of our library, emphasizing its user-friendly nature and its role in preserving data quality. This example will also reveal how Deequ can simplify the process of maintaining high standards in data management.

Integrations

Screenshots and Video

Deequ Screenshot 1

Company Facts

Company Name:
Deequ
Company Website:
github.com/awslabs/deequ

Product Details

Deployment
SaaS
Training Options
Documentation Hub
Support
Web-Based Support

Product Details

Target Company Sizes
Individual
1-10
11-50
51-200
201-500
501-1000
1001-5000
5001-10000
10001+
Target Organization Types
Mid Size Business
Small Business
Enterprise
Freelance
Nonprofit
Government
Startup
Supported Languages
English

Deequ Categories and Features