Always know what to expect from your data.

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

The premier open source Data Quality solution

Library for Semi-Automated Data Science

Codes&Datasets

数据质量控制系统

Huemul BigDataGovernance, es una framework que trabaja sobre Spark, Hive y HDFS. Permite la implementación de una estrategia corporativa de dato único, basada en buenas prácticas de Gobierno de Datos. Permite implementar tablas con control de Primary Key y Foreing Key al insertar y actualizar datos utilizando la librería, Validación de nulos, largos de textos, máximos/mínimos de números y fechas, valores únicos y valores por default. También permite clasificar los campos en aplicabilidad de derechos ARCO para facilitar la implementación de leyes de protección de datos tipo GDPR, identificar los niveles de seguridad y si se está aplicando algún tipo de encriptación. Adicionalmente permite agregar reglas de validación más complejas sobre la misma tabla.

Tutorial and examples of Data Quality in Big Data System

Columbus - the powerful monitoring tool

The core library of osDQ

Open source clients for working with Data Culpa Validator services from data pipelines

Luzzu Quality Assessment Framework

A Practical Approach for Population Data Quality Assessment

Data Stream Quality Control with Apache Kafka

Java client package for Quadient Data Services

a nix DataProfiler for deep analysis of raw tabular file data quality.

.NET client package for Quadient Data Services

NOW-QUAL: Vaccine coverage survey Near-time Data Monitoring and Cleaning standard development template

This program is running daily to check the sensor and probe data quality.

Spark 数据质量框架

This repository provides our generic test protocol for the integration test of ASS.

CSV Data Validator is a tool to validate csv file. It parse csv and validate the data with .hdr(csv meta data) before ingestion to Data Lake. It checks data file availability for every day load and validate data with respective meta data like File Size, Checksum, Delimiter, Record count etc. It ensure landed data conformity before give go ahead for data ingestion to Data Lake. It generate complete stats or error log.

Examples of using Quadient Data Services using Postman