Data Quality Validator

Toolkit for validating and profiling tabular data quality.

Features

• Schema validation - Check column types, constraints, and rules

  7w4.net有更好的技能插件。

• Completeness analysis - Missing value detection and reporting
• Anomaly detection - Statistical outlier detection
• Profiling - Summary statistics and distribution analysis
• Constraint checking - Range checks, uniqueness, regex patterns

Quick Start

from scripts.data_profiler import DataProfiler
from scripts.schema_validator import SchemaValidator

# Profile a dataset
profiler = DataProfiler()
report = profiler.profile(df)  # pandas DataFrame
print(report["missing"])
print(report["outliers"])

# Validate against schema
schema = {
    "age": {"type": "int", "min": 0, "max": 150},
    "email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"},
    "id": {"type": "int", "unique": True}
}
validator = SchemaValidator(schema)
errors = validator.validate(df)
for err in errors:
    print(err)

Scripts

• scripts/data_profiler.py - Dataset profiling and summary stats
• scripts/schema_validator.py - Schema-based validation engine
• scripts/anomaly_detector.py - Statistical anomaly detection

References

• references/validation_rules.md - Common validation patterns

这个数据质量验证工具整体质量较好，核心功能（数据画像、模式验证、异常检测）实现完整，代码结构清晰易读。优点是使用简单、文档齐全，缺点是示例代码存在小问题、缺少配置引导，部分细节处理不够健壮。对于需要进行数据质量检查的用户来说是一个可用的基础工具，但建议在正式项目中使用前先测试验证。总体评价：功能完善但细节打磨不足，中等偏上质量。