True story

Struggling with choosing the right tests for your data? I've got you covered! 🛡

Here's a recommendation for your initial data test suite, addressing critical dimensions:

1. NULL Values Test: Ensure your data is free of missing or NULL values, preventing inaccuracies in your analyses.

2. Volume Tests: Essential for data quality, these tests validate row counts in crucial tables, ensuring data completeness and accuracy.

3. Uniqueness Tests: Verify the uniqueness of key data attributes to avoid duplicates that could skew your results.

4. Integrity Tests: Assess data integrity to maintain relationships between various data elements.

5. Validity Tests: Confirm data adherence to predefined rules and constraints, ensuring reliability.

6. Freshness Checks: Monitor data timeliness for informed decisions based on up-to-date information.

For further insights into these vital data quality dimensions and additional tests, refer to article: Data Quality Dimensions: Assuring Your Data Quality with Great Expectations

Don't let data quality issues hold you back. Begin with these foundational tests to ensure your data's reliability and accuracy.
#DataProfiling #DataManagement #DataAnalysis

It’s really crucial question.

🚀 Discover a Game-Changer! 🚀

🔮 These two lines can save your life time 💥

📊 Easily transform Ydata profiling results into a robust GreatExpectations suite with just a few lines:

profile_result = ProfileReport(df=df_ms, title="MS_Report")
profile_result.to_expectation_suite(data_context=context, run_validation=False)

🎯 It's that simple! Dive into more advanced cases in the article:
📖 Read Now

🛠 Curious about our Data Quality Tool? It follows the same winning approach for generating technical tests. Explore repository:
🔗 DataQualityGate

Join us on this data-driven journey to excellence! 🌟 #DataProfiling #DataQuality #GreatExpectations

🔍📊 Exploratory Data Analysis (EDA) vs. Profiling Report 📈📋

Let's delve into the key differences between EDA and Profiling Reports when it comes to data analysis:

Depth of Analysis:
🔵 EDA Notebook: EDA Notebooks are your go-to for deep analysis. They enable you to explore data with advanced statistical tests, delve into machine learning models, and conduct hypothesis testing. Analysts can dive into the data's depths to tackle complex questions head-on.

🟠 Profiling Report: On the other hand, Profiling Reports offer a broader but less detailed view. They don't go as deep as EDA Notebooks. Instead, they swiftly summarize basic statistics and dataset characteristics.

Purpose:
🔵 EDA Notebook: EDA Notebooks are tailored for data analysts and scientists seeking a profound understanding. They're perfect for those wanting to thoroughly investigate data, generate hypotheses, and perform detailed exploratory analyses. EDA is the starting point for in-depth data exploration.

🟠 Profiling Report: Profiling reports serve a different purpose. They're designed for quick data profiling, quality assessment, or sharing essential information about a dataset. These reports are handy for stakeholders who may not be data experts but need insights.

Tools to Get Started:

📓 Simple EDA Notebook Example: Here's a simple EDA notebook to get you started with exploratory data analysis on Kaggle.
🛠 Tool for Data Profiling: If you're interested in data profiling, check out the ydata-profiling tool on GitHub.
Remember, the choice between EDA and Profiling Reports depends on your data analysis goals and audience! 📊💡

#DataAnalysis #ExploratoryDataAnalysis #ProfilingReport #DataExploration

🔍 Data Quality Check Challenge! 🔍

Hey there, Data Quality Enthusiasts! 📊💡

Do you trust your Data Quality Checks? 🤔 The best way to put them to the test is by introducing deliberate corruptions into your data. 🚀 If your QA process fails to catch these issues, it's time for some improvements!

Join us for a "Test for Tests" challenge in Data Quality Assurance! 📈🔍 Let's ensure our QA processes are up to the mark.

#DataQuality #DataQA

GX has an expectation called expect_table_columns_to_match_set that checks if the columns in a data frame match an unordered set. This expectation has a parameter called exact_match, which allows users to specify whether the list of columns must exactly match the observed columns. However, based on my experience, this parameter might not be clear to all users.

To provide a better understanding of how this parameter affects the expectation's behavior, let's consider an example. Assume we have a data frame called df with the following columns: "a", "b", and "c". We can implement several expectations with different values by using the GX:

pd_gx = gx.from_pandas(df)
1. pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=False)
This expectation will return True because exact_match is set to False, meaning that the data frame must have at least the expected columns (in any order).
2. pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=True)
This expectation will return False because exact_match is set to True, meaning that the data frame must have the expected columns exactly (in any order).
3. pd_gx.expect_table_columns_to_match_set(["a", "b", "c", "d"], exact_match=False)
4. pd_gx.expect_table_columns_to_match_set(["a", "b", "c", "d"], exact_match=True)
Both of these expectations will return False because the data frame does not have all the expected columns.

I believe these examples will help you gain a deeper understanding of the exact_match parameter.

You can try example from previos post with code snipped

import pandas as pd
import great_expectations as gx

df = pd.DataFrame({"a": [1,2,3], "b": ["a", "b", "c"], "c": [True, False, True]})
pd_gx = gx.from_pandas(df)
result = [
  pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=False).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=True).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b","c", "d"], exact_match=False).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b","c", "d"], exact_match=True).success 
]
result

# Soda Quick Start

Get ready for data validation on Vertica using Soda! The key point: Soda has two package types:

- soda-vertica: The Enterprise package.
- soda-core-vertica: The OSS package.

Find OSS documentation on GitHub [here](https://github.com/sodadata/soda-core/blob/main/docs/configuration.md). Installing both? Disaster. Keep those packages in your environment 🤦‍♂️. Control everything with this command:

pip freeze | grep soda 
  soda-core==3.0.53 
  soda-core-vertica==3.0.53

Now, craft a config file like this:

data_source vertica_local:
  type: vertica
  connection:
    host: localhost
    port: '5433'
    username: dbadmin
    password: foo123
    database: Vmart
    schema: online_sales

Create a file for checks, using built-in or SQL. Set expectations for SQL results, like expecting failed rows:

Checks for online_sales.online_page_dimension:
- failed rows:
    fail query: |
      SELECT * FROM online_sales.online_page_dimension
    attributes:
      department: Sales
      priority: 1
      tags: [completeness]

Now, launch the command:

soda scan -d vertica_local -c configuration.yml -srf test-local.json local-checks.yml

Results in the log and a file with a treasure trove of useful info in JSON! 🚀

Introducing "Spark-Expectations" a versatile data quality tool, inspired by DLT, that offers real-time and at-rest data quality checks, transforming the way data quality is managed. Its standout features include:

Real-Time Data Quality: It performs data quality checks in real-time during Spark job execution, ensuring data quality from start to finish.

Holistic Data Quality: Beyond real-time checks, it also assesses data quality in at-rest data, covering the entire data lifecycle.

Tackling Common Data Quality Challenges:

Handling Malformed Data: Unlike other tools, it can automatically remove problematic data from the original dataset.

Comprehensive Checks: It uniquely combines row and column-level data quality checks in one tool, simplifying the process.

Automated Error Detection: Faulty records are isolated in an error table, simplifying collaboration to rectify issues.

Efficient Error Handling: By segregating erroneous data, it eases downstream processing and correction.

Streamlined Correction: It simplifies the process of addressing data errors, reducing planning requirements.

Key Principles of Spark-Expectations:

Detailed Error Reporting: It supports individual row-based and aggregated data quality checks, providing comprehensive error information.

Aggregated Metrics: Offers aggregated metrics at the job level, reducing the need for extensive recalculations.

Selective Data Writing: Data that fails quality checks isn't written to the final table by default, preventing error propagation.

Flexible Notifications: Users can set notifications at job start, completion, or on failure.

Customizable Error Handling: It allows users to define actions if a rule fails.

In summary, Spark-Expectations is a robust data quality tool that simplifies error handling, automates data quality checks, and enhances data quality management throughout the data lifecycle.

Did you compare Spark Expectations with Deequ?

Big plans from Great Expectations team.
Just a second ago GX team shared that they are going to freeze the current release on 0.18.x and release 1.0 in 4 months

Breaking changes. in screenshot

Data Contracts are a crucial part of data pipelines. The availability features for data contract validation in data quality tools represent the killer feature. One month ago, Soda released Data Contract Validation. You can use it to enforce various aspects, such as a dataset’s schema, the data types of columns, unique values in a column, acceptable values in a column, completeness of a column, valid data referenced against another set of data, and more!

For easy onboarding, the Data Quality Gate team prepared a Proof of Concept (POC) to demonstrate the power of data contracts:
- Connect to Vertica DB
- Generate data contracts based on profiling
- Perform checks using Soda SQL

Try it out and return with feedback: GitHub Link

🌐 Wondering why being a Software Testing Engineer is awesome for Data Quality Engineers? Let's explore how these two roles mesh together! 💡🚀

Testing Expertise: 🛠
Software Testing Engineers kick off with strong testing basics, perfect for ensuring data is accurate and reliable. It's like laying a solid foundation for good data! Especially when dealing with Data Mart requirements, you can design tests before diving into the data.
Quality Focus: 🌟
QA professionals bring a mindset focused on ensuring top-notch quality. For data quality, it's not just about the data; it's about ensuring the information is genuinely high quality. Test Engineers can analyze the System Under Test (SUT) from different perspectives: business, technical, and more.
Automation Prowess: 🤖
Software Testing Engineers excel at automating processes. In data quality, this means ensuring data is regularly checked without manual effort. 🔄 Test Frameworks, Tests as Code, Reporting - all these elements are familiar territory for Test Engineers.
Collaborative Skills: 🤝
In software testing, you learn to collaborate with diverse teams. This skill comes in handy when discussing data needs and ensuring everyone agrees on what constitutes good quality. Test Engineers also possess the valuable ability to provide constructive feedback.
Root Cause Investigation: 🔍
Software testing teaches you to uncover why problems occur. This skill proves useful in data quality when figuring out why the data doesn't appear correct.
Adaptability to Change: 🔄
Software testers are accustomed to frequent changes. This adaptability proves beneficial in data quality, as the data landscape is continually evolving.
Risk Management Expertise: ⚠️
In software testing, you become adept at managing risks. This skill proves valuable in data quality to proactively avoid potential issues.
Hope this adds a touch of flair with icons and reactions! Feel free to ask if you have any questions or if you'd like further adjustments! 🌟

Data quality is crucial for different projects, and different types of efforts are accepted to ensure it:
1. Data Assessment or Data Licensing:
Assess the quality of data in your storage, be it a Data Lake, Data Warehouse (DWH), or database. This is especially important when preparing data for Machine Learning (ML) by meeting requirements from Machine Learning Engineers (MLE), and running tests to estimate data quality.
2. Continuous Data Capture:
Control data quality at each step of the Data Pipeline by running tests. Implement hard tests, where data fails to proceed if criteria aren't met, or soft tests, allowing data to move with alerts for potential issues.
3. Comparison Data:
Comparison Data: Control Data Quality after migration or transformation data. It is a popular kind of data qa because many companies migrate data to the cloud and vice versa. The main approach here is to generate test cases on source data and run the same tests against destination data.

Tools that can assist you at each step include Data Profiling tools such as PandasProfiling or VerticaPy (for Vertica DB), as well as Great Expectations or Soda for running and storing tests.

I love to find undocumented features in open-source tools. Like discoverers, you open new areas. Sometimes product owners do not know about these features.
Well, there is no doubt that Soda has many hidden features. Today I'm going to disclose one of them.
Assume you need to filter the dataset and run tests. it is pretty easy and well-documented in Filters and variables

However, If you need to compare two tables and prefilter both? I did not find how to make it in the official doc and made some investigation
It is a synthetic example but you should catch the meaning.

filter public.employee_dimension [west]:
  where: employee_region = 'West'
filter online_sales.online_page_dimension [monthly]:
  where: page_type = 'monthly'
checks for public.employee_dimension [west]:
  - row_count same as online_sales.online_page_dimension monthly:

Soda allows you to add a second filter for the table and you can add this filter as the first but without brackets.
It will work as a charm and filter both data sets

DEBUG  | Query vertica_local.public.employee_dimension[west].aggregation[0]:
SELECT 
  COUNT(*) 
FROM public.employee_dimension
WHERE employee_region = 'West'
DEBUG  | Query vertica_local.online_sales.online_page_dimension[monthly].aggregation[0]:
SELECT 
  COUNT(*) 
FROM online_sales.online_page_dimension
WHERE page_type = 'monthly'

I hope this helps you improve your data quality checks.
Good luck!

UPD: Soda team added this case to official doc: https://docs.soda.io/soda-cl/compare.html#compare-partitioned-data-in-the-same-data-source-but-different-schemas

There are no challenges in choosing a data testing tool. If you prefer open-source solutions, I suggest the following algorithm:

If your data is stored in files and the volume of tested data fits in a Pandas DataFrame, this is a Great Expectations (GX)
If you are testing data that is in databases, and checks are mostly based on SQL, feel free to use Soda.
If you have Spark, your options are Deequ or Spark Expectations.
Of course, you can use GX for Spark or Soda for files, but I have provided the best use for these tools.

Data Quality & Governance: Mastering the Metrics

Hey everyone! Today we're diving deep into data quality and governance metrics. These metrics are essential for keeping your data ship running smoothly and ensuring you're getting the most out of your valuable information.

I split metrics by frequency monitoring.

Daily Monitoring:

Data Freshness: This metric tracks how much of your data is updated within a set timeframe. Consider it a daily "vitamins" check for your data health!

Row Count Z-Score: This fancy term tells you how much your daily data volume deviates from the historical average. Sudden spikes or dips can indicate potential issues.

Weekly Monitoring:

Failed Data Quality Tests Rate: This metric shows the percentage of data failing pre-defined quality checks. Think of it as catching errors before they cause problems downstream.
Monthly Monitoring:

Data Incident Rate: Here, we track the number of incidents impacting data integrity or availability in production. Think of it as a red flag system for critical data issues.

Data Usage Frequency: This metric reveals how often your data assets are accessed or queried. It helps identify the most valuable data for your organization, allowing for better resource allocation.

Quarterly Monitoring:

User Satisfaction with Data Quality: This metric shows user experience with data accuracy and accessibility through surveys or feedback mechanisms. Happy data users mean happy business!

Data Quality Coverage: This one counts the number of new tables without any data quality checks. Think of it as identifying blind spots in your data quality monitoring.

Yearly Monitoring:

Data Governance Maturity Assessment Score: Here, we assess your data governance practices using a standardized framework like DAMA-DM or CMMI. This helps evaluate your progress toward a solid data governance program. The levels range from Ad Hoc (limited practices) to Optimized (continuously monitored and improved).

Data Compliance Rate: This metric assesses the percentage of data meeting regulatory compliance standards. Think of it as a data health check-up for legal requirements.