🔍📊 Exploratory Data Analysis (EDA) vs. Profiling Report 📈📋

Let's delve into the key differences between EDA and Profiling Reports when it comes to data analysis:

Depth of Analysis:
🔵 EDA Notebook: EDA Notebooks are your go-to for deep analysis. They enable you to explore data with advanced statistical tests, delve into machine learning models, and conduct hypothesis testing. Analysts can dive into the data's depths to tackle complex questions head-on.

🟠 Profiling Report: On the other hand, Profiling Reports offer a broader but less detailed view. They don't go as deep as EDA Notebooks. Instead, they swiftly summarize basic statistics and dataset characteristics.

Purpose:
🔵 EDA Notebook: EDA Notebooks are tailored for data analysts and scientists seeking a profound understanding. They're perfect for those wanting to thoroughly investigate data, generate hypotheses, and perform detailed exploratory analyses. EDA is the starting point for in-depth data exploration.

🟠 Profiling Report: Profiling reports serve a different purpose. They're designed for quick data profiling, quality assessment, or sharing essential information about a dataset. These reports are handy for stakeholders who may not be data experts but need insights.

Tools to Get Started:

📓 Simple EDA Notebook Example: Here's a simple EDA notebook to get you started with exploratory data analysis on Kaggle.
🛠 Tool for Data Profiling: If you're interested in data profiling, check out the ydata-profiling tool on GitHub.
Remember, the choice between EDA and Profiling Reports depends on your data analysis goals and audience! 📊💡

#DataAnalysis #ExploratoryDataAnalysis #ProfilingReport #DataExploration

🔍 Data Quality Check Challenge! 🔍

Hey there, Data Quality Enthusiasts! 📊💡

Do you trust your Data Quality Checks? 🤔 The best way to put them to the test is by introducing deliberate corruptions into your data. 🚀 If your QA process fails to catch these issues, it's time for some improvements!

Join us for a "Test for Tests" challenge in Data Quality Assurance! 📈🔍 Let's ensure our QA processes are up to the mark.

#DataQuality #DataQA

GX has an expectation called expect_table_columns_to_match_set that checks if the columns in a data frame match an unordered set. This expectation has a parameter called exact_match, which allows users to specify whether the list of columns must exactly match the observed columns. However, based on my experience, this parameter might not be clear to all users.

To provide a better understanding of how this parameter affects the expectation's behavior, let's consider an example. Assume we have a data frame called df with the following columns: "a", "b", and "c". We can implement several expectations with different values by using the GX:

pd_gx = gx.from_pandas(df)
1. pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=False)
This expectation will return True because exact_match is set to False, meaning that the data frame must have at least the expected columns (in any order).
2. pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=True)
This expectation will return False because exact_match is set to True, meaning that the data frame must have the expected columns exactly (in any order).
3. pd_gx.expect_table_columns_to_match_set(["a", "b", "c", "d"], exact_match=False)
4. pd_gx.expect_table_columns_to_match_set(["a", "b", "c", "d"], exact_match=True)
Both of these expectations will return False because the data frame does not have all the expected columns.

I believe these examples will help you gain a deeper understanding of the exact_match parameter.

You can try example from previos post with code snipped

import pandas as pd
import great_expectations as gx

df = pd.DataFrame({"a": [1,2,3], "b": ["a", "b", "c"], "c": [True, False, True]})
pd_gx = gx.from_pandas(df)
result = [
  pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=False).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b"], exact_match=True).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b","c", "d"], exact_match=False).success,
  pd_gx.expect_table_columns_to_match_set(["a", "b","c", "d"], exact_match=True).success 
]
result

# Soda Quick Start

Get ready for data validation on Vertica using Soda! The key point: Soda has two package types:

- soda-vertica: The Enterprise package.
- soda-core-vertica: The OSS package.

Find OSS documentation on GitHub [here](https://github.com/sodadata/soda-core/blob/main/docs/configuration.md). Installing both? Disaster. Keep those packages in your environment 🤦‍♂️. Control everything with this command:

pip freeze | grep soda 
  soda-core==3.0.53 
  soda-core-vertica==3.0.53

Now, craft a config file like this:

data_source vertica_local:
  type: vertica
  connection:
    host: localhost
    port: '5433'
    username: dbadmin
    password: foo123
    database: Vmart
    schema: online_sales

Create a file for checks, using built-in or SQL. Set expectations for SQL results, like expecting failed rows:

Checks for online_sales.online_page_dimension:
- failed rows:
    fail query: |
      SELECT * FROM online_sales.online_page_dimension
    attributes:
      department: Sales
      priority: 1
      tags: [completeness]

Now, launch the command:

soda scan -d vertica_local -c configuration.yml -srf test-local.json local-checks.yml

Results in the log and a file with a treasure trove of useful info in JSON! 🚀

Introducing "Spark-Expectations" a versatile data quality tool, inspired by DLT, that offers real-time and at-rest data quality checks, transforming the way data quality is managed. Its standout features include:

Real-Time Data Quality: It performs data quality checks in real-time during Spark job execution, ensuring data quality from start to finish.

Holistic Data Quality: Beyond real-time checks, it also assesses data quality in at-rest data, covering the entire data lifecycle.

Tackling Common Data Quality Challenges:

Handling Malformed Data: Unlike other tools, it can automatically remove problematic data from the original dataset.

Comprehensive Checks: It uniquely combines row and column-level data quality checks in one tool, simplifying the process.

Automated Error Detection: Faulty records are isolated in an error table, simplifying collaboration to rectify issues.

Efficient Error Handling: By segregating erroneous data, it eases downstream processing and correction.

Streamlined Correction: It simplifies the process of addressing data errors, reducing planning requirements.

Key Principles of Spark-Expectations:

Detailed Error Reporting: It supports individual row-based and aggregated data quality checks, providing comprehensive error information.

Aggregated Metrics: Offers aggregated metrics at the job level, reducing the need for extensive recalculations.

Selective Data Writing: Data that fails quality checks isn't written to the final table by default, preventing error propagation.

Flexible Notifications: Users can set notifications at job start, completion, or on failure.

Customizable Error Handling: It allows users to define actions if a rule fails.

In summary, Spark-Expectations is a robust data quality tool that simplifies error handling, automates data quality checks, and enhances data quality management throughout the data lifecycle.

Did you compare Spark Expectations with Deequ?

Big plans from Great Expectations team.
Just a second ago GX team shared that they are going to freeze the current release on 0.18.x and release 1.0 in 4 months

Breaking changes. in screenshot

Data Contracts are a crucial part of data pipelines. The availability features for data contract validation in data quality tools represent the killer feature. One month ago, Soda released Data Contract Validation. You can use it to enforce various aspects, such as a dataset’s schema, the data types of columns, unique values in a column, acceptable values in a column, completeness of a column, valid data referenced against another set of data, and more!

For easy onboarding, the Data Quality Gate team prepared a Proof of Concept (POC) to demonstrate the power of data contracts:
- Connect to Vertica DB
- Generate data contracts based on profiling
- Perform checks using Soda SQL

Try it out and return with feedback: GitHub Link

🌐 Wondering why being a Software Testing Engineer is awesome for Data Quality Engineers? Let's explore how these two roles mesh together! 💡🚀

Testing Expertise: 🛠
Software Testing Engineers kick off with strong testing basics, perfect for ensuring data is accurate and reliable. It's like laying a solid foundation for good data! Especially when dealing with Data Mart requirements, you can design tests before diving into the data.
Quality Focus: 🌟
QA professionals bring a mindset focused on ensuring top-notch quality. For data quality, it's not just about the data; it's about ensuring the information is genuinely high quality. Test Engineers can analyze the System Under Test (SUT) from different perspectives: business, technical, and more.
Automation Prowess: 🤖
Software Testing Engineers excel at automating processes. In data quality, this means ensuring data is regularly checked without manual effort. 🔄 Test Frameworks, Tests as Code, Reporting - all these elements are familiar territory for Test Engineers.
Collaborative Skills: 🤝
In software testing, you learn to collaborate with diverse teams. This skill comes in handy when discussing data needs and ensuring everyone agrees on what constitutes good quality. Test Engineers also possess the valuable ability to provide constructive feedback.
Root Cause Investigation: 🔍
Software testing teaches you to uncover why problems occur. This skill proves useful in data quality when figuring out why the data doesn't appear correct.
Adaptability to Change: 🔄
Software testers are accustomed to frequent changes. This adaptability proves beneficial in data quality, as the data landscape is continually evolving.
Risk Management Expertise: ⚠️
In software testing, you become adept at managing risks. This skill proves valuable in data quality to proactively avoid potential issues.
Hope this adds a touch of flair with icons and reactions! Feel free to ask if you have any questions or if you'd like further adjustments! 🌟

Data quality is crucial for different projects, and different types of efforts are accepted to ensure it:
1. Data Assessment or Data Licensing:
Assess the quality of data in your storage, be it a Data Lake, Data Warehouse (DWH), or database. This is especially important when preparing data for Machine Learning (ML) by meeting requirements from Machine Learning Engineers (MLE), and running tests to estimate data quality.
2. Continuous Data Capture:
Control data quality at each step of the Data Pipeline by running tests. Implement hard tests, where data fails to proceed if criteria aren't met, or soft tests, allowing data to move with alerts for potential issues.
3. Comparison Data:
Comparison Data: Control Data Quality after migration or transformation data. It is a popular kind of data qa because many companies migrate data to the cloud and vice versa. The main approach here is to generate test cases on source data and run the same tests against destination data.

Tools that can assist you at each step include Data Profiling tools such as PandasProfiling or VerticaPy (for Vertica DB), as well as Great Expectations or Soda for running and storing tests.

I love to find undocumented features in open-source tools. Like discoverers, you open new areas. Sometimes product owners do not know about these features.
Well, there is no doubt that Soda has many hidden features. Today I'm going to disclose one of them.
Assume you need to filter the dataset and run tests. it is pretty easy and well-documented in Filters and variables

However, If you need to compare two tables and prefilter both? I did not find how to make it in the official doc and made some investigation
It is a synthetic example but you should catch the meaning.

filter public.employee_dimension [west]:
  where: employee_region = 'West'
filter online_sales.online_page_dimension [monthly]:
  where: page_type = 'monthly'
checks for public.employee_dimension [west]:
  - row_count same as online_sales.online_page_dimension monthly:

Soda allows you to add a second filter for the table and you can add this filter as the first but without brackets.
It will work as a charm and filter both data sets

DEBUG  | Query vertica_local.public.employee_dimension[west].aggregation[0]:
SELECT 
  COUNT(*) 
FROM public.employee_dimension
WHERE employee_region = 'West'
DEBUG  | Query vertica_local.online_sales.online_page_dimension[monthly].aggregation[0]:
SELECT 
  COUNT(*) 
FROM online_sales.online_page_dimension
WHERE page_type = 'monthly'

I hope this helps you improve your data quality checks.
Good luck!

UPD: Soda team added this case to official doc: https://docs.soda.io/soda-cl/compare.html#compare-partitioned-data-in-the-same-data-source-but-different-schemas

There are no challenges in choosing a data testing tool. If you prefer open-source solutions, I suggest the following algorithm:

If your data is stored in files and the volume of tested data fits in a Pandas DataFrame, this is a Great Expectations (GX)
If you are testing data that is in databases, and checks are mostly based on SQL, feel free to use Soda.
If you have Spark, your options are Deequ or Spark Expectations.
Of course, you can use GX for Spark or Soda for files, but I have provided the best use for these tools.

Data Quality & Governance: Mastering the Metrics

Hey everyone! Today we're diving deep into data quality and governance metrics. These metrics are essential for keeping your data ship running smoothly and ensuring you're getting the most out of your valuable information.

I split metrics by frequency monitoring.

Daily Monitoring:

Data Freshness: This metric tracks how much of your data is updated within a set timeframe. Consider it a daily "vitamins" check for your data health!

Row Count Z-Score: This fancy term tells you how much your daily data volume deviates from the historical average. Sudden spikes or dips can indicate potential issues.

Weekly Monitoring:

Failed Data Quality Tests Rate: This metric shows the percentage of data failing pre-defined quality checks. Think of it as catching errors before they cause problems downstream.
Monthly Monitoring:

Data Incident Rate: Here, we track the number of incidents impacting data integrity or availability in production. Think of it as a red flag system for critical data issues.

Data Usage Frequency: This metric reveals how often your data assets are accessed or queried. It helps identify the most valuable data for your organization, allowing for better resource allocation.

Quarterly Monitoring:

User Satisfaction with Data Quality: This metric shows user experience with data accuracy and accessibility through surveys or feedback mechanisms. Happy data users mean happy business!

Data Quality Coverage: This one counts the number of new tables without any data quality checks. Think of it as identifying blind spots in your data quality monitoring.

Yearly Monitoring:

Data Governance Maturity Assessment Score: Here, we assess your data governance practices using a standardized framework like DAMA-DM or CMMI. This helps evaluate your progress toward a solid data governance program. The levels range from Ad Hoc (limited practices) to Optimized (continuously monitored and improved).

Data Compliance Rate: This metric assesses the percentage of data meeting regulatory compliance standards. Think of it as a data health check-up for legal requirements.

Freshness Metric in Data Quality

Introduction:
Freshness is a crucial metric in data quality that measures the timeliness and currency of data. It provides insights into how up-to-date the data is and is essential for making informed decisions and ensuring data reliability.
1. What is the Freshness Metric in Data Quality?
The freshness metric in data quality refers to how recently the data has been updated or inserted into a dataset. It indicates the time elapsed since the last update or insertion of data. A high freshness score indicates that the data is more current and reliable.

2. How to Measure Freshness?
There are different approaches to measuring freshness depending on the context and requirements. Two common methods are:
a. Business Column (updated_at): In some cases, the freshness of data is best measured using a business column, such as the "updated_at" column. This column captures the timestamp of the last update made to the data. By comparing the current timestamp with the "updated_at" value, we can determine the freshness of the data.

b. Technical Column (inserted_at): Alternatively, the freshness of data can be measured using a technical column, such as the "inserted_at" column. This column captures the timestamp when the data was inserted into the dataset. By comparing the current timestamp with the "inserted_at" value, we can assess the freshness of the data.

3. Choosing the Right Column for Freshness Measurement:
It means that you will have two types of Freshness:
1. Technical - when the table was updated
2. Business - when data was updated
Not always these params will be same

Anyway, you could always use Freshness and Latency to capture all needs in your data quality

Conclusion:
Freshness is a vital metric in data quality that helps organizations ensure the reliability and timeliness of their data. Businesses can make better decisions based on up-to-date and accurate data by understanding what freshness metric is, how to measure it, and choosing the right column for measurement.

GX Cloud pricing


Great Expectations published pricing for GX Cloud
They presented 3 options
- Developer - Free plan but with huge limitations. For instance, you can monitor only 5 data assets
- Team - 100$/months. 10 data assets are included and you will pay 10$/months per additional asset
- Enterprise - as usual call and make an agreement
Full plan: https://greatexpectations.io/pricing
Remember that count of data asset >= count of tables. You can have more than one data asset for one table.
Will use in your projects?

Unveiling the Layers of Data Quality Maturity

There are several approaches that help define Maturity Models. One of them is Data Quality Maturity Model by David Loshin
This model has such levels
* Initial. The process used for data quality assurance is mostly ad hoc with the most effort to respond to data quality issues.
* Repeatable. There is some management in the organization and simple information-sharing activities. There are some process disciplines, mostly it is adopted from good practice and tries to imitate the practice in the same situation.
* Defined. At this level, the team that handles data quality begins to document things like data governance policies, processes to define expectations of data quality, technology components, data quality, and report of validation processes.
* Managed. DQM includes business impact analysis, defines expectations of data quality, and measures compliance with those expectations.
* Optimized. Performance measurement across the organization can be used to identify opportunities for systemically improving data quality.
And the check is carried out on the following components
* Expectation
* Dimension
* Policy
* Procedure
* Governance
* Standardization
* Technology
* Performance Management

The assessment takes place according to a checklist and based on interviews with participants.
Then all this is summarized into results. Additionally you can make all sorts of recommendations.

Article where I got information:
[1] The Maturity Model of Data Quality Management in Banking Industry: PTXYZ Core System Customer Data
[2] Data Quality Management Maturity Measurement of Government-Owned Property Transaction in BMKG
[3] And book of David : The Practitioner's Guide to Data Quality Improvement


What is your organisation's maturity level?

Failed Rows

Do you think about best practices in data quality checks, especially when using SQL? When you run checks against any database, you should consider the information you will receive if a test fails. The information should clearly demonstrate why the check failed and the severity of the issue.

If the test returns FALSE, you must make additional SQL queries to understand the reason for the failure. It's crucial to grasp the cause of the check's result. The best way to implement this is by using the "failed rows" pattern. If a check fails, it returns the failed rows. These rows can include IDs from a table, allowing you to identify issues or the maximum time in a freshness check.

For instance, consider a freshness check for a table:

SELECT max(ss.created_date_time) AS m_time
FROM store.sales ss
HAVING max(ss.created_date_time) < NOW() - INTERVAL '24 hours';

This query returns the maximum time if the freshness exceeds 24 hours, providing complete information about the extent of your freshness issue.

By the way, the same pattern exists in SODA CL.

Understanding Metadata Types

There are three main types of metadata:

Business Metadata: This includes information about who owns the data, as well as definitions and descriptions of data assets. Business metadata helps provide context and meaning to data from a business perspective. It includes details like the data domain, owner information, business names, and definitions.

Technical Metadata: This type of metadata provides information about the technical aspects of data, such as column names in databases, data types, and other structural information. It describes the technical characteristics and format of data assets.

Operational Metadata: This includes information like logs and other details about how data is used, processed, and maintained over time. Operational metadata can help track data lineage, usage patterns, data quality score, and other operational aspects of data management.

Understanding these types of metadata is essential for maintaining high data quality standards and ensuring that data is accurate, reliable, and meaningful.

Do you collect metadata?