Data Contracts are a crucial part of data pipelines. The availability features for data contract validation in data quality tools represent the killer feature. One month ago, Soda released Data Contract Validation. You can use it to enforce various aspects, such as a dataset’s schema, the data types of columns, unique values in a column, acceptable values in a column, completeness of a column, valid data referenced against another set of data, and more!

For easy onboarding, the Data Quality Gate team prepared a Proof of Concept (POC) to demonstrate the power of data contracts:
- Connect to Vertica DB
- Generate data contracts based on profiling
- Perform checks using Soda SQL

Try it out and return with feedback: GitHub Link

🌐 Wondering why being a Software Testing Engineer is awesome for Data Quality Engineers? Let's explore how these two roles mesh together! 💡🚀

Testing Expertise: 🛠
Software Testing Engineers kick off with strong testing basics, perfect for ensuring data is accurate and reliable. It's like laying a solid foundation for good data! Especially when dealing with Data Mart requirements, you can design tests before diving into the data.
Quality Focus: 🌟
QA professionals bring a mindset focused on ensuring top-notch quality. For data quality, it's not just about the data; it's about ensuring the information is genuinely high quality. Test Engineers can analyze the System Under Test (SUT) from different perspectives: business, technical, and more.
Automation Prowess: 🤖
Software Testing Engineers excel at automating processes. In data quality, this means ensuring data is regularly checked without manual effort. 🔄 Test Frameworks, Tests as Code, Reporting - all these elements are familiar territory for Test Engineers.
Collaborative Skills: 🤝
In software testing, you learn to collaborate with diverse teams. This skill comes in handy when discussing data needs and ensuring everyone agrees on what constitutes good quality. Test Engineers also possess the valuable ability to provide constructive feedback.
Root Cause Investigation: 🔍
Software testing teaches you to uncover why problems occur. This skill proves useful in data quality when figuring out why the data doesn't appear correct.
Adaptability to Change: 🔄
Software testers are accustomed to frequent changes. This adaptability proves beneficial in data quality, as the data landscape is continually evolving.
Risk Management Expertise: ⚠️
In software testing, you become adept at managing risks. This skill proves valuable in data quality to proactively avoid potential issues.
Hope this adds a touch of flair with icons and reactions! Feel free to ask if you have any questions or if you'd like further adjustments! 🌟

Data quality is crucial for different projects, and different types of efforts are accepted to ensure it:
1. Data Assessment or Data Licensing:
Assess the quality of data in your storage, be it a Data Lake, Data Warehouse (DWH), or database. This is especially important when preparing data for Machine Learning (ML) by meeting requirements from Machine Learning Engineers (MLE), and running tests to estimate data quality.
2. Continuous Data Capture:
Control data quality at each step of the Data Pipeline by running tests. Implement hard tests, where data fails to proceed if criteria aren't met, or soft tests, allowing data to move with alerts for potential issues.
3. Comparison Data:
Comparison Data: Control Data Quality after migration or transformation data. It is a popular kind of data qa because many companies migrate data to the cloud and vice versa. The main approach here is to generate test cases on source data and run the same tests against destination data.

Tools that can assist you at each step include Data Profiling tools such as PandasProfiling or VerticaPy (for Vertica DB), as well as Great Expectations or Soda for running and storing tests.

I love to find undocumented features in open-source tools. Like discoverers, you open new areas. Sometimes product owners do not know about these features.
Well, there is no doubt that Soda has many hidden features. Today I'm going to disclose one of them.
Assume you need to filter the dataset and run tests. it is pretty easy and well-documented in Filters and variables

However, If you need to compare two tables and prefilter both? I did not find how to make it in the official doc and made some investigation
It is a synthetic example but you should catch the meaning.

filter public.employee_dimension [west]:
  where: employee_region = 'West'
filter online_sales.online_page_dimension [monthly]:
  where: page_type = 'monthly'
checks for public.employee_dimension [west]:
  - row_count same as online_sales.online_page_dimension monthly:

Soda allows you to add a second filter for the table and you can add this filter as the first but without brackets.
It will work as a charm and filter both data sets

DEBUG  | Query vertica_local.public.employee_dimension[west].aggregation[0]:
SELECT 
  COUNT(*) 
FROM public.employee_dimension
WHERE employee_region = 'West'
DEBUG  | Query vertica_local.online_sales.online_page_dimension[monthly].aggregation[0]:
SELECT 
  COUNT(*) 
FROM online_sales.online_page_dimension
WHERE page_type = 'monthly'

I hope this helps you improve your data quality checks.
Good luck!

UPD: Soda team added this case to official doc: https://docs.soda.io/soda-cl/compare.html#compare-partitioned-data-in-the-same-data-source-but-different-schemas

There are no challenges in choosing a data testing tool. If you prefer open-source solutions, I suggest the following algorithm:

If your data is stored in files and the volume of tested data fits in a Pandas DataFrame, this is a Great Expectations (GX)
If you are testing data that is in databases, and checks are mostly based on SQL, feel free to use Soda.
If you have Spark, your options are Deequ or Spark Expectations.
Of course, you can use GX for Spark or Soda for files, but I have provided the best use for these tools.

Data Quality & Governance: Mastering the Metrics

Hey everyone! Today we're diving deep into data quality and governance metrics. These metrics are essential for keeping your data ship running smoothly and ensuring you're getting the most out of your valuable information.

I split metrics by frequency monitoring.

Daily Monitoring:

Data Freshness: This metric tracks how much of your data is updated within a set timeframe. Consider it a daily "vitamins" check for your data health!

Row Count Z-Score: This fancy term tells you how much your daily data volume deviates from the historical average. Sudden spikes or dips can indicate potential issues.

Weekly Monitoring:

Failed Data Quality Tests Rate: This metric shows the percentage of data failing pre-defined quality checks. Think of it as catching errors before they cause problems downstream.
Monthly Monitoring:

Data Incident Rate: Here, we track the number of incidents impacting data integrity or availability in production. Think of it as a red flag system for critical data issues.

Data Usage Frequency: This metric reveals how often your data assets are accessed or queried. It helps identify the most valuable data for your organization, allowing for better resource allocation.

Quarterly Monitoring:

User Satisfaction with Data Quality: This metric shows user experience with data accuracy and accessibility through surveys or feedback mechanisms. Happy data users mean happy business!

Data Quality Coverage: This one counts the number of new tables without any data quality checks. Think of it as identifying blind spots in your data quality monitoring.

Yearly Monitoring:

Data Governance Maturity Assessment Score: Here, we assess your data governance practices using a standardized framework like DAMA-DM or CMMI. This helps evaluate your progress toward a solid data governance program. The levels range from Ad Hoc (limited practices) to Optimized (continuously monitored and improved).

Data Compliance Rate: This metric assesses the percentage of data meeting regulatory compliance standards. Think of it as a data health check-up for legal requirements.

Freshness Metric in Data Quality

Introduction:
Freshness is a crucial metric in data quality that measures the timeliness and currency of data. It provides insights into how up-to-date the data is and is essential for making informed decisions and ensuring data reliability.
1. What is the Freshness Metric in Data Quality?
The freshness metric in data quality refers to how recently the data has been updated or inserted into a dataset. It indicates the time elapsed since the last update or insertion of data. A high freshness score indicates that the data is more current and reliable.

2. How to Measure Freshness?
There are different approaches to measuring freshness depending on the context and requirements. Two common methods are:
a. Business Column (updated_at): In some cases, the freshness of data is best measured using a business column, such as the "updated_at" column. This column captures the timestamp of the last update made to the data. By comparing the current timestamp with the "updated_at" value, we can determine the freshness of the data.

b. Technical Column (inserted_at): Alternatively, the freshness of data can be measured using a technical column, such as the "inserted_at" column. This column captures the timestamp when the data was inserted into the dataset. By comparing the current timestamp with the "inserted_at" value, we can assess the freshness of the data.

3. Choosing the Right Column for Freshness Measurement:
It means that you will have two types of Freshness:
1. Technical - when the table was updated
2. Business - when data was updated
Not always these params will be same

Anyway, you could always use Freshness and Latency to capture all needs in your data quality

Conclusion:
Freshness is a vital metric in data quality that helps organizations ensure the reliability and timeliness of their data. Businesses can make better decisions based on up-to-date and accurate data by understanding what freshness metric is, how to measure it, and choosing the right column for measurement.

GX Cloud pricing


Great Expectations published pricing for GX Cloud
They presented 3 options
- Developer - Free plan but with huge limitations. For instance, you can monitor only 5 data assets
- Team - 100$/months. 10 data assets are included and you will pay 10$/months per additional asset
- Enterprise - as usual call and make an agreement
Full plan: https://greatexpectations.io/pricing
Remember that count of data asset >= count of tables. You can have more than one data asset for one table.
Will use in your projects?

Unveiling the Layers of Data Quality Maturity

There are several approaches that help define Maturity Models. One of them is Data Quality Maturity Model by David Loshin
This model has such levels
* Initial. The process used for data quality assurance is mostly ad hoc with the most effort to respond to data quality issues.
* Repeatable. There is some management in the organization and simple information-sharing activities. There are some process disciplines, mostly it is adopted from good practice and tries to imitate the practice in the same situation.
* Defined. At this level, the team that handles data quality begins to document things like data governance policies, processes to define expectations of data quality, technology components, data quality, and report of validation processes.
* Managed. DQM includes business impact analysis, defines expectations of data quality, and measures compliance with those expectations.
* Optimized. Performance measurement across the organization can be used to identify opportunities for systemically improving data quality.
And the check is carried out on the following components
* Expectation
* Dimension
* Policy
* Procedure
* Governance
* Standardization
* Technology
* Performance Management

The assessment takes place according to a checklist and based on interviews with participants.
Then all this is summarized into results. Additionally you can make all sorts of recommendations.

Article where I got information:
[1] The Maturity Model of Data Quality Management in Banking Industry: PTXYZ Core System Customer Data
[2] Data Quality Management Maturity Measurement of Government-Owned Property Transaction in BMKG
[3] And book of David : The Practitioner's Guide to Data Quality Improvement


What is your organisation's maturity level?

Failed Rows

Do you think about best practices in data quality checks, especially when using SQL? When you run checks against any database, you should consider the information you will receive if a test fails. The information should clearly demonstrate why the check failed and the severity of the issue.

If the test returns FALSE, you must make additional SQL queries to understand the reason for the failure. It's crucial to grasp the cause of the check's result. The best way to implement this is by using the "failed rows" pattern. If a check fails, it returns the failed rows. These rows can include IDs from a table, allowing you to identify issues or the maximum time in a freshness check.

For instance, consider a freshness check for a table:

SELECT max(ss.created_date_time) AS m_time
FROM store.sales ss
HAVING max(ss.created_date_time) < NOW() - INTERVAL '24 hours';

This query returns the maximum time if the freshness exceeds 24 hours, providing complete information about the extent of your freshness issue.

By the way, the same pattern exists in SODA CL.

Understanding Metadata Types

There are three main types of metadata:

Business Metadata: This includes information about who owns the data, as well as definitions and descriptions of data assets. Business metadata helps provide context and meaning to data from a business perspective. It includes details like the data domain, owner information, business names, and definitions.

Technical Metadata: This type of metadata provides information about the technical aspects of data, such as column names in databases, data types, and other structural information. It describes the technical characteristics and format of data assets.

Operational Metadata: This includes information like logs and other details about how data is used, processed, and maintained over time. Operational metadata can help track data lineage, usage patterns, data quality score, and other operational aspects of data management.

Understanding these types of metadata is essential for maintaining high data quality standards and ensuring that data is accurate, reliable, and meaningful.

Do you collect metadata?

Friday

Friday - 2

🗞 Netflix has open-sourced their Data/ML Workflows orchestrator Maestro. It’s been in
It is focused on scalability and developer UX. DAGs can be defined in YAML (very similar to Databricks Jobs), Python DSL, or Java DSL.
From the post:

Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can package their business logic in various formats such as Docker images, notebooks, bash script, SQL, Python, and more. Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.

Friday. Actual meme :)

This is a huge milestone for GX

Introducing GX 1.0

when you disable not critical test for unblocking data pipeline
#Friday

In the last two months, I minimalied incoming articles about data qa and focused on books instead of this. My opinion is that books give more structure and a deeper understanding of problems and solutions.
Today I want to recommend several books about DQ:
Data Quality Fundamentals - the book covering the main ideas about data quality. The book is good if you have nothing for DQ and you are ready to make the first step. Some ideas are out of date and relate to Monte Carlo but in general, I can recommend this book.
A step-by-step guide to improving data quality. This book is from the author of service DQOps and looks like a manual on how to set up a service. However, you can get some ideas about incident management and a dashboard view for your process.
Several days ago Anomalo published a book about Anomalo Detection "Automating Data Quality Monitoring" I had a chance to be in early subscribe to this book and have read a couple of chapters. This book is interesting because describes how to build anomaly detections (surprise ) in your data.

Now I can say that all these books are on my table and I use information from them every day. See picture

#Friday Do you use Pomodoro technic?