If you're using SQLAlchemy and a database with Great Expectations, here's a helpful tip: You can use logging to view the queries that GX executes for validations. By enabling logging, you can gain insights into the database interactions and improve your understanding of how GX works with your data.

If you want to speed up your validations in Great Expectations, try running them in parallel. One way to do this is using #pytest, which allows you to run tests concurrently. By wrapping your checkpoints in tests and running them in parallel, you can significantly reduce the time it takes to validate your data. So give it a try and see if it improves your workflow!
#DataQuality #tips #datamanagement #GXtips #data

Why is Data Quality Important?

I have one answer which should cover all doubts about DQ's importance. Agree?

Data Management as Real Estate Management. Does it work for you?
Source

Friday !

The first course for GX. You can try GX in action without struggle with setting: Great Expectations, a data validation library for Python

True story.

Hey, do you use AWS Athena to analyze your data? If so, you might want to check out this awesome article. It shows you two ways to keep your data quality in check: AWS Glue Data Quality and Great Expectations. Looks like GX is still the best tool for Data QA

📢 Good news from Soda🎉

This week, Soda released Rule Suggestions, joining the ranks of GreatExpectation and Deequ in offering this powerful feature. 🚀 Now, you have multiple options to generate initial tests for your data source and improve your data quality management. 💪

Soda currently supports 6 rules for generating these suggestions, and you can find more information in the official documentation. 📋 Make the most out of these amazing tools and optimize your data with ease. 🌟
#DataQuality #SodaUpdates #GreatExpectation #Deequ #RuleSuggestions #Efficiency

Friday! 😜

Read the article: The new role of the Data Quality Engineer if you don't know why the Data Quality engineer is needed.
🔍 Main takeaways:

- Poor data quality costs organizations an average of $15 million per year.
- Data observability and data quality are crucial for organizations to become more data-driven and increase revenue.
- Data Quality (DQ) Engineers are needed to address data quality challenges and ensure the success of data products.
- DQ Engineers validate data flow, establish data quality metrics, and perform root cause analysis of data quality issues.
- Required skills for DQ Engineers include data management, data analysis, programming languages, data observability tools, and communication skills.
- DQ Engineers collaborate with stakeholders such as data owners, business stakeholders, data analysts, data engineering teams, and data governance teams.
- The role of DQ Engineers is to maintain high-quality data and break the cycle of "Garbage In, Garbage Out" in organizations.
🌟 Data Quality Engineers: The Key to High-Quality Data and Business Success

Nice infographic about Quality Assurance in Data. My vision is to prevent bugs is cheaper and easy than bug fixes

With Great Expectations, you can easily parameterise tests, increasing coverage while maintaining a minimal test count. What's more, you have the flexibility to hide sensitive information from the test template, ensuring the privacy and security of your data.
Let's take a closer look at how it works. In your my_suite.json file, you can define your expectations using the JSON Template format. For example, you can specify the expectation to check if column values fall within a specific range look my_suite.json. Once you have defined your suite, you can run your tests with specific parameters using a validator or checkpoint.
Once you have defined your suite, you can run your tests with specific parameters using a validator or checkpoint. Look at test.py as an example.
By leveraging GX's parameterisation capabilities, you can easily customize your tests and run them with different parameter values, maximizing coverage and ensuring the integrity of your data.
So, why settle for limited test coverage or compromise data security? Take advantage of Great Expectations and unlock the full potential of your testing efforts today!

Data Quality importance in article
"One data leader I spoke with at a major transportation company told me that, on average, it took his team of 45 engineers and analysts 140 hours per week to manually check for data issues in their pipelines. Even if you have a data team of 10 people, that’s five whole days that could have spent on revenue-generating activities."
https://www.montecarlodata.com/blog-how-to-fix-your-data-quality-problem/

"According to the latest Data Quality Survey conducted by the Monte Carlo team, a significant number of respondents stated that Data Engineers are primarily responsible for Data Quality. However, I hold a different perspective on this matter; I believe that ensuring data quality is a collective responsibility that extends to the entire team. 📊✨ #DataQuality #TeamEffort"

🔍 Data Detectives Unite! 🕵️‍♀️🕵️‍♂️

When navigating the mysterious world of data, it's essential to distinguish between bugs and features. Here's your investigative roadmap:

1️⃣ Read Documentation: Start by delving into your dataset's documentation for clues. If it's from an external source, reach out to uncover additional context. 📑

2️⃣ Critical Questioning: Pose the pivotal query: "Is this absence intentional or an inadvertent hiccup?" 🤔

3️⃣ Data Doesn't Exist: If it's by design (think childless parents' child height), it's a feature, not a bug. Leave it be. 🚀

4️⃣ Data Wasn't Recorded: If it should've been there but isn't, you might have a sneaky bug on your hands. Investigate further. 🐞

With this flow, you'll master the art of discerning between data quirks and data quirks that need fixing. Happy data sleuthing! 🔍📊 #DataAnalysis #BugOrFeature #DataQuality

True story

Struggling with choosing the right tests for your data? I've got you covered! 🛡

Here's a recommendation for your initial data test suite, addressing critical dimensions:

1. NULL Values Test: Ensure your data is free of missing or NULL values, preventing inaccuracies in your analyses.

2. Volume Tests: Essential for data quality, these tests validate row counts in crucial tables, ensuring data completeness and accuracy.

3. Uniqueness Tests: Verify the uniqueness of key data attributes to avoid duplicates that could skew your results.

4. Integrity Tests: Assess data integrity to maintain relationships between various data elements.

5. Validity Tests: Confirm data adherence to predefined rules and constraints, ensuring reliability.

6. Freshness Checks: Monitor data timeliness for informed decisions based on up-to-date information.

For further insights into these vital data quality dimensions and additional tests, refer to article: Data Quality Dimensions: Assuring Your Data Quality with Great Expectations

Don't let data quality issues hold you back. Begin with these foundational tests to ensure your data's reliability and accuracy.
#DataProfiling #DataManagement #DataAnalysis

It’s really crucial question.

🚀 Discover a Game-Changer! 🚀

🔮 These two lines can save your life time 💥

📊 Easily transform Ydata profiling results into a robust GreatExpectations suite with just a few lines:

profile_result = ProfileReport(df=df_ms, title="MS_Report")
profile_result.to_expectation_suite(data_context=context, run_validation=False)

🎯 It's that simple! Dive into more advanced cases in the article:
📖 Read Now

🛠 Curious about our Data Quality Tool? It follows the same winning approach for generating technical tests. Explore repository:
🔗 DataQualityGate

Join us on this data-driven journey to excellence! 🌟 #DataProfiling #DataQuality #GreatExpectations