The Truth about Prefect, Mage, and Airflow.
https://dataengineeringcentral.substack.com/p/the-truth-about-prefect-mage-and
https://dataengineeringcentral.substack.com/p/the-truth-about-prefect-mage-and
Substack
The Truth about Prefect, Mage, and Airflow.
The Battle for the Orchestration Future.
MLOps Is Overfitting: Here’s Why
https://lakefs.io/blog/mlops-is-overfitting/
https://lakefs.io/blog/mlops-is-overfitting/
Git for Data - lakeFS
How To Improve ML Pipeline Development With Reproducibility
In this article we'll explore how to improve your ML pipeline development with MLOps tools for reproducible experiments. Read on to learn more.
The future of the data engineer — Part I | by Analytics at Meta | Apr, 2023 | Medium
https://medium.com/@AnalyticsAtMeta/the-future-of-the-data-engineer-part-i-32bd125465be
https://medium.com/@AnalyticsAtMeta/the-future-of-the-data-engineer-part-i-32bd125465be
Medium
The future of the data engineer — Part I
Introduction
❤1
Introducing AI Functions: Integrating Large Language Models with Databricks SQL - The Databricks Blog
https://www.databricks.com/blog/2023/04/18/introducing-ai-functions-integrating-large-language-models-databricks-sql.html?utm_source=bambu&utm_medium=social&utm_campaign=advocacy
https://www.databricks.com/blog/2023/04/18/introducing-ai-functions-integrating-large-language-models-databricks-sql.html?utm_source=bambu&utm_medium=social&utm_campaign=advocacy
Open source self-hosted Delta Sharing server | Delta Lake
https://delta.io/blog/2023-04-24-open-source-selfhosted-delta-sharing-server/
https://delta.io/blog/2023-04-24-open-source-selfhosted-delta-sharing-server/
delta.io
Open source self-hosted Delta Sharing server
This post explains Kotosiro Delta Sharing server basic instructions
👍5
dbt Guide | GitLab
https://about.gitlab.com/handbook/business-technology/data-team/platform/dbt-guide/
https://about.gitlab.com/handbook/business-technology/data-team/platform/dbt-guide/
👍5
Microsoft introduced Fabric, which is a combination of Power BI, Azure Synapse, Data Factory and Data Explorer on top of ADLS gen2 using Delta (Parquet) as data lake format. A new component is Data Activator which seems to be a no-code rule engine.
https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/
https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/
Microsoft Azure Blog
Introducing Microsoft Fabric: The data platform for the era of AI | Microsoft Azure Blog | Microsoft Azure
Announcing Microsoft Fabric—a unified analytics platform that brings together all the data and analytics tools that organizations need. Learn more.
❤1
Microsoft OneLake in Fabric, the OneDrive for data.
Microsoft
Microsoft OneLake in Fabric, the OneDrive for data | Microsoft Fabric Blog | Microsoft Fabric
Introducing Microsoft OneLake – “the OneDrive for Data”. OneLake is a complete, rich, ready-to-go enterprise-wide data lake provided as a SaaS service.
This is huge! PowerBI now has
Git integration and Power BI Desktop ‘Developer Mode' which means you can edit files in code editor like VSCode and deploy them using PowerBI deployment pipelines. In other words "dashboards as a code". Apparently, this functionality was added to enable Copilot.
Git integration and Power BI Desktop ‘Developer Mode' which means you can edit files in code editor like VSCode and deploy them using PowerBI deployment pipelines. In other words "dashboards as a code". Apparently, this functionality was added to enable Copilot.
Microsoft
Introducing git integration in Microsoft Fabric for seamless source control management | Microsoft Fabric Blog | Microsoft Fabric
Git integration enables developers to integrate their development processes, tools, and best practices straight into the Microsoft Fabric workspace.
👍4
Empower every BI professional to do more with Microsoft Fabric
https://build.microsoft.com/en-US/sessions/8b23c96e-7c35-463d-88b4-564d23dc14a5
https://build.microsoft.com/en-US/sessions/8b23c96e-7c35-463d-88b4-564d23dc14a5
Choosing an open table format for your transactional data lake on AWS | AWS Big Data Blog
https://aws.amazon.com/blogs/big-data/choosing-an-open-table-format-for-your-transactional-data-lake-on-aws/
https://aws.amazon.com/blogs/big-data/choosing-an-open-table-format-for-your-transactional-data-lake-on-aws/
Amazon
Choosing an open table format for your transactional data lake on AWS | Amazon Web Services
August 2023: This post was updated to include Apache Iceberg support in Amazon Redshift. Disclaimer: Due to rapid advancements in AWS service support for open table formats, recent developments might not yet be reflected in this post. For the latest information…
AWS Glue Data Quality is Generally Available | AWS Big Data Blog
https://aws.amazon.com/blogs/big-data/aws-glue-data-quality-is-generally-available/
https://aws.amazon.com/blogs/big-data/aws-glue-data-quality-is-generally-available/
Amazon
AWS Glue Data Quality is Generally Available | Amazon Web Services
We are excited to announce the General Availability of AWS Glue Data Quality. Our journey started by working backward from our customers who create, manage, and operate data lakes and data warehouses for analytics and machine learning. To make confident business…
Data1984
Choosing an open table format for your transactional data lake on AWS | AWS Big Data Blog https://aws.amazon.com/blogs/big-data/choosing-an-open-table-format-for-your-transactional-data-lake-on-aws/
While AWS tries to support all three formats, and helps to choose right one for your use-case, Databricks introduces unified format, so you will not need to pick 😎
Datanami
Databricks Puts Unified Data Format on the Table with Delta Lake 3.0
Databricks today rolled out a new open table format in Delta Lake 3.0 that it says will eliminate the possibility of picking the wrong one. Dubbed