Data Build Tool (dbt). Transformation in Modern data stack | by Amit Singh Rathore | Jan, 2023 | Dev Genius
https://blog.devgenius.io/data-build-tool-dbt-1f0b03d97cc6

Как запушить в Gitlab пакет npm, помогло 👍

Publishing your private npm packages to Gitlab NPM Registry

https://shivamarora.medium.com/publishing-your-private-npm-packages-to-gitlab-npm-registry-39d30a791085

Очередная подборочка инструментов Awesome-Selfhosted

A list of Free Software network services and web applications which can be hosted on your own servers

https://github.com/awesome-selfhosted/awesome-selfhosted

Prescriber-ETL-data-pipeline

An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports

https://github.com/judeleonard/Prescriber-ETL-data-pipeline

airflow-docker

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

https://github.com/anilkulkarni87/airflow-docker

Неплохая сравнительная табличка по инструментам metadata management

Awesome Data Discovery and Observability
https://github.com/opendatadiscovery/awesome-data-catalogs

2023 — Rockstar Data Engineer Roadmap
https://akaabachi.medium.com/2023-rockstar-data-engineer-roadmap-c990da37f625

Примеры из курса про Apache Airflow 2.0
https://github.com/adilkhash/apache-airflow-course-materials

How to Orchestrate an ETL Data Pipeline with Apache Airflow
https://www.freecodecamp.org/news/orchestrate-an-etl-data-pipeline-with-apache-airflow/

OpenMetadata vs DataHub

Один из пунктов "Против" решения Datahub - это их раздражающий функционал открытия Data Lineage.
Почему нельзя сделать кнопку открытия всего дерева - для меня загадка.
Пока при сравнении OpenMetadata vs DataHub лидирует OpenMetadata продукт.

Data Engineering with Python
Packt Publishing

Key Features
▫️Become well-versed in data architectures, data preparation, and data optimization skills with the help of practical examples
▫️Design data models and learn how to extract, transform, and load (ETL) data using Python
▫️Schedule, automate, and monitor complex data pipelines in production

👉 @devops_dataops

Data Engineering - Open Source Tools/Databases

A curated list of docker-compose files prepared for testing data engineering tools, databases and open source libraries.

Airflow
Cassandra
ClickHouse
Drill
Druid
ELK
Grafana-Prometheus
Hadoop
Kafka
LakeFS
Mariadb
Minio
Postgres
Redis
Spark
Superset
Trino
mongo

https://github.com/irbigdata/data-dockerfiles

Apache Druid in 5 minutes
https://youtu.be/X8ZnwwmCBAA

PySpark Tutorial
https://youtu.be/_C8kWso4ne4

GitHub code: https://github.com/krishnaik06/Pyspark-With-Python

The 2023 MAD (Machine Learning, Artificial Intelligence & Data) Landscape – Matt Turck

Source: https://mattturck.com/mad2023/

Подборка проектов с GitHub

〰️〰️〰️〰️〰️〰️〰️〰️
🔸 Engineering Python
Welcome to Engineering Python. This is a Python programming course for engineers.

This GitHub repository hosts the Jupyter Notebooks and Python source code for the open course on YouTube (http://youtube.com/yongtwang).

A tutorial on how to use these course materials is in this YouTube video: 02C Course Materials and Jupyter Notebook.

〰️〰️〰️〰️〰️〰️〰️〰️
🔸 Fun and useful projects with Python
You can find the corresponding tutorials on my channel: https://www.youtube.com/c/PythonEngineer

〰️〰️〰️〰️〰️〰️〰️〰️
🔸 Python Engineer Roadmap
Python can be used in a lot of computer science fields. In this repository, we have collected resources for each field of computer science that are related to Python.

〰️〰️〰️〰️〰️〰️〰️〰️
🔸 PyTorch Beginner Tutorials from my YouTube channel

• Installation
• Tensor Basics
• Autograd
• Backpropagation
• Gradient Descent With Autograd and Backpropagation
• Training Pipeline: Model, Loss, and Optimizer
• Linear Regression
• Logistic Regression
• Dataset and DataLoader
• Dataset Transforms
• Softmax And Cross Entropy
• Activation Functions
• Feed-Forward Neural Net
• Convolutional Neural Net (CNN)
• Transfer Learning
• Tensorboard
• Save and Load Models

😐 Docker's New Ultimatum Can Affect Open-Source Projects in a Big, Negative Way

https://news.itsfoss.com/docker-dropping-free-team-orgs/

Ловите гайд по Apache Superset Сбер 😁
https://developers.sber.ru/docs/ru/sdp/sdpanalytics/guidelines-reports-SDPBI

Яндекс выкладывает в опенсорс одну из основных инфраструктурных BigData-систем собственной разработки — YTsaurus. Это платформа, предназначенная для распределённого хранения и обработки больших данных.

Максим Бабенко, руководитель отдела технологий распределённых вычислений в Яндексе, рассказал историю возникновения YT, а также зачем нужна YTsaurus и где её можно применять.

В Github-репозитории — серверный код YTsaurus, инфраструктура развёртывания с использованием k8s, а также веб-интерфейс системы и клиентский SDK для распространённых языков программирования — C++, Java, Go и Python.

Ссылки на посты на Хабре и Медиуме.