Pandas vs PySpark DataFrame With Examples

Let’s learn the difference between Pandas vs PySpark DataFrame, their definitions, features, advantages, how to create them and transform one to another with Examples.

👉 @devops_dataops

https://sparkbyexamples.com/pyspark/pandas-vs-pyspark-dataframe-with-examples/

Артем Шутак — Вставить в ClickHouse и не умереть

Презентация

Подборка ClickHouse + Spark - Altinity Knowledge Base
-----------
👉 @devops_dataops

Github Actions - Введение в CI/CD

00:00 - О чем курс
03:50 - Github вводный курс
12:35 - Начало работы с Github Actions
18:20 - Пишем первый workflow
29:17 - Автоматически тестируем React
37:57 - Что такое Actions
48:25 - Усложняем workflow (практика)
53:40 - Зависимость job и их порядок
01:00:18 - Context & Events
01:21:19 - Добавление cache
01:28:13 - Matrix
01:35:44 - Artifacts
01:45:25 - Environment & Secrets

https://www.youtube.com/watch?v=e0A2hDObLmg

Интересная модель монетизации у этого софта, вроде опенсоурс, но и есть разумные плюшки, которые можно получить только в платной версии (пользователи и роли + поддержка).
Ну и сама идея появления платформ с low-code подходом как open-source тоже интересная.
----
Tooljet | Open-source low-code platform to build internal tools

Extensible low-code framework for building business applications. Connect to databases, cloud storages, GraphQL, API endpoints, Airtable, etc and build apps using drag and drop application builder. Built using JavaScript/TypeScript.

https://www.tooljet.com/

Kubernetes Explained in 6 Minutes | k8s Architecture - YouTube
https://m.youtube.com/watch?v=TlHvYWVUZyc

ChatGPT Tutorial for Developers - 38 Ways to 10x Your Productivity

https://www.youtube.com/watch?v=sTeoEFzVNSc

Data Build Tool (dbt). Transformation in Modern data stack | by Amit Singh Rathore | Jan, 2023 | Dev Genius
https://blog.devgenius.io/data-build-tool-dbt-1f0b03d97cc6

Как запушить в Gitlab пакет npm, помогло 👍

Publishing your private npm packages to Gitlab NPM Registry

https://shivamarora.medium.com/publishing-your-private-npm-packages-to-gitlab-npm-registry-39d30a791085

Очередная подборочка инструментов Awesome-Selfhosted

A list of Free Software network services and web applications which can be hosted on your own servers

https://github.com/awesome-selfhosted/awesome-selfhosted

Prescriber-ETL-data-pipeline

An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports

https://github.com/judeleonard/Prescriber-ETL-data-pipeline

airflow-docker

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

https://github.com/anilkulkarni87/airflow-docker

Неплохая сравнительная табличка по инструментам metadata management

Awesome Data Discovery and Observability
https://github.com/opendatadiscovery/awesome-data-catalogs

2023 — Rockstar Data Engineer Roadmap
https://akaabachi.medium.com/2023-rockstar-data-engineer-roadmap-c990da37f625

Примеры из курса про Apache Airflow 2.0
https://github.com/adilkhash/apache-airflow-course-materials