Software Engineering for Absolute Beginners - 2021

What You Will Learn
🔹 Explore the concepts that you will encounter in the majority of companies doing software development
🔹 Create readable code that is neat as well as well-designed
🔹 Build code that is source controlled, containerized, and deployable
🔹 Secure your codebase
🔹 Optimize your workspace

🔥 Awesome Docker Compose samples

These samples provide a starting point for how to integrate different services using a Compose file and to manage their deployment with Docker Compose.

👉 @devops_dataops

https://github.com/docker/awesome-compose

GitHub - martandsingh/ApacheSpark: This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

https://github.com/martandsingh/ApacheSpark

Проектирование ETL-пайплайна в Apache Airflow / Хабр
https://habr.com/ru/company/otus/blog/679402/

Глубокое погружение в Data Quality / Хабр
https://habr.com/ru/company/vk/blog/674876/

Примерчик ETL pipeline на python
https://github.com/iamaziz/etl

Mara Pipelines

This package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. It has a number of baked-in assumptions/ principles:
- Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code.
- PostgreSQL as a data processing engine.
- Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines.
- GNU make semantics. Nodes depend on the completion of upstream nodes. No data dependencies or data flows.
- No in-app data processing: command line tools as the main tool for interacting with databases and data.
- Single machine pipeline execution based on Python's multiprocessing. No need for distributed task queues. Easy debugging and output logging.
- Cost based priority queues: nodes with higher cost (based on recorded run times) are run first.

https://github.com/mara/mara-pipelines

Open Source Guides

Open source software is made by people just like you. Learn how to launch and grow your project.

https://opensource.guide/

Automate without limits n8n
The workflow automation platform that doesn't box you in, that you never outgrow

GitHub 27k+

Usage
🔹 Learn how to install and use it from the command line
🔹 Learn how to run n8n in Docker

Self-Hosted -> Free
🔹 Data stays on your infrastructure
🔹 Open & extendable
🔹 One-line npm command or Docker deployment

Habr: n8n. Автоматизация ИБ со вкусом смузи

GitHub - ClickHouse/clickhouse-presentations: Presentations, meetups and talks about ClickHouse
https://github.com/ClickHouse/clickhouse-presentations

Репозиторий с проектами Data Engineering
https://github.com/san089/Udacity-Data-Engineering-Projects

Инженерия машинного обучения

Содержит множество рекомендаций и паттернов проектирования надежных и масштабируемых решений в области машинного обучения.

Odoo. Open Source Apps To Grow Your Business
https://github.com/odoo/odoo

Обработка больших данных с
Apache Spark– СПб: Университет ИТМО, 2019г.

Учебно-методическое пособие содержит теоретический материал и примеры выполнения задач для курса «Введение в технологии обработки больших данных». Пособие составлено с учётом проведения лабораторных работ с помощью фреймворка Apache Spark. Содержание дисциплины охватывает круг вопросов, связанных с организацией построения ETLконвейеров на основе Spark SQL и DataFrame API для распределенного выполнения на кластерных вычислительных система, включая использование итеративных вычислений, важных для машинного обучения, рассмотрения shuffle механизмов и принципов организации управлением памятью в Spark.
В результате освоения дисциплины студенты приобретают способности разработки программ и построения конвейеров обработки различных данных, навыки по работе с распределенными кластерными системами, а также способности к применению машинного обучения на распределенных наборах данных.

Перевод книги Problem Solving with Algorithms and Data Structures

https://aliev.me/runestone/index.html