#python #airflow #airflow_operators #aws #aws_ec2 #aws_s3 #aws_sdk #cassandra #cassandra_database #cloudformation #cluster #data #data_engineering #data_engineering_pipeline #data_lake #data_modeling #data_warehouse #etl_pipeline #infrastructure #postgres #postgresql_database

https://github.com/san089/Udacity-Data-Engineering-Projects

#other #applied_data_science #applied_machine_learning #data_engineering #data_science #machine_learning #nlp #papers #production #recommendation #recsys #reinforcement_learning #search

https://github.com/eugeneyan/applied-ml

#other #data #data_engineering #engineering

https://github.com/oleg-agapov/data-engineering-book

#python #cleandata #data_engineering #data_profilers #data_profiling #data_quality #data_science #data_unit_tests #datacleaner #datacleaning #dataquality #dataunittest #eda #exploratory_analysis #exploratory_data_analysis #exploratorydataanalysis #mlops #pipeline #pipeline_debt #pipeline_testing #pipeline_tests

https://github.com/great-expectations/great_expectations

#typescript #ab_testing #abtest #abtesting #analytics #bigquery #clickhouse #continuous_delivery #data_analysis #data_engineering #data_science #experimentation #feature_flagging #feature_flags #mixpanel #redshift #remote_config #snowflake #split_testing #statistics

https://github.com/growthbook/growthbook

#python #data_engineering #data_science #jupyter #jupyter_notebooks #machine_learning #mlops #papermill #pipelines #pycharm #vscode #workflow

https://github.com/ploomber/ploomber

#scala #data_engineering #data_science #deep_learning #feature_engineering #feature_extraction #kubernetes #machine_learning #personalization #ranking #search

https://github.com/metarank/metarank

#java #data #data_engineering #data_orchestration #data_orchestrator #data_pipeline #dataflow #elt #etl #kestra #orchestration #pipeline #scheduler #workflow #workflow_automation #workflow_engine

https://github.com/kestra-io/kestra

#python #data_engineering #data_quality #data_quality_monitoring #data_science #database #databricks_sql #dataengineering #dataquality #mysql #oracle_database #postgres #postgresql #rdbms #snowflake #sql #trino

https://github.com/datafold/data-diff

#other #data_analysis #data_engineering #data_science #data_visualization #deep_learning #machine_learning #mathematics #probability #python #sql #statistics

https://github.com/Moataz-Elmesmary/Data-Science-Roadmap

#go #data #data_engineering #data_pipeline #data_stream_processing #data_streaming #golang #kubernetes #messaging_queue

https://github.com/memphisdev/memphis-broker

#javascript #csv #csv_export #csv_import #csv_parser #csv_parsing #csv_reader #data_analysis #data_engineering #flatfile #mongo #mongodb #nextjs #open_source #saas #streaming

https://github.com/yobulkdev/yobulkdev

#python #big_data #data_engineering #data_quality #data_science #feature_store #features #machine_learning #ml #mlops

https://github.com/feast-dev/feast

#rust #ckan #cli #csv #data_engineering #data_wrangling #datapackage #excel #geocode #luau #opendata #parquet #polars #postgresql #snappy #sql #sqlite #tsv

https://github.com/jqnatividad/qsv

#python #data #data_engineering #data_lake #data_loading #data_warehouse #elt #extract #load #transform

https://github.com/dlt-hub/dlt

#typescript #analytics #apache #apache_superset #asf #bi #business_analytics #business_intelligence #data_analysis #data_analytics #data_engineering #data_science #data_visualization #data_viz #flask #python #react #sql_editor #superset

Superset is a powerful business intelligence tool that helps you explore and visualize data easily. It offers a no-code interface for building charts, a robust SQL Editor for advanced queries, and support for nearly any SQL database or data engine. You can create beautiful visualizations, define custom dimensions and metrics quickly, and use a lightweight caching layer to reduce database load. Superset also provides extensible security roles and authentication options, an API for customization, and a cloud-native architecture designed for scale. This makes it easier to analyze and present your data in a user-friendly way, replacing or augmenting proprietary BI tools effectively.

https://github.com/apache/superset

#python #analytics #dagster #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #etl #metadata #mlops #orchestration #python #scheduler #workflow #workflow_automation

Dagster is a tool that helps you manage and automate your data workflows. You can define your data assets, like tables or machine learning models, using Python functions. Dagster then runs these functions at the right time and keeps your data up-to-date. It offers features like integrated lineage and observability, making it easier to track and manage your data. This tool is useful for every stage of data development, from local testing to production, and it integrates well with other popular data tools. Using Dagster, you can build reusable components, spot data quality issues early, and scale your data pipelines efficiently. This makes your work more productive and helps maintain control over complex data systems.

https://github.com/dagster-io/dagster

#python #airflow #apache #apache_airflow #automation #dag #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #elt #etl #machine_learning #mlops #orchestration #python #scheduler #workflow #workflow_engine #workflow_orchestration

Apache Airflow is a tool that helps you manage and automate workflows. You can write your workflows as code, making them easier to maintain, version, test, and collaborate on. Airflow lets you schedule tasks and monitor their progress through a user-friendly interface. It supports dynamic pipeline generation, is highly extensible, and scalable, allowing you to define your own operators and executors.

Using Airflow benefits you by making your workflows more organized, efficient, and reliable. It simplifies the process of managing complex tasks and provides clear visualizations of your workflow's performance, helping you identify and troubleshoot issues quickly. This makes it easier to manage data processing and other automated tasks effectively.

https://github.com/apache/airflow