#java #data #data_engineering #data_orchestration #data_orchestrator #data_pipeline #dataflow #elt #etl #kestra #orchestration #pipeline #scheduler #workflow #workflow_automation #workflow_engine

https://github.com/kestra-io/kestra

#python #data_engineering #data_quality #data_quality_monitoring #data_science #database #databricks_sql #dataengineering #dataquality #mysql #oracle_database #postgres #postgresql #rdbms #snowflake #sql #trino

https://github.com/datafold/data-diff

#other #data_analysis #data_engineering #data_science #data_visualization #deep_learning #machine_learning #mathematics #probability #python #sql #statistics

https://github.com/Moataz-Elmesmary/Data-Science-Roadmap

#go #data #data_engineering #data_pipeline #data_stream_processing #data_streaming #golang #kubernetes #messaging_queue

https://github.com/memphisdev/memphis-broker

#javascript #csv #csv_export #csv_import #csv_parser #csv_parsing #csv_reader #data_analysis #data_engineering #flatfile #mongo #mongodb #nextjs #open_source #saas #streaming

https://github.com/yobulkdev/yobulkdev

#python #big_data #data_engineering #data_quality #data_science #feature_store #features #machine_learning #ml #mlops

https://github.com/feast-dev/feast

#rust #ckan #cli #csv #data_engineering #data_wrangling #datapackage #excel #geocode #luau #opendata #parquet #polars #postgresql #snappy #sql #sqlite #tsv

https://github.com/jqnatividad/qsv

#python #data #data_engineering #data_lake #data_loading #data_warehouse #elt #extract #load #transform

https://github.com/dlt-hub/dlt

#typescript #analytics #apache #apache_superset #asf #bi #business_analytics #business_intelligence #data_analysis #data_analytics #data_engineering #data_science #data_visualization #data_viz #flask #python #react #sql_editor #superset

Superset is a powerful business intelligence tool that helps you explore and visualize data easily. It offers a no-code interface for building charts, a robust SQL Editor for advanced queries, and support for nearly any SQL database or data engine. You can create beautiful visualizations, define custom dimensions and metrics quickly, and use a lightweight caching layer to reduce database load. Superset also provides extensible security roles and authentication options, an API for customization, and a cloud-native architecture designed for scale. This makes it easier to analyze and present your data in a user-friendly way, replacing or augmenting proprietary BI tools effectively.

https://github.com/apache/superset

#python #analytics #dagster #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #etl #metadata #mlops #orchestration #python #scheduler #workflow #workflow_automation

Dagster is a tool that helps you manage and automate your data workflows. You can define your data assets, like tables or machine learning models, using Python functions. Dagster then runs these functions at the right time and keeps your data up-to-date. It offers features like integrated lineage and observability, making it easier to track and manage your data. This tool is useful for every stage of data development, from local testing to production, and it integrates well with other popular data tools. Using Dagster, you can build reusable components, spot data quality issues early, and scale your data pipelines efficiently. This makes your work more productive and helps maintain control over complex data systems.

https://github.com/dagster-io/dagster

#python #airflow #apache #apache_airflow #automation #dag #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #elt #etl #machine_learning #mlops #orchestration #python #scheduler #workflow #workflow_engine #workflow_orchestration

Apache Airflow is a tool that helps you manage and automate workflows. You can write your workflows as code, making them easier to maintain, version, test, and collaborate on. Airflow lets you schedule tasks and monitor their progress through a user-friendly interface. It supports dynamic pipeline generation, is highly extensible, and scalable, allowing you to define your own operators and executors.

Using Airflow benefits you by making your workflows more organized, efficient, and reliable. It simplifies the process of managing complex tasks and provides clear visualizations of your workflow's performance, helping you identify and troubleshoot issues quickly. This makes it easier to manage data processing and other automated tasks effectively.

https://github.com/apache/airflow

#python #automation #data #data_engineering #data_ops #data_science #infrastructure #ml_ops #observability #orchestration #pipeline #prefect #python #workflow #workflow_engine

Prefect is a tool that helps you automate and manage data workflows in Python. It makes it easy to turn your scripts into reliable and flexible workflows that can handle unexpected changes. With Prefect, you can schedule tasks, retry failed operations, and monitor your workflows. You can install it using `pip install -U prefect` and start creating workflows with just a few lines of code. This helps data teams work more efficiently, reduce errors, and save time. You can also use Prefect Cloud for more advanced features and support.

https://github.com/PrefectHQ/prefect

#python #cleandata #data_engineering #data_profilers #data_profiling #data_quality #data_science #data_unit_tests #datacleaner #datacleaning #dataquality #dataunittest #eda #exploratory_analysis #exploratory_data_analysis #exploratorydataanalysis #mlops #pipeline #pipeline_debt #pipeline_testing #pipeline_tests

GX Core is a powerful tool for ensuring data quality. It allows you to write simple tests, called "Expectations," to check if your data meets certain standards. This helps teams work together more effectively and keeps everyone informed about the data's quality. You can automatically generate reports, making it easy to share results and preserve your organization's knowledge about its data. To get started, you just need to install GX Core in a Python virtual environment and follow some simple steps. This makes managing data quality much simpler and more efficient.

https://github.com/great-expectations/great_expectations

#rust #artificial_intelligence #big_data #data_engineering #distributed_computing #machine_learning #multimodal #python #rust

Daft is a powerful, easy-to-use data engine that lets you process large-scale data using Python or SQL with high speed and efficiency. It supports complex data types like images and tensors, works well interactively for quick data exploration, and can scale to huge cloud clusters using Ray. Daft integrates smoothly with cloud storage and data catalogs, making it ideal for data engineering, analytics, and machine learning workflows. By using Daft, you can handle big, multimodal datasets faster and more flexibly, improving your ability to analyze and prepare data for AI models without complex setup or slowdowns.

https://github.com/Eventual-Inc/Daft

#rust #ai #change_data_capture #context_engineering #data #data_engineering #data_indexing #data_infrastructure #data_processing #etl #hacktoberfest #help_wanted #indexing #knowledge_graph #llm #pipeline #python #rag #real_time #rust #semantic_search

**CocoIndex** is a fast, open-source Python tool (Rust core) for transforming data into AI formats like vector indexes or knowledge graphs. Define simple data flows in ~100 lines of code using plug-and-play blocks for sources, embeddings, and targets—install via `pip install cocoindex`, add Postgres, and run. It auto-syncs fresh data with minimal recompute on changes, tracking lineage. **You save time building scalable RAG/semantic search pipelines effortlessly, avoiding complex ETL and stale data issues for production-ready AI apps.**

https://github.com/cocoindex-io/cocoindex