Post about data engineering. I also answered comments here

https://www.linkedin.com/posts/christinastathopoulos_truth-dataengineering-datascience-activity-6878371323064684544-lN3K

I love the smell of some nice SQL in the morning! I keep telling you for structured data use cases SQL was and still is the gold standard. Here's another great article from Zach creating tables, doing nested queries and more.

Preview on the content:
- How to effectively use subqueries to structure complex SQL queries
- How to use dynamic date filters to avoid hard coding date ranges 
- How to use conditional logic to return a decision

Read "Data Engineering IRL: How to Use SQL to Track Your Spending" in our publication "Plumbers of Data Science" on Medium.

https://medium.com/plumbersofdatascience/date-engineering-irl-how-to-use-sql-to-track-your-spending-79f47512af2b

Check out Benjamin's post about 2022 predictions for Data Engineering. I also sent him my 2 cents.
Make sure that you follow him on Linkedin and on YouTube. He does really great videos (I'm a bit jealous).
Ben also just quit his job at Facebook to start his own company.
BIG CONGRATS!!!!

https://seattledataguy.substack.com/p/5-big-data-experts-predictions-for

I’m trying out Rockset real-time analytics database today. Join me live on YouTube or LinkedIn:

https://youtu.be/0xdQNheAGe8

https://www.linkedin.com/posts/andreas-kretz_quick-test-with-example-project-activity-6884398644326490113-epN6

I just released a NEW Course Python for Data Engineers!

You come from a different field and haven't coded before? No problem! This course is picking up where our Python 1 course from @Amit Jain has ended.

You learn all the important basics a Data Engineer needs. From advanced Python features, how to transform data with pandas to Working with APIs and Postgres databases.

Kristijan Bakaric and I created hands on examples for every lesson. In 2.5 hours of videos we go through each of them together. We also prepared the source codes in our GitHub.
🚀

Course Content:
✅ Exception handling
✅ Understand what classes and objects are
✅ how to use modules
✅ Log out messages into files
✅ how to work with dates and JSONs
✅ Understand unit tests and data validation
✅ Pandas to transform your data
✅ Numpy to apply mathematical functions
✅ Working with Postgres

https://learndataengineering.com/p/python-for-data-engineers

The new Machine Learning & Containerization on AWS project is online!! 🚀
As always, active members oft the Data Engineering Academy already have access to the course.

What’s the course about:
In this example project you learn how to create a data pipeline where you pull data from the Twitter API, analyze, store and visualize it.
You will host your Machine Learning algorithm on AWS using Lambda and setup your own postgres database with RDS. You create a Streamlit dashboard and gain experience hosting it using Elastic Container Registry (ECR) and Elastic Container Service (ECS). 

This project also gives you insights on how to handle dependency management with Poetry.
Have fun!

Course Content:
✅ Setup and configure Twitter API
✅ Launching RDS Postgres DB
✅ Create S3 bucket for raw storage
✅ Create ML Lambda that extracts & analyses Tweets
✅ Schedule Lambda with Event Bridge
✅ Create Streamlit visualization app
✅ Dependency management with Poetry & create Docker image
✅ Install & configure AWS CLI
✅ Setup Elastic Container Registry ECR
✅ Create Elastic Container Service ECS Fargate cluster
✅ Run our Streamlit app as ECS task

Learn how to build a NLP pipeline and host containers in the cloud

Link to the course in the Data Engineering Academy - Trusted by over 500 students: https://learndataengineering.com/p/ml-on-aws

I was live today: How to analyze Tweet sentiments with ML on AWS

https://youtu.be/FMHa1eCj4k0

Join the Data Engineering Discord server! https://discord.gg/Wxy2mQA7Fy

Chat with Andreas how the Academy and the Full-Stack Coaching can help you:
https://t.me/+V87Cha4h1_VGR9ac

New Podcast episode!

In today's episode, I’m talking with Tom Schamberger from msg. He leads their cloud data platform team and brings a ton of experience from consulting, startups, and platform design.
We talked about:
✅ The skills data engineers actually need in consulting
✅ Why soft skills are underrated
✅ Tool debates (Databricks, Snowflake, SAP…)
✅ What consulting projects really look like
and more!

Listen to the episode here: https://creators.spotify.com/pod/show/andreaskayy/episodes/125-These-Skills-Get-You-a-Data-Consulting-Job--with-Tom-Schamberger-e35at82
Or watch on YouTube: https://youtu.be/jWaVtLNYNIw

Tomorrow we are going to look into streaming with Kafka.
How to creat streaming pipelines, explore the data and debug the pipelines as well.
It's oging to be super interesting

https://youtube.com/live/Nx1Tl4gta6Q?feature=share

Some people say I don't do free content. 🤡
Here's YouTube video #477 and a FREE playground app to experience Kafka and Spark streaming.

I created this playground to show you what happens when:
- You increase the produced messages
- Kafka runs out of resources
- Spark runs out of resources
- The database you use get's in trouble and overpowered

It will help you understand the right variables to look at if you have problems.

🖥️ Watch the video here: https://youtu.be/ScG4spWqMj4?si=2Q-v8pE1nF40bJqj
🛝 Check out the playground app on GitHub Pages: https://bit.ly/4mbsLqY

🎥 The whole topics fits also very well to our FREE live stream tomorrow:
Add it to your calendar: https://www.addevent.com/event/SR26261795

Have fun and let me know what you think about the playground in the comments.

PS: I completely vibe coded this app, so there are still a lot of things missing or not 100% right. Let me know your opinions. If enough people like the playground I'll continue extending it 🙂

Is Cursor Really Worth The Price Or Is ChatGPT better?
People constantly tell me to try out Cursor. So I did and...

I'm not sold.
Especially not with the current rate limiting 😬

⚠️ Check out why in today's video: https://youtu.be/fPURYpawNQA

How about you?