New video: IMPORTANT: Watch this before you use APIs and Message Queues!
https://youtu.be/rKs-BS7Qvv8
https://youtu.be/rKs-BS7Qvv8
YouTube
IMPORTANT: Watch this before you use APIs and Message Queues!
You want to build streaming data pipelines using APIs and Message Queues? Check out these design tricks. I'll also show you an important drawback using queues that are going to "break" your http request. Once your data is in the Queue the API can no longer…
New video on YouTube:
https://youtu.be/DuwMwEavHmE
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
https://youtu.be/DuwMwEavHmE
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
YouTube
How to use Kafka Connect
Most of us know Apache Kafka for being a message queue, but there are more features.
Confluent has a few really cool additional features like Kafka Connect for data integration. With connect Kafka can connect to sources, get the data, transform it and send…
Confluent has a few really cool additional features like Kafka Connect for data integration. With connect Kafka can connect to sources, get the data, transform it and send…
Do you think Kafka Connect is an alternative to Airflow?
Anonymous Poll
24%
Yes, for streaming data
14%
Maybe
24%
No
39%
Just show results
What's the biggest danger when using MongoDB?
Anonymous Poll
24%
The db becoming a dump
34%
Messing up the indexing
42%
Using it but not knowing why
Also going to upload a Vlog later today 😉
New behind the scenes vlog: Got myself a iPhone 13. And quick overview of the topics I showed my students at the last weekly coaching meetup.
https://youtu.be/_fkf87ZrxWE
https://youtu.be/_fkf87ZrxWE
YouTube
New iPhone 13 unboxing and Weekly Coaching Meetup
Got myself a new iPhone 13 for filming my vlogs. Let's unbox it together.
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
How would you feel if I announce a 25% Black Friday sale and you just bought my Data engineering academy full price
Anonymous Poll
48%
Hey, I'm happy for the new guys
17%
Ah, it's OK
27%
I would be annoyed
9%
I'd want a refund and never trust you again
!! Update !! I decided that there will be no Black Friday discount.
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
Learndataengineering
Academy
By the way, today's video:
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
YouTube
Problems with bulk importing data using Python
Bulk importing large files into a database at once can be problematic. Especially if you do it with a Python script that runs on a server. Here are a few problems you are going to run into.
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
Hey, here's the link to today's video on YouTube: https://youtu.be/kZOEk7WOlw8
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
YouTube
How to design an API with Swagger and OpenAPI
Creating APIs isn't really witchcraft. Tools like the OpenAPI specification and Swagger UI make it fairly easy to start (if you understand the basic concept). Here's an example of an API designed with Swagger that uses the OpenAPI specification.
► Learn…
► Learn…
Totally forgot to post the newest YouTube video:
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
YouTube
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
Modern Data Warehouses like Snowflake, Redshift or BigQuery work great with Data Lakes. The benefit from this is that data does not need to be written into the warehouse. The warehouse can read data direct from the files laying around in the data lake. Here's…
Today's YouTube video:
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
https://youtu.be/Z5RL663nFhs
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
https://youtu.be/Z5RL663nFhs
YouTube
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
Modern Data Warehouses like Snowflake, Redshift or BigQuery work great with Data Lakes. The benefit from this is that data does not need to be written into the warehouse. The warehouse can read data direct from the files laying around in the data lake. Here's…
Post on our medium publication:
https://medium.com/plumbersofdatascience/how-use-data-engineering-to-process-the-results-of-a-data-maturity-assessment-8ffdb0f72e19
https://medium.com/plumbersofdatascience/how-use-data-engineering-to-process-the-results-of-a-data-maturity-assessment-8ffdb0f72e19
Medium
How use data engineering to process the results of a data maturity assessment
Please follow me on LinkedIn. This article was first published on the website of my data boutique: Xudo.
Did a small upgrade of the power plant.
Mini ITX life
Mini ITX life
Before with the old cpu and small cooler