Data Engineering Expert Reacts to Orange Theory's AWS Infrastructure
I am speaking at the online conference DATAcated Expo tomorrow Tuesday, 5th Oct 2021 at 12.30pm ET.
My talk will (of course) be about Data Engineering and us Plumbers of Data Science.
Join me tomorrow and together with 23 other great speakers, we will make the DataCated Expo something special!
https://datacated.com/expo/
My talk will (of course) be about Data Engineering and us Plumbers of Data Science.
Join me tomorrow and together with 23 other great speakers, we will make the DataCated Expo something special!
https://datacated.com/expo/
Hey everyone, I just added a new course to the Data Engineering Academy: Storing & visualizing time series data with InfluxDB and Grafana. Check it out:
https://learndataengineering.com/p/time-series-influxdb-grafana
https://learndataengineering.com/p/time-series-influxdb-grafana
Learndataengineering
Time series processing with InfluxDB & Grafana
Complete example project with code and over 2hours of videos
New YouTube video:
Which one is better? Data Architect or Data Engineer?
There's no real better. As the name suggests, the Architect is more concerned with the planning of pipelines, data models and the platform. The Engineer is more hands on and has a lot to do with development and operations.
I made an analysis of two job descriptions: One for a Data Architect and one for a Data Engineer.
⚠️ Watch this video of my weekly Data Engineering Coaching meeting:
Link: https://youtu.be/rjvH1m4JNmA
Which one is better? Data Architect or Data Engineer?
There's no real better. As the name suggests, the Architect is more concerned with the planning of pipelines, data models and the platform. The Engineer is more hands on and has a lot to do with development and operations.
I made an analysis of two job descriptions: One for a Data Architect and one for a Data Engineer.
⚠️ Watch this video of my weekly Data Engineering Coaching meeting:
Link: https://youtu.be/rjvH1m4JNmA
YouTube
Job Analysis: Data Architect or Data Engineer which one is best?
This is a video of my weekly Data Engineering Coaching meeting. I showed two job descriptions: One for a Data Architect and one for Data Engineer. One of my students were asking during the week which one is better. There's no real better. As the name suggests…
New video: IMPORTANT: Watch this before you use APIs and Message Queues!
https://youtu.be/rKs-BS7Qvv8
https://youtu.be/rKs-BS7Qvv8
YouTube
IMPORTANT: Watch this before you use APIs and Message Queues!
You want to build streaming data pipelines using APIs and Message Queues? Check out these design tricks. I'll also show you an important drawback using queues that are going to "break" your http request. Once your data is in the Queue the API can no longer…
New video on YouTube:
https://youtu.be/DuwMwEavHmE
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
https://youtu.be/DuwMwEavHmE
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
YouTube
How to use Kafka Connect
Most of us know Apache Kafka for being a message queue, but there are more features.
Confluent has a few really cool additional features like Kafka Connect for data integration. With connect Kafka can connect to sources, get the data, transform it and send…
Confluent has a few really cool additional features like Kafka Connect for data integration. With connect Kafka can connect to sources, get the data, transform it and send…
Do you think Kafka Connect is an alternative to Airflow?
Anonymous Poll
24%
Yes, for streaming data
14%
Maybe
24%
No
39%
Just show results
What's the biggest danger when using MongoDB?
Anonymous Poll
24%
The db becoming a dump
34%
Messing up the indexing
42%
Using it but not knowing why
Also going to upload a Vlog later today 😉
New behind the scenes vlog: Got myself a iPhone 13. And quick overview of the topics I showed my students at the last weekly coaching meetup.
https://youtu.be/_fkf87ZrxWE
https://youtu.be/_fkf87ZrxWE
YouTube
New iPhone 13 unboxing and Weekly Coaching Meetup
Got myself a new iPhone 13 for filming my vlogs. Let's unbox it together.
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
How would you feel if I announce a 25% Black Friday sale and you just bought my Data engineering academy full price
Anonymous Poll
48%
Hey, I'm happy for the new guys
17%
Ah, it's OK
27%
I would be annoyed
9%
I'd want a refund and never trust you again
!! Update !! I decided that there will be no Black Friday discount.
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
Learndataengineering
Academy
By the way, today's video:
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
YouTube
Problems with bulk importing data using Python
Bulk importing large files into a database at once can be problematic. Especially if you do it with a Python script that runs on a server. Here are a few problems you are going to run into.
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
Hey, here's the link to today's video on YouTube: https://youtu.be/kZOEk7WOlw8
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
YouTube
How to design an API with Swagger and OpenAPI
Creating APIs isn't really witchcraft. Tools like the OpenAPI specification and Swagger UI make it fairly easy to start (if you understand the basic concept). Here's an example of an API designed with Swagger that uses the OpenAPI specification.
► Learn…
► Learn…
Totally forgot to post the newest YouTube video:
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
YouTube
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
Modern Data Warehouses like Snowflake, Redshift or BigQuery work great with Data Lakes. The benefit from this is that data does not need to be written into the warehouse. The warehouse can read data direct from the files laying around in the data lake. Here's…