What's the biggest danger when using MongoDB?
Anonymous Poll
24%
The db becoming a dump
34%
Messing up the indexing
42%
Using it but not knowing why
Also going to upload a Vlog later today 😉
New behind the scenes vlog: Got myself a iPhone 13. And quick overview of the topics I showed my students at the last weekly coaching meetup.
https://youtu.be/_fkf87ZrxWE
https://youtu.be/_fkf87ZrxWE
YouTube
New iPhone 13 unboxing and Weekly Coaching Meetup
Got myself a new iPhone 13 for filming my vlogs. Let's unbox it together.
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
And quick overview of the topics I showed my students at the last weekly coaching meetup.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from…
How would you feel if I announce a 25% Black Friday sale and you just bought my Data engineering academy full price
Anonymous Poll
48%
Hey, I'm happy for the new guys
17%
Ah, it's OK
27%
I would be annoyed
9%
I'd want a refund and never trust you again
!! Update !! I decided that there will be no Black Friday discount.
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
I heard a lot of comments and options here and on LinkedIn. I think it would be unfair to the many people who recently bought it full price.
I personally would be super annoyed too.
Not going to do it
If you want a really good data engineering education then check out my academy: https://learndataengineering.com/p/academy
Its a holistic approach to data engineering with a lot of courses inside. Very cheap compared to for instance Udemy's data engineering course.
I'm working every day on bringing more courses and improving the current ones.
We also have a private slack community without the Bitcoin scammers.
if you have any questions just let me know I'm here on telegram
Learndataengineering
Academy
By the way, today's video:
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
Problems with bulk importing data using Python https://youtu.be/QqCgjAJuPgM
YouTube
Problems with bulk importing data using Python
Bulk importing large files into a database at once can be problematic. Especially if you do it with a Python script that runs on a server. Here are a few problems you are going to run into.
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
► Learn Data Engineering at my Ultimate Data Engineering Academy!…
Hey, here's the link to today's video on YouTube: https://youtu.be/kZOEk7WOlw8
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything you need from basics to full example projects:
https://www.learndataengineering.com
YouTube
How to design an API with Swagger and OpenAPI
Creating APIs isn't really witchcraft. Tools like the OpenAPI specification and Swagger UI make it fairly easy to start (if you understand the basic concept). Here's an example of an API designed with Swagger that uses the OpenAPI specification.
► Learn…
► Learn…
Totally forgot to post the newest YouTube video:
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog https://youtu.be/Z5RL663nFhs
YouTube
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
Modern Data Warehouses like Snowflake, Redshift or BigQuery work great with Data Lakes. The benefit from this is that data does not need to be written into the warehouse. The warehouse can read data direct from the files laying around in the data lake. Here's…
Today's YouTube video:
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
https://youtu.be/Z5RL663nFhs
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
https://youtu.be/Z5RL663nFhs
YouTube
Modern Data Warehouse and Data Lake example with S3, Athena, Redshift, Glue Data Catalog
Modern Data Warehouses like Snowflake, Redshift or BigQuery work great with Data Lakes. The benefit from this is that data does not need to be written into the warehouse. The warehouse can read data direct from the files laying around in the data lake. Here's…
Post on our medium publication:
https://medium.com/plumbersofdatascience/how-use-data-engineering-to-process-the-results-of-a-data-maturity-assessment-8ffdb0f72e19
https://medium.com/plumbersofdatascience/how-use-data-engineering-to-process-the-results-of-a-data-maturity-assessment-8ffdb0f72e19
Medium
How use data engineering to process the results of a data maturity assessment
Please follow me on LinkedIn. This article was first published on the website of my data boutique: Xudo.
Did a small upgrade of the power plant.
Mini ITX life
Mini ITX life
Before with the old cpu and small cooler
NEW COURSE!!! Modern Data Warehouses no longer need you to load the data into them. Many warehouses like AWS Redshift, BigQuery or Snowflake allow you to load data directly from files in your Data Lake. This Data Lake integration is the key to flexibility of how you interact with your data. It makes a modern Data Warehouse so nice to use for all kinds of analytics workloads.
In this course you will learn how easy it is to use Data Lakes, Warehouses and BI tools. Load your files into the lake and visualize it in a report. 🚀
Course Contents
✅ Where Data Warehouses fit into a platform
✅ Data Warehouses ETL vs ELT data integration
✅ Direct Access of Data Lake?
✅ Data Warehouses and Data Lakes on AWS & GCP
✅ GCP hands on Example with Cloud Storage, BigQuery, & Data Studio
✅ AWS hands on example with S3, Glue, Athena, and Quicksight
✅ Example with AWS Redshift
https://learndataengineering.com/p/modern-data-warehouses
In this course you will learn how easy it is to use Data Lakes, Warehouses and BI tools. Load your files into the lake and visualize it in a report. 🚀
Course Contents
✅ Where Data Warehouses fit into a platform
✅ Data Warehouses ETL vs ELT data integration
✅ Direct Access of Data Lake?
✅ Data Warehouses and Data Lakes on AWS & GCP
✅ GCP hands on Example with Cloud Storage, BigQuery, & Data Studio
✅ AWS hands on example with S3, Glue, Athena, and Quicksight
✅ Example with AWS Redshift
https://learndataengineering.com/p/modern-data-warehouses
Learndataengineering
Building a Lakehouse on AWS and GCP
Moving everything to one cloud is most of the time not possible. Or should we just keep the data where it is and try to connect where possible? Could be a practical approach. Always use the right tool for the job, or is this too complicated?
Link to the video on YouTube: https://youtu.be/bsSUa1CrWqo
Link to the video on YouTube: https://youtu.be/bsSUa1CrWqo
YouTube
Is a multi-cloud strategy worth it?
Moving everything to one cloud is most of the time not possible. Or should we just keep the data where it is and try to connect where possible? Could be a practical approach.
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything…
► Learn Data Engineering at my Ultimate Data Engineering Academy!
Everything…
Cloud billing can be very difficult to forecast. There are so many variables in play. Are you looking for a solution? In this post @Zach Quinn is showing you a way how to use BigQuery's meta data to calculate and forecast the costs. Really cool idea! Makes me wonder what other services offer this kind of meta data, too.
Here is a quick view in the article:
Data engineers can leverage SQL statements to fetch database metadata in order to calculate costs incurred with PaaS products like BigQuery.
In the article you’ll learn:
-How to access table metadata in BigQuery
-How to use standard SQL to convert bytes to GB and TB
-How to calculate per gigabyte rates
Read "How Data Engineers Can Use SQL to Estimate BigQuery Storage Costs" in our publication "Plumbers of Data Science" on Medium.
https://medium.com/plumbersofdatascience/how-data-engineers-can-use-sql-to-estimate-bigquery-storage-costs-cbcdfca18899
Here is a quick view in the article:
Data engineers can leverage SQL statements to fetch database metadata in order to calculate costs incurred with PaaS products like BigQuery.
In the article you’ll learn:
-How to access table metadata in BigQuery
-How to use standard SQL to convert bytes to GB and TB
-How to calculate per gigabyte rates
Read "How Data Engineers Can Use SQL to Estimate BigQuery Storage Costs" in our publication "Plumbers of Data Science" on Medium.
https://medium.com/plumbersofdatascience/how-data-engineers-can-use-sql-to-estimate-bigquery-storage-costs-cbcdfca18899
Medium
How Data Engineers Can Use SQL to Estimate BigQuery Storage Costs
For data engineers, SQL’s applications go beyond analysis; it can be a powerful tool for determining resource allocations.