Architecture Weeekly #68 Highlights
This week I held the interview with Vitaly Sharovatov. We discussed the team dynamics, what managers can do to improve team performance and back it all up with scientific papers! Watch it here.
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy and Nadia for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Highlights
Datadog long-awaited postmortem π·ββοΈ
Datadog had a 24 hours long outage on March, 8th. Datadog being an observability company was kinda expected to publish the postmortem soon enough, but 2 months later there was nothing published. Some researchers ever tried to write their own version, but luckily the company decided to publish the PM themselves. Read a fascinating story on how Linux upgrades can get you down even if you're deployed to 3 different cloud providers for reliability.
#pm #reliability #upgrade
What Happens When You Type an URL Into Your Browser? π·ββοΈ
I remember several years ago I was going through an interview in Amazon. After the questions about the cloud advantages, the interviewer asked the questions in the title. And I think I managed to do pretty well: I described the 21h interruption, the events in the operating system, the DNS stuff including local caches, HTTP protocol... There was no second interview. So in case you get the same question - get the answer!
#systemdesign
How to run a Decision-Making Architecture Board πΌ
The autonomy of decisions in team is a good thing; however if the organization just allows everybody to do whatever they want, it soon will face a zoo of technologies and approaches. So at some point it makes sense to have a board where at least those decisions can be discussed. How to create and run one? Read a guest post in the blog of Olad Zommermann!
#adr #architectureboard
This week I held the interview with Vitaly Sharovatov. We discussed the team dynamics, what managers can do to improve team performance and back it all up with scientific papers! Watch it here.
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy and Nadia for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Highlights
Datadog long-awaited postmortem π·ββοΈ
Datadog had a 24 hours long outage on March, 8th. Datadog being an observability company was kinda expected to publish the postmortem soon enough, but 2 months later there was nothing published. Some researchers ever tried to write their own version, but luckily the company decided to publish the PM themselves. Read a fascinating story on how Linux upgrades can get you down even if you're deployed to 3 different cloud providers for reliability.
#pm #reliability #upgrade
What Happens When You Type an URL Into Your Browser? π·ββοΈ
I remember several years ago I was going through an interview in Amazon. After the questions about the cloud advantages, the interviewer asked the questions in the title. And I think I managed to do pretty well: I described the 21h interruption, the events in the operating system, the DNS stuff including local caches, HTTP protocol... There was no second interview. So in case you get the same question - get the answer!
#systemdesign
How to run a Decision-Making Architecture Board πΌ
The autonomy of decisions in team is a good thing; however if the organization just allows everybody to do whatever they want, it soon will face a zoo of technologies and approaches. So at some point it makes sense to have a board where at least those decisions can be discussed. How to create and run one? Read a guest post in the blog of Olad Zommermann!
#adr #architectureboard
YouTube
Team Dynamics with Vitaly Sharovatov
Software Engineering teams can be frequently rather the working groups than actual teams. We will discuss with Vitaly Sharovatov what the team actually is and how managers can help building them or at least not harm them.
https://vvsevolodovich.dev
Patreon:β¦
https://vvsevolodovich.dev
Patreon:β¦
π1
Architecture Weekly #68 Follow-up
Raft does not guarantee liveness in the face of Network Faults πΌ
Well, Raft as one of the consensus algorithms should guarantee the leader election during network faults. This post(but rather old one) showcased the 2 cases where the leader will not be able to be elected. The fixes suggested in the article as well, so take a closer look.
#distributedsystem #raft #consensus
Core Solution Architecture Methods π·ββοΈ
I am sharing an article from the Solution Architecture training. In this chapter the shared vision is considered: what you actually need to do in order to share the understanding of the system including defining boundaries, external interfaces, internal components etc. Get more details inside!
#architecture #documentation
Hotspot performance engineering fails πΌ
Some companies believe that software can be fast, if you find some hotspots in the code and optimize those. But as an architect, you can easily guess that enormous performance problems happen from inappropriate architecture. Daniel Lemire explains it in little more detail.
#performance #pareto
Postgres Superpowers in Practice π·ββοΈ
Postgres being a universal database for the majority of small and medium enterprises gets supported by the post of Oskar Dudycz, where he demonstrates how you can turn PostgreSQL into a multimodal database using the extensions. Look, how easy to convert it for example into a time-series db!
#db #postgres #timeseries
I built an AI Avatars Generator using Stable Diffusion π·ββοΈ
AI is on hype here. My colleague from Bolt wrote a blog post how he made his own AI Avatars Generator. He describes the request ingestion, cron jobs, model deployment and training and provides the architecture he used. Follow the post!
#ai #ml
Connecting Block Business Units with AWS API Gateway π€
Company acquisition or merging can be a tricky process from a technical perspective. Different ecosystems, programming languages, deployment and runtime approaches are among those complexities. However, Block(which is an owning company for Square and Cash App) does it almost on a regular basis. Find a very thorough post on how AWS API Gateway and Fargate help them integrate new companies into their infrastructure with minimal possible effort.
#integration #cloud #aws
Raft does not guarantee liveness in the face of Network Faults πΌ
Well, Raft as one of the consensus algorithms should guarantee the leader election during network faults. This post(but rather old one) showcased the 2 cases where the leader will not be able to be elected. The fixes suggested in the article as well, so take a closer look.
#distributedsystem #raft #consensus
Core Solution Architecture Methods π·ββοΈ
I am sharing an article from the Solution Architecture training. In this chapter the shared vision is considered: what you actually need to do in order to share the understanding of the system including defining boundaries, external interfaces, internal components etc. Get more details inside!
#architecture #documentation
Hotspot performance engineering fails πΌ
Some companies believe that software can be fast, if you find some hotspots in the code and optimize those. But as an architect, you can easily guess that enormous performance problems happen from inappropriate architecture. Daniel Lemire explains it in little more detail.
#performance #pareto
Postgres Superpowers in Practice π·ββοΈ
Postgres being a universal database for the majority of small and medium enterprises gets supported by the post of Oskar Dudycz, where he demonstrates how you can turn PostgreSQL into a multimodal database using the extensions. Look, how easy to convert it for example into a time-series db!
#db #postgres #timeseries
I built an AI Avatars Generator using Stable Diffusion π·ββοΈ
AI is on hype here. My colleague from Bolt wrote a blog post how he made his own AI Avatars Generator. He describes the request ingestion, cron jobs, model deployment and training and provides the architecture he used. Follow the post!
#ai #ml
Connecting Block Business Units with AWS API Gateway π€
Company acquisition or merging can be a tricky process from a technical perspective. Different ecosystems, programming languages, deployment and runtime approaches are among those complexities. However, Block(which is an owning company for Square and Cash App) does it almost on a regular basis. Find a very thorough post on how AWS API Gateway and Fargate help them integrate new companies into their infrastructure with minimal possible effort.
#integration #cloud #aws
Decentralized Thoughts
Raft does not Guarantee Liveness in the face of Network Faults
Last month, Cloudflare published a postmortem of a recent 6-hour outage caused by a partial switch failure which left etcd unavailable as it was unable to establish a stable leader. This outage has understandably led to discussion online about exactly whatβ¦
β€2π1
π₯ Developer Advocates Vs Product Managers
https://youtube.com/shorts/asDydhWXBDI?feature=share
https://youtube.com/shorts/asDydhWXBDI?feature=share
YouTube
Developer Advocates π₯ vs Product Managers #shorts
#productmanagement #devrel #prioritization #priority
Architecture Weekly #69
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy and Nadia for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Highlights
Understanding database indexes in PostgreSQL π€
You have definitely heard about indexes in databases - they make our queries fast. But how exactly? What different types of indexes are out there? What are the ones supported by your database engine? Find out a detailed piece on indexes in PostrgreSQL with nice illustrations insde.
#db #index #performanse
Container Loading in AWS Lambda π€
This week a new paper has been published on the long-awaited container support for AWS Lambda. In this article Marc Brooker dissect this paper and explains the biggest problem with supporting containers: performance. One of the tactics for performance is caching, but in case of containers it can become tricky. Follow the article to know about how much unique bytes a typical container has π
#paper #aws #serverless
The State of Frontend 2022 πΌ
It's always interesting to see the surveys on the parts of the IT industry. This time I am happy to share the State of Frontend Report which covers tons of topics from developer's work conditions to frameworks, static-site generators, hosting, micro-frontends and the future of Frontend itself. Nicely built report inside.
#frontend #report
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy and Nadia for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Highlights
Understanding database indexes in PostgreSQL π€
You have definitely heard about indexes in databases - they make our queries fast. But how exactly? What different types of indexes are out there? What are the ones supported by your database engine? Find out a detailed piece on indexes in PostrgreSQL with nice illustrations insde.
#db #index #performanse
Container Loading in AWS Lambda π€
This week a new paper has been published on the long-awaited container support for AWS Lambda. In this article Marc Brooker dissect this paper and explains the biggest problem with supporting containers: performance. One of the tactics for performance is caching, but in case of containers it can become tricky. Follow the article to know about how much unique bytes a typical container has π
#paper #aws #serverless
The State of Frontend 2022 πΌ
It's always interesting to see the surveys on the parts of the IT industry. This time I am happy to share the State of Frontend Report which covers tons of topics from developer's work conditions to frameworks, static-site generators, hosting, micro-frontends and the future of Frontend itself. Nicely built report inside.
#frontend #report
boosty.to
Vladimir Ivanov - Author of Architecture Weekly newsletter
Exclusive content from Vladimir Ivanov, subscribe and be the first to access!
π₯1
Follow-Up
Apache Flink Architecture π€
Apache Flink is a robust stream processing framework used at Airbnb or Uber. Aurimas GriciΕ«nas posted a long read on the internal structure of Apache Flink so that you can know what is a Flink Program, what's the difference between JobManager and TaskManager and how to get it all going in HA mode.
#bigdata
How to manage your technical backlog π·ββοΈ
Technical debt is probably the second thing a developer knows after "Hello, world!" as starts right at the same moment. Despite having millions of articles and books on refactoring, the architecture or say manager perspective is rarely presented. Fixing it now with the article by DarΓo RodrΔ«guez on managing technical debt including cost-benefit analysis and other tactics.
#quality #process
Scaling Salt for Remote Execution to support LinkedIn Infra Growth π·ββοΈ
LinkedIn is changing infrastructure using the opensource Salt engine. But there are different ways how you can run it. At some point they realized their current setup didn't scale well, so they decided to change the architecture slightly to unleash the full power of the instrument. Grab a case study inside!
#casestudy #architecture
How to Architect Android Apps π·ββοΈ
I remember the times when there was no such thing as a Mobile Application Architecture - you would just put some code right into your UI Controller, make a couple of swear words on the network operations on main thread and you're done. Modern mobile applications are enormous with millions of LOCs, and you need a good architecture for them. Grab an article with Android application architecture, principles and a comparison of modern Google approach with good old Clean Architecture.
#architecture #mobile
ChatGPT already knows πΌ
Uwe Friedrechsen dropped another series on the so discussed ChatGPT and if the chatbot will replace professional programmers. We really do not want it to happen(and I even recorded an episode on the topic), but Uwe is not that optimistic. And the simple reason is that you learn at a very limited speed... and ChatGPT learns instantly. Think about it!
#phylosophy
CNCF Platforms White Paper πΌ
And another time - about platform engineering. Cloud Native Computing Foundation published a whitepaper on the platforms, explaining why they are important in today's computing world, what exactly we should call a platform and what are the biggest challenges of them.
#platform #cloud
Apache Flink Architecture π€
Apache Flink is a robust stream processing framework used at Airbnb or Uber. Aurimas GriciΕ«nas posted a long read on the internal structure of Apache Flink so that you can know what is a Flink Program, what's the difference between JobManager and TaskManager and how to get it all going in HA mode.
#bigdata
How to manage your technical backlog π·ββοΈ
Technical debt is probably the second thing a developer knows after "Hello, world!" as starts right at the same moment. Despite having millions of articles and books on refactoring, the architecture or say manager perspective is rarely presented. Fixing it now with the article by DarΓo RodrΔ«guez on managing technical debt including cost-benefit analysis and other tactics.
#quality #process
Scaling Salt for Remote Execution to support LinkedIn Infra Growth π·ββοΈ
LinkedIn is changing infrastructure using the opensource Salt engine. But there are different ways how you can run it. At some point they realized their current setup didn't scale well, so they decided to change the architecture slightly to unleash the full power of the instrument. Grab a case study inside!
#casestudy #architecture
How to Architect Android Apps π·ββοΈ
I remember the times when there was no such thing as a Mobile Application Architecture - you would just put some code right into your UI Controller, make a couple of swear words on the network operations on main thread and you're done. Modern mobile applications are enormous with millions of LOCs, and you need a good architecture for them. Grab an article with Android application architecture, principles and a comparison of modern Google approach with good old Clean Architecture.
#architecture #mobile
ChatGPT already knows πΌ
Uwe Friedrechsen dropped another series on the so discussed ChatGPT and if the chatbot will replace professional programmers. We really do not want it to happen(and I even recorded an episode on the topic), but Uwe is not that optimistic. And the simple reason is that you learn at a very limited speed... and ChatGPT learns instantly. Think about it!
#phylosophy
CNCF Platforms White Paper πΌ
And another time - about platform engineering. Cloud Native Computing Foundation published a whitepaper on the platforms, explaining why they are important in today's computing world, what exactly we should call a platform and what are the biggest challenges of them.
#platform #cloud
Swirlai
SAI Notes #03: Apache Flink - Architecture.
Let's take a closer look into internals of Apache Flink.
π₯2
Intro to Frugal Software Architecture
https://vvsevolodovich.dev/introducing-frugal-software-architecture-intro/
https://vvsevolodovich.dev/introducing-frugal-software-architecture-intro/
Vladimir Ivanov Dev Blog
Frugal Software Architecture: Balancing Strategic Investments and Cost Optimization. Part 1: Intro
Imagine the scenario of countless startups that secure substantial investments only to squander them on expensive hires and extravagant resources without much thought. These companies, lured by the promise of rapid growth and success, fail to assess the longβ¦
π₯2π1π1
Book your calendard! Speaking about Data Engineering with Pasha Finkelshteyn
https://www.youtube.com/watch?v=GFZd0UjFxUQ
https://www.youtube.com/watch?v=GFZd0UjFxUQ
YouTube
Data Engineering with Pasha Finkelshteyn
This interview aims to explore the field of data engineering, focusing on the roles, responsibilities, and technical skills required in the profession. The questions cover various aspects of data engineering, including data processing, data pipelines, ETLβ¦
π₯3
Architecture Weekly #70
Evolutionary Architecture from an Organizational Perspective πΌ
Evolutionary Architecture frequently only considered from the technical point of view, forgetting about the the fact that without the organization support evolving architecture is barely feasible task. The article on InfoQ blog discusses the necessity of IT and Business working as one team, alignment on company-wide goals and the importance of contracts within the organization for achieving truly evolutionary architecture.
#architecture
An educational side project π·ββοΈ
Have you tried building a product like Uber or Bolt on your own? Juraj Majerik decided to try! Across the span of 7 months he configured the infrastructure, built the backend, frontend and the load emulator and created the demo for the system, learning a lot meanwhile. Gergely Orosz describes his work and the main takeaways from such experience alongside with the system architecture.
#casestudy
Frugal Software Architecture πΌ
This week I wrote an article myself. I want to introduce you to the concept of Frugal Software Architecture. But it's not a mere cost efficiency metric of the software, but rather a set of principles allowing to balance strategic investments and cost optimizations. Find them inside.
#architecture #frugality
Evolutionary Architecture from an Organizational Perspective πΌ
Evolutionary Architecture frequently only considered from the technical point of view, forgetting about the the fact that without the organization support evolving architecture is barely feasible task. The article on InfoQ blog discusses the necessity of IT and Business working as one team, alignment on company-wide goals and the importance of contracts within the organization for achieving truly evolutionary architecture.
#architecture
An educational side project π·ββοΈ
Have you tried building a product like Uber or Bolt on your own? Juraj Majerik decided to try! Across the span of 7 months he configured the infrastructure, built the backend, frontend and the load emulator and created the demo for the system, learning a lot meanwhile. Gergely Orosz describes his work and the main takeaways from such experience alongside with the system architecture.
#casestudy
Frugal Software Architecture πΌ
This week I wrote an article myself. I want to introduce you to the concept of Frugal Software Architecture. But it's not a mere cost efficiency metric of the software, but rather a set of principles allowing to balance strategic investments and cost optimizations. Find them inside.
#architecture #frugality
InfoQ
Evolutionary Architecture from an Organizational Perspective
Creating an architecture that can evolve over time is not simply a technical challenge, and requires collaboration with non-technical people in an organization to ensure the software adapts as needed.
π2π₯1π€©1
Follow-Up
Investigating the impact of HTTP3 on network latency for search π€
A peculiar case study on improving the search results latency. Some can say half a second for a search result is good enough, but not Dropbox. They wanted to decrease that value significantly and started exploring the UDP-based HTTP3 protocol. Find what they achieved inside!
#network #casestudy
Effective Test Automation Approaches for Modern CI/CD pipelines π·ββοΈ
Unit tests are widely accepted, but do we do them right? Do we have the good test design? Do we cover all the necessary cases? Maybe if we involve quality assurance engineers into the designing the unit tests we will have better tests with more coverage? Craig Risi considers this question in his article on effective test automation at InfoQ.
#cicd #deployment #quality #tests
"Software design is just theory" πΌ
The question architects frequently hear is "how are you legit if you don't observe the results of your design?" and fair enough. Mathias Verraes claims that 5 years is approximately the time when the project starts suffering all the decisions made. And at this point just the practical experience is far from enough. Read what's important in the old blog post of Mathias.
#architecture #philosophy #legacy
Don't outsource thinking πΌ
Domain-Driven Design teaches us about general subdomains which frequently make sense to buy a solution for, basically outsourcing it. Gregor Hohpe shares his thoughts on the "buy" strategy: of course, you can outsource a solution, but you would never want to outsource your strategy or in simpler words: thinking.
#philosophy
Embracing Simplicity in Software Architecture Diagramming π·ββοΈ
In this article the author reasons that whatever you're documenting, be it business, solution or deployment architecture, you need to have a top-down approach starting with the highest level of abstract(old system -> new system), slowly increasing the level of details. Find some fancy diagrams inside.
#documentation
5 Ways I Bypassed Your Web Application Firewall π·ββοΈ
Not me though. Ally Petitt wrote a neat article on how you can detect the particular type of the Web Application Firewall used at a site and then try to bypass the firewall using different methods of modifying the payload. The article is not surely intended to bypass WAFs, rather show you how it is possible for you to prepare for those attacks.
#security
Investigating the impact of HTTP3 on network latency for search π€
A peculiar case study on improving the search results latency. Some can say half a second for a search result is good enough, but not Dropbox. They wanted to decrease that value significantly and started exploring the UDP-based HTTP3 protocol. Find what they achieved inside!
#network #casestudy
Effective Test Automation Approaches for Modern CI/CD pipelines π·ββοΈ
Unit tests are widely accepted, but do we do them right? Do we have the good test design? Do we cover all the necessary cases? Maybe if we involve quality assurance engineers into the designing the unit tests we will have better tests with more coverage? Craig Risi considers this question in his article on effective test automation at InfoQ.
#cicd #deployment #quality #tests
"Software design is just theory" πΌ
The question architects frequently hear is "how are you legit if you don't observe the results of your design?" and fair enough. Mathias Verraes claims that 5 years is approximately the time when the project starts suffering all the decisions made. And at this point just the practical experience is far from enough. Read what's important in the old blog post of Mathias.
#architecture #philosophy #legacy
Don't outsource thinking πΌ
Domain-Driven Design teaches us about general subdomains which frequently make sense to buy a solution for, basically outsourcing it. Gregor Hohpe shares his thoughts on the "buy" strategy: of course, you can outsource a solution, but you would never want to outsource your strategy or in simpler words: thinking.
#philosophy
Embracing Simplicity in Software Architecture Diagramming π·ββοΈ
In this article the author reasons that whatever you're documenting, be it business, solution or deployment architecture, you need to have a top-down approach starting with the highest level of abstract(old system -> new system), slowly increasing the level of details. Find some fancy diagrams inside.
#documentation
5 Ways I Bypassed Your Web Application Firewall π·ββοΈ
Not me though. Ally Petitt wrote a neat article on how you can detect the particular type of the Web Application Firewall used at a site and then try to bypass the firewall using different methods of modifying the payload. The article is not surely intended to bypass WAFs, rather show you how it is possible for you to prepare for those attacks.
#security
dropbox.tech
Investigating the impact of HTTP3 on network latency for search
π3β€1π₯1π€©1
I am holding a public Mock System Design interview with Igor Bondarenko. The task is secret, but it's gonna be fun. Make sure to join the stream and ask tricky questions! At the end I will give my recommendations and decide at what level the interview was passed.
Friday, June 9th, 18.00 GMT+3
https://youtube.com/live/zUi_5eSv0wY?feature=share
Friday, June 9th, 18.00 GMT+3
https://youtube.com/live/zUi_5eSv0wY?feature=share
YouTube
Mock System Design Interview
We are going to design some software system live. The candidate - Igor - will collect the requirements, suggest the solution and adapt it for bigger scale.
#systemdesign #softwaredevelopment #architecture #softwarearchitecture
#systemdesign #softwaredevelopment #architecture #softwarearchitecture
π1
On Monday we are talking about Intro to Kubernetes with Carlos Sanchez from Adobe!
Monday, June 12th, 19.00 GMT+3
https://youtube.com/live/v5c7RFTVuHY?feature=share
Monday, June 12th, 19.00 GMT+3
https://youtube.com/live/v5c7RFTVuHY?feature=share
YouTube
Intro to Kubernetes with Carlos Sanchez
With Carlos Sanchez - Senior Cloud Developer at Adobe - we are going to talk about Kubernetes - why it is required, what parts are there, how to learn K8s the best way and of course how it became so popular.
Project Docs: https://kubernetes.io/docs/home/β¦
Project Docs: https://kubernetes.io/docs/home/β¦
β€6
Live in 20 minutes! Come to the stream to ask questions about the solution!
https://youtube.com/live/zUi_5eSv0wY?feature=share
https://youtube.com/live/zUi_5eSv0wY?feature=share
YouTube
Mock System Design Interview
We are going to design some software system live. The candidate - Igor - will collect the requirements, suggest the solution and adapt it for bigger scale.
#systemdesign #softwaredevelopment #architecture #softwarearchitecture
#systemdesign #softwaredevelopment #architecture #softwarearchitecture
π1
Architecture Weekly #71 - Highlights
The State of Data Engineering 2023 πΌ
Data Engineering is a huge domain. That's why state reports are so valuable! Please find one here: Ingestion tools, Data Lakes, Metadata management, Analytics management and many more!
#dataengineering
Intro to Data Engineering πΌ
Speaking about Data Engineering: if the words above do not tell you much, please follow the conversation I had with Pasha Finkelshteyn, a Developer Advocate at JetBrains. He explains what data engineering is all about, what are the biggest problems there and much more!
#dataengineering #video
Building Efficient Experimentation Environments for ML Projects π·ββοΈ
Looks like the 71st Highlights are all about Data Engineering π SwirlAI newsletter gives us a post on organizing the Experimentation Environment for Data Scientists to improve the ML models. The article looks into the properties that an efficient Experimentation Environments should have. And, as a MLOps engineer, you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for.
#ml #dataengineering
The State of Data Engineering 2023 πΌ
Data Engineering is a huge domain. That's why state reports are so valuable! Please find one here: Ingestion tools, Data Lakes, Metadata management, Analytics management and many more!
#dataengineering
Intro to Data Engineering πΌ
Speaking about Data Engineering: if the words above do not tell you much, please follow the conversation I had with Pasha Finkelshteyn, a Developer Advocate at JetBrains. He explains what data engineering is all about, what are the biggest problems there and much more!
#dataengineering #video
Building Efficient Experimentation Environments for ML Projects π·ββοΈ
Looks like the 71st Highlights are all about Data Engineering π SwirlAI newsletter gives us a post on organizing the Experimentation Environment for Data Scientists to improve the ML models. The article looks into the properties that an efficient Experimentation Environments should have. And, as a MLOps engineer, you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for.
#ml #dataengineering
lakeFS
The State of Data Engineering 2023 - lakeFS
Explore the leading tools and trends that shaped data engineering in 2023. Read the detailed report on data version control at scale.
π2π€©1
Architecture Weekly #71 - Follow-Up
Contract Testing Case Study π·ββοΈ
We covered Contract Testing several times. This time I would like to share the article by Ebay Engineering. The folks wanted to ensure the API provided by the Notification team was functioning well for all of it's consumers. They considered the BDD, but found some caveats and pivoted to the Contract Testing. Find out what they managed to achieve!
#api #contracttesting
Six Ways to shoot yourself in the foot with health checks π·ββοΈ
Health checks seem to be pretty simple: check if the application responds in time and connects to a database and a Kafka topic... But it's just first impression. You can easily break your application with small health check tweaks. Find out 6 ways to do that!
#devops #observability
Microsoft Azure Well-Architected Framework πΌ
We covered the Amazon Well-Architectured framework previously. But obviously, they are not the only cloud provider there. It's time for the Azure Well-Architected Framework, which focuses on Reliability, Security, Cost Optimization, Operational Excellence and Performance Efficiency.
#cloud #architecture #bestpractice
The Full Circle on Developer Productivity πΌ
Steve Yegge used to work at Amazon, Google, Grab and now working at Sourcegraph. He shares his long journey and discusses why the developer tooling is so important and why it's so hard to make it right. I couldn't stop reading!
#philosophy
Securing the API access π·ββοΈ
When you expose the API you need to protect it from the malicious access. The first idea is to use some type of a token, but it comes with the problems of invalidation, limiting the scope and many more. OAuth tokens combined with JWT yields more efficient solution without comprising the security and even solving the mentioned problems. Follow the Zapier's blog for further info!
#api #security
Mock System Design Interview: Video Portal π·ββοΈ
Another bit of content from my YouTube Channel: mock system design interview. Here we are architecting a simple video portal, which is capable of uploading the videos, converting it to different qualities and stream it to mobile and web clients. Follow the video for the designing process!
#video #mockinterview #systemdesign
Contract Testing Case Study π·ββοΈ
We covered Contract Testing several times. This time I would like to share the article by Ebay Engineering. The folks wanted to ensure the API provided by the Notification team was functioning well for all of it's consumers. They considered the BDD, but found some caveats and pivoted to the Contract Testing. Find out what they managed to achieve!
#api #contracttesting
Six Ways to shoot yourself in the foot with health checks π·ββοΈ
Health checks seem to be pretty simple: check if the application responds in time and connects to a database and a Kafka topic... But it's just first impression. You can easily break your application with small health check tweaks. Find out 6 ways to do that!
#devops #observability
Microsoft Azure Well-Architected Framework πΌ
We covered the Amazon Well-Architectured framework previously. But obviously, they are not the only cloud provider there. It's time for the Azure Well-Architected Framework, which focuses on Reliability, Security, Cost Optimization, Operational Excellence and Performance Efficiency.
#cloud #architecture #bestpractice
The Full Circle on Developer Productivity πΌ
Steve Yegge used to work at Amazon, Google, Grab and now working at Sourcegraph. He shares his long journey and discusses why the developer tooling is so important and why it's so hard to make it right. I couldn't stop reading!
#philosophy
Securing the API access π·ββοΈ
When you expose the API you need to protect it from the malicious access. The first idea is to use some type of a token, but it comes with the problems of invalidation, limiting the scope and many more. OAuth tokens combined with JWT yields more efficient solution without comprising the security and even solving the mentioned problems. Follow the Zapier's blog for further info!
#api #security
Mock System Design Interview: Video Portal π·ββοΈ
Another bit of content from my YouTube Channel: mock system design interview. Here we are architecting a simple video portal, which is capable of uploading the videos, converting it to different qualities and stream it to mobile and web clients. Follow the video for the designing process!
#video #mockinterview #systemdesign
eBay Inc.
API Evolution Is a Challenge. Could Contract Testing Be the Solution?
Contract testing has grown in popularity in recent years with the widespread adoption of microservice architectures. In this article, we will share our experiences with contract testing at eBay.
π4β€1
Frugal Software Architecture. Part 2: Strategic Investments
https://vvsevolodovich.dev/frugal-software-architecture-part-2-strategic-investments/
https://vvsevolodovich.dev/frugal-software-architecture-part-2-strategic-investments/
π1π€©1
Architecture Weekly #72 - Highlights
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia and Daria for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Relational Databases Explained π€
Recently I interviewed a pretty Senior guy who was doing very well but lacked the understanding of indexes and how to scale relational databases. Fortunately, Architecture Notes came up with a new long post about RDBMS, including the indexes and B-Trees explained. BTW, I have a video on B-Trees as well π
#db
Creating a Multi-Region Application with AWS Services π·ββοΈ
This week the US-EAST-1 region of AWS was affected by an outage caused by AWS Lambda capacity management. There is no postmortem yet, but it is worth returning to the roots of multi-region application architecture. So grab an article from AWS themselves!
#cloud #aws #availability
Frugal Software Architecture. Part 2: Strategic Investments πΌ
The very first principle of frugal software architecture is strategic investments. While the name might imply a purely financial perspective, it extends further into time, technology, and talent. Grab the second article from the series!
#frugality #architecture
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia and Daria for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Relational Databases Explained π€
Recently I interviewed a pretty Senior guy who was doing very well but lacked the understanding of indexes and how to scale relational databases. Fortunately, Architecture Notes came up with a new long post about RDBMS, including the indexes and B-Trees explained. BTW, I have a video on B-Trees as well π
#db
Creating a Multi-Region Application with AWS Services π·ββοΈ
This week the US-EAST-1 region of AWS was affected by an outage caused by AWS Lambda capacity management. There is no postmortem yet, but it is worth returning to the roots of multi-region application architecture. So grab an article from AWS themselves!
#cloud #aws #availability
Frugal Software Architecture. Part 2: Strategic Investments πΌ
The very first principle of frugal software architecture is strategic investments. While the name might imply a purely financial perspective, it extends further into time, technology, and talent. Grab the second article from the series!
#frugality #architecture
boosty.to
Vladimir Ivanov - Author of Architecture Weekly newsletter
Exclusive content from Vladimir Ivanov, subscribe and be the first to access!
π3
Architecture Weekly #72 - Follow-Up
Good availability measure π€
Following up on the availability topic. How do you measure the availability? Surprisingly, every cloud provider uses its own approach. In this paper, you will find what makes a good availability metric, why the current ones are not fulfilling those requirements and find a proposal for a true metric.
#availability #paper
Architecture Principles πΌ
Architecture Principles can guide the major decisions in your system like picking up a technology for Identity Management or picking up a communication protocol. Still the question is what makes a good architecture principle? How to formulate one? Find those rules covered in the article alongside with good examples.
#architecture
Intro to Kubernetes with Carlos Sanchez πΌ
Another interview! I spoke with the Principal Scientist at Adobe Experience Manager and the author of Jenkins Kubernetes Plugin - Carlos Sanchez! We figured out what is Kubernetes, what are the basic building blocks of it and what benefit it brings to managing your payload.
#kubernetes #k8s #interview #video
Database Partition Conversion with minimum downtime π€
Partitioning the database is a known practice of handling the growing amount of data. However it is important to also know how to create a new partition while under the heavy load: cause what other load led you to a db requiring a partition? Find the great piece of migrating to a new partition for a PostgreSQL.
#db #casestudy
PayPals' Key-Value Store open-sourced π·ββοΈ
Some of the news here! PayPal open-sourced JunoDB, a distributed key-value store that uses RocksDB as the underlying storage engine. PayPal is serving 350 billion requests which requires a highly available and security-focused database. Find out how it can be achieved!
#db
Introduction to encryption π€
And the last one for today - the introduction to encryption. I decided that it is always good to get back to the fundamentals - and I stumbled upon a good article. So grab the read!
Good availability measure π€
Following up on the availability topic. How do you measure the availability? Surprisingly, every cloud provider uses its own approach. In this paper, you will find what makes a good availability metric, why the current ones are not fulfilling those requirements and find a proposal for a true metric.
#availability #paper
Architecture Principles πΌ
Architecture Principles can guide the major decisions in your system like picking up a technology for Identity Management or picking up a communication protocol. Still the question is what makes a good architecture principle? How to formulate one? Find those rules covered in the article alongside with good examples.
#architecture
Intro to Kubernetes with Carlos Sanchez πΌ
Another interview! I spoke with the Principal Scientist at Adobe Experience Manager and the author of Jenkins Kubernetes Plugin - Carlos Sanchez! We figured out what is Kubernetes, what are the basic building blocks of it and what benefit it brings to managing your payload.
#kubernetes #k8s #interview #video
Database Partition Conversion with minimum downtime π€
Partitioning the database is a known practice of handling the growing amount of data. However it is important to also know how to create a new partition while under the heavy load: cause what other load led you to a db requiring a partition? Find the great piece of migrating to a new partition for a PostgreSQL.
#db #casestudy
PayPals' Key-Value Store open-sourced π·ββοΈ
Some of the news here! PayPal open-sourced JunoDB, a distributed key-value store that uses RocksDB as the underlying storage engine. PayPal is serving 350 billion requests which requires a highly available and security-focused database. Find out how it can be achieved!
#db
Introduction to encryption π€
And the last one for today - the introduction to encryption. I decided that it is always good to get back to the fundamentals - and I stumbled upon a good article. So grab the read!
β€1π₯1