Architecture Weeekly #61
Video
Highlights
Google Sheets as backend 👷♂️
When you start a new project you probably think, hey, I would pick MongoDB, or I want a PostgreSQL installation. But some successful projects start with none other than Google Sheets as their storage. It appears it is scalable, simple, and it's easy to integrate with. Find a great post by Levels.fyi - a site with literally salary bands of IT industry - where they show the evolution of their system design, which started with Google Forms and Google Sheets.
#casestudy
Reality Check Pattern 🍼
Have you ever encountered your navigation software telling you you're in a river while you were driving a highway? Well, Uwe Friedrechson just did, and a resiliency architect, he immediately came up with the pattern which he called "A reality check". It means that your system should check the signals from the outside world, and, if they stop making sense, change the state to something like safe mode, where it could not mislead the user or harm them. Grab more examples and conclusions inside 🙂
#architecture #pattern #resilience
Distributed Machine Learning at Instacart 👷♂️
Several issues ago I mentioned the KubeRay - a platform for running ML workloads on Kubernetes. And this week Instacart published a piece on building the distributed ML platform using exactly that. Find out, how they managed to build a scalable, efficient, diverse platform which allows for development and maintenance simplicity, as well as seamless integration with the previous platform called Griffin.
#ml #casestudy #k8s #kubernetes #kuberay
Video
Highlights
Google Sheets as backend 👷♂️
When you start a new project you probably think, hey, I would pick MongoDB, or I want a PostgreSQL installation. But some successful projects start with none other than Google Sheets as their storage. It appears it is scalable, simple, and it's easy to integrate with. Find a great post by Levels.fyi - a site with literally salary bands of IT industry - where they show the evolution of their system design, which started with Google Forms and Google Sheets.
#casestudy
Reality Check Pattern 🍼
Have you ever encountered your navigation software telling you you're in a river while you were driving a highway? Well, Uwe Friedrechson just did, and a resiliency architect, he immediately came up with the pattern which he called "A reality check". It means that your system should check the signals from the outside world, and, if they stop making sense, change the state to something like safe mode, where it could not mislead the user or harm them. Grab more examples and conclusions inside 🙂
#architecture #pattern #resilience
Distributed Machine Learning at Instacart 👷♂️
Several issues ago I mentioned the KubeRay - a platform for running ML workloads on Kubernetes. And this week Instacart published a piece on building the distributed ML platform using exactly that. Find out, how they managed to build a scalable, efficient, diverse platform which allows for development and maintenance simplicity, as well as seamless integration with the previous platform called Griffin.
#ml #casestudy #k8s #kubernetes #kuberay
YouTube
Google Sheets as Backend, Reality Check Pattern and Distributed ML Platform at Instacart
Architecture Weekly #61
Text Version: https://vvsevolodovich.dev/architecture-weekly-61/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Start
0:15 Link to Interview…
Text Version: https://vvsevolodovich.dev/architecture-weekly-61/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Start
0:15 Link to Interview…
Architecture Weekly #61 Follow Up
AWS Aurora receives Write Through Cache 👷♂️
Replication lag appears when the replication stream is generated too fast for a replica to keep up. To improve this situation, Write-Through Cache was introduced which decreases the replication lag by 44%. More details in the article.
#database #cloud #aws
Architects, Anti-Patterns and Organizational Fuckery 🍼
What is the difference between Principal Engineer and Software Architect? Should Architect be a Role or a Title? How to avoid screwing up your organization by introducing the Architects rotting up in their Ivory Tower? The person named mipsytipsi wrote a long read after a discussion in Twitter where the attitude toward the title was collected. Funny, that the post confuses the title and the role... but it's still an entertaining read nonetheless.
#architect #philosofy
Cloud Native Landscape 👷♂️
The amount of tools for developing, deploying, running and monitoring cloud-native apps is over 1000. So you face a problem: what tools you would like to solve the expected issues? Worry no more: Cloud-Native Foundation has a tool called landscape, where you can find all the tools conveniently laid out by the area of applicability.
#cloud #cloudnative
Workflow Orchestration 👷♂️
The common approach to a distributed transaction is a saga either orchestrated or via choreography. However, sagas compensation can not be always possible or at least expensive, like refunding the money sent for payment due to commissions. One of the ideas to solve the problem is the workflow approach, which camunda or temporal.io is implementing. This way you have a separate component which responsibility is only to manage the workflows and ensure they finish or complete gracefully. Follow the article for details.
#patterns #distributedsystems
Contract-Driven Development 👷♂️
The "Contract-Driven Adoption" article on InfoQ explores how contract-driven development can improve the adoption of new technologies and tools in organizations. It discusses the benefits of using contracts to define expected behaviours and outcomes, reduce ambiguity, and enable faster feedback and iteration. The article covers different types of contracts, provides examples of real-world projects, and offers practical tips for using contract-driven development. Overall, it provides a useful perspective on how contract-driven development can facilitate technology adoption and improve collaboration among teams.
#microservices #contract #api
Microservice Dependency Graphs 🤟
The "Characterizing Microservice Dependency Graphs" article on MuratBuffalo's blog discusses a study on analyzing over 100 microservice systems using network science techniques to identify common patterns and properties. The article highlights key findings, such as the prevalence of power-law distributions, clustering, and hierarchical structures in microservice dependency graphs. The authors discuss the implications of these findings on the design and management of microservice architectures, such as the need for monitoring, fault tolerance, and modularization. Overall, the article offers insights for improving the scalability, reliability, and maintainability of microservice architectures.
#microservice #paper #tracing
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
AWS Aurora receives Write Through Cache 👷♂️
Replication lag appears when the replication stream is generated too fast for a replica to keep up. To improve this situation, Write-Through Cache was introduced which decreases the replication lag by 44%. More details in the article.
#database #cloud #aws
Architects, Anti-Patterns and Organizational Fuckery 🍼
What is the difference between Principal Engineer and Software Architect? Should Architect be a Role or a Title? How to avoid screwing up your organization by introducing the Architects rotting up in their Ivory Tower? The person named mipsytipsi wrote a long read after a discussion in Twitter where the attitude toward the title was collected. Funny, that the post confuses the title and the role... but it's still an entertaining read nonetheless.
#architect #philosofy
Cloud Native Landscape 👷♂️
The amount of tools for developing, deploying, running and monitoring cloud-native apps is over 1000. So you face a problem: what tools you would like to solve the expected issues? Worry no more: Cloud-Native Foundation has a tool called landscape, where you can find all the tools conveniently laid out by the area of applicability.
#cloud #cloudnative
Workflow Orchestration 👷♂️
The common approach to a distributed transaction is a saga either orchestrated or via choreography. However, sagas compensation can not be always possible or at least expensive, like refunding the money sent for payment due to commissions. One of the ideas to solve the problem is the workflow approach, which camunda or temporal.io is implementing. This way you have a separate component which responsibility is only to manage the workflows and ensure they finish or complete gracefully. Follow the article for details.
#patterns #distributedsystems
Contract-Driven Development 👷♂️
The "Contract-Driven Adoption" article on InfoQ explores how contract-driven development can improve the adoption of new technologies and tools in organizations. It discusses the benefits of using contracts to define expected behaviours and outcomes, reduce ambiguity, and enable faster feedback and iteration. The article covers different types of contracts, provides examples of real-world projects, and offers practical tips for using contract-driven development. Overall, it provides a useful perspective on how contract-driven development can facilitate technology adoption and improve collaboration among teams.
#microservices #contract #api
Microservice Dependency Graphs 🤟
The "Characterizing Microservice Dependency Graphs" article on MuratBuffalo's blog discusses a study on analyzing over 100 microservice systems using network science techniques to identify common patterns and properties. The article highlights key findings, such as the prevalence of power-law distributions, clustering, and hierarchical structures in microservice dependency graphs. The authors discuss the implications of these findings on the design and management of microservice architectures, such as the need for monitoring, fault tolerance, and modularization. Overall, the article offers insights for improving the scalability, reliability, and maintainability of microservice architectures.
#microservice #paper #tracing
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
InfoQ
Amazon Aurora PostgreSQL Adds Write-Through Cache to Improve Logical Replication Performance
Amazon Aurora PostgreSQL-Compatible databases recently introduced a logical replication write-through cache to reduce the amount of storage I/O during logical decoding. The new feature decreases the replication lag, improving read I/O and transaction catch…
🔥3
Why do people fail system design interviews?
https://youtube.com/shorts/KgxTv48mzK0
https://youtube.com/shorts/KgxTv48mzK0
YouTube
Biggest mistake during system design interview #shorts
Why people fail the system design interview?#systemdesign #interview
❤7👍2🥴1
The biggest lie of IT industry
https://youtube.com/shorts/5lAIt_L6Hps
https://youtube.com/shorts/5lAIt_L6Hps
👍2
👎4😁3
Architecture Weekly #62 Highlights
This week I decided to give YouTube shorts a try and made a couple of them. Tell me how did you like it? What are the good ideas for shorts?
Capturing a billion Emojis 👷♂️
Sports events are crowded - and on the internet even to a bigger degree. The companies who handle the broadcasting are willing to make them interactive so that users can show their reactions to particular events of the show - with emojis of course. To make it happen, there should be some architecture in place. Find a good piece of engineering to solve this peculiar problem.
#highload #casestudy
Uptime Guarantees - A Pragmatic Perspective 👷♂️
How frequently do you hear from the business they need 100% availability? More often than me getting incorrectly called Vlad(that's another name!). How do you turn down such requests? Get a pragmatic perspective: how to reason about the actual availability, how much does it cost to come even to 4 nines(99,99%) and what do the cloud guarantees mean?
#cloud #cost
DuckDB Hype 👷♂️
A few issues ago I featured a post by Jordan Tigani with the statement that Big Data is Dead. But what is up to the replacement then? EasyData! DuckDB is trying to do the same for data analytics as SQLite doing for transactional processing. Easy, small, embeddable, lightning fast - all of this is about DuckDB. Check out the post by Oliver Molander where he analyzes all the hype around it.
#db #data #duckdb
This week I decided to give YouTube shorts a try and made a couple of them. Tell me how did you like it? What are the good ideas for shorts?
Capturing a billion Emojis 👷♂️
Sports events are crowded - and on the internet even to a bigger degree. The companies who handle the broadcasting are willing to make them interactive so that users can show their reactions to particular events of the show - with emojis of course. To make it happen, there should be some architecture in place. Find a good piece of engineering to solve this peculiar problem.
#highload #casestudy
Uptime Guarantees - A Pragmatic Perspective 👷♂️
How frequently do you hear from the business they need 100% availability? More often than me getting incorrectly called Vlad(that's another name!). How do you turn down such requests? Get a pragmatic perspective: how to reason about the actual availability, how much does it cost to come even to 4 nines(99,99%) and what do the cloud guarantees mean?
#cloud #cost
DuckDB Hype 👷♂️
A few issues ago I featured a post by Jordan Tigani with the statement that Big Data is Dead. But what is up to the replacement then? EasyData! DuckDB is trying to do the same for data analytics as SQLite doing for transactional processing. Easy, small, embeddable, lightning fast - all of this is about DuckDB. Check out the post by Oliver Molander where he analyzes all the hype around it.
#db #data #duckdb
YouTube
Biggest mistake during system design interview #shorts
Why people fail the system design interview?#systemdesign #interview
❤2
Architecture Weekly #62 Follow Up
Vespa vs Elasticsearch vs Solt - Search Engine Comparison 🍼
Picking up a full-text search engine can be tricky, considering the different use cases and the range of capabilities offered by the engines out there. Should you pick Vespa? Maybe Solr will fit your task better? Grab a short article with a comparison table between Vespa, Elasticsearch and Solt.
#searchengine #solr #elasticsearch
Predicting demand at Airports 🤟
The taxi operators at airports face a problem: if there is not enough demand, drivers waste their time. If there are not enough taxis though, the experience for riders is unpleasant. Uber is trying to solve this problem with Machine Learning and predictive models. In this article you will find the formulas and the reference architecture for it, but I still wonder why checking out the arrivals table is not sufficient.
#ml
Operating Lambda: Performance Optimization 👷♂️
When you start using Lambdas, the first problem you observe is a long startup time. A typical solution is to schedule an event which will ping your function every once in a while to ensure a warm environment. However, this solution is not universal: it does not help during increased load or starting a Lambda in a different AZ. Grab the read about available performance optimizations for Lambda and the anatomy of Lambda's start.
#cloud #aws #lambda #serverless
The complete guide to System Design in 2023
And another guide to System Design Interviews! This one will get you through all the foundational terms you need to be aware of to pass an interview. It covers storage types, scaling, redundancy, patterns, databases and many more. Follow it on!
#interview #fundamentals
A Simple Framework for Architectural Decisions
It sounds a bit too pretentious, but this article covers some clues you would like to adopt to simplify the technical decision-making in your organization. Those include building a Tech Radar, formulating the best practices like REST API or Observability standards, and documenting the decisions in the form of RFCs and ADRs. Grab more details inside!
#architecture #documentation
Migrating 125.000 users from Auth0 to Supabase
Kevin Peters decided to share the experience of Parqet company of migrating user authentication and authorization from Auth0 to Supabase. The author highlights the reasons for the migration, explains the process of setting up Supabase, and provides an in-depth analysis of the migration process. The article also compares the pricing of Auth0 and Supabase. It serves as a helpful guide for anyone looking to migrate user authentication and authorization from Auth0 to Supabase.
#casestudy #migration #authentication
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course, every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
Vespa vs Elasticsearch vs Solt - Search Engine Comparison 🍼
Picking up a full-text search engine can be tricky, considering the different use cases and the range of capabilities offered by the engines out there. Should you pick Vespa? Maybe Solr will fit your task better? Grab a short article with a comparison table between Vespa, Elasticsearch and Solt.
#searchengine #solr #elasticsearch
Predicting demand at Airports 🤟
The taxi operators at airports face a problem: if there is not enough demand, drivers waste their time. If there are not enough taxis though, the experience for riders is unpleasant. Uber is trying to solve this problem with Machine Learning and predictive models. In this article you will find the formulas and the reference architecture for it, but I still wonder why checking out the arrivals table is not sufficient.
#ml
Operating Lambda: Performance Optimization 👷♂️
When you start using Lambdas, the first problem you observe is a long startup time. A typical solution is to schedule an event which will ping your function every once in a while to ensure a warm environment. However, this solution is not universal: it does not help during increased load or starting a Lambda in a different AZ. Grab the read about available performance optimizations for Lambda and the anatomy of Lambda's start.
#cloud #aws #lambda #serverless
The complete guide to System Design in 2023
And another guide to System Design Interviews! This one will get you through all the foundational terms you need to be aware of to pass an interview. It covers storage types, scaling, redundancy, patterns, databases and many more. Follow it on!
#interview #fundamentals
A Simple Framework for Architectural Decisions
It sounds a bit too pretentious, but this article covers some clues you would like to adopt to simplify the technical decision-making in your organization. Those include building a Tech Radar, formulating the best practices like REST API or Observability standards, and documenting the decisions in the form of RFCs and ADRs. Grab more details inside!
#architecture #documentation
Migrating 125.000 users from Auth0 to Supabase
Kevin Peters decided to share the experience of Parqet company of migrating user authentication and authorization from Auth0 to Supabase. The author highlights the reasons for the migration, explains the process of setting up Supabase, and provides an in-depth analysis of the migration process. The article also compares the pricing of Auth0 and Supabase. It serves as a helpful guide for anyone looking to migrate user authentication and authorization from Auth0 to Supabase.
#casestudy #migration #authentication
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course, every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
👍4❤2
Observability with Charity Majors(CTO at honeycomb.io). This Wednesday, 19.00
https://youtube.com/live/2rSqFCkHdg4
https://youtube.com/live/2rSqFCkHdg4
YouTube
Traces is not Observability / Charity Majors(CTO @ Honeycomb)
What is the observability in Software world? How do you think about it? What is the state of the art observability nowadays? We are speaking with Charity Majors - CTO of honeycomb.io - about all of that.
https://vvsevolodovich.dev
Patreon: https://patr…
https://vvsevolodovich.dev
Patreon: https://patr…
❤5
Architecture Weekly #63 Highlights
This week I conducted an interview with Charity Majors(CTO at honeycomb.io). She told me so much about state-of-the-art modern observability, and it will be useful for you as well! Watch thoroughly!
Growth of AI Through a Cloud Lens 🍼
Do you remember the start of cloud computing? There was a lot of scepcis, like "real business won't use it", shifting to "Fortune 500 companies won't use it", until everybody was using cloud. Mitchell Hashimoto, a CTO of Hashicorp, recalls this trends and tries to analyze if it applies to the raise of LLMs. How much impractical for business the LLMs are? And how this area of impracticality will shrink in the future? Get a long read!
#ai #cloud
Twitter's Recommendation Algorithm 👷♂️
Have you ever ranted about how your tweeter feed contains some weird replies to the people you're not even interested in? Well, last week Twitter shared the source code for their recommendation service and wrote a blog post about it's architecture. Grab the details inside to both get some knowledge and improve your twitter visibility!
#casestudy
Spyware vendors use 0-days against mobile OSs and Chrome 👷♂️
Security news here. In Google, there is a Threat Analysis Group(TAG), which tracks malware vendors. Recently they discovered couple of highly impactful campaigns which use 0-days in Android, iOS and Google Chrome. In the article you will find the exploit chains description and what TAG has done about them
#security
This week I conducted an interview with Charity Majors(CTO at honeycomb.io). She told me so much about state-of-the-art modern observability, and it will be useful for you as well! Watch thoroughly!
Growth of AI Through a Cloud Lens 🍼
Do you remember the start of cloud computing? There was a lot of scepcis, like "real business won't use it", shifting to "Fortune 500 companies won't use it", until everybody was using cloud. Mitchell Hashimoto, a CTO of Hashicorp, recalls this trends and tries to analyze if it applies to the raise of LLMs. How much impractical for business the LLMs are? And how this area of impracticality will shrink in the future? Get a long read!
#ai #cloud
Twitter's Recommendation Algorithm 👷♂️
Have you ever ranted about how your tweeter feed contains some weird replies to the people you're not even interested in? Well, last week Twitter shared the source code for their recommendation service and wrote a blog post about it's architecture. Grab the details inside to both get some knowledge and improve your twitter visibility!
#casestudy
Spyware vendors use 0-days against mobile OSs and Chrome 👷♂️
Security news here. In Google, there is a Threat Analysis Group(TAG), which tracks malware vendors. Recently they discovered couple of highly impactful campaigns which use 0-days in Android, iOS and Google Chrome. In the article you will find the exploit chains description and what TAG has done about them
#security
YouTube
Traces is not Observability / Charity Majors(CTO @ Honeycomb)
What is the observability in Software world? How do you think about it? What is the state of the art observability nowadays? We are speaking with Charity Majors - CTO of honeycomb.io - about all of that.
https://vvsevolodovich.dev
Patreon: https://patr…
https://vvsevolodovich.dev
Patreon: https://patr…
❤1
Follow Up
Media Understanding Platform for ML Innovations 👷♂️
Netflix serves you content, but it also should help you finding it. Previously, the editor had to go through the hours of the shows hosted on the platform to pick up highlights or choose the engaging, story-telling dialogues. So Netflix decided to improve the efficiency of editors' work, introducing a solution to find the scenes in the show by a request, like "red racing car" or "dialogue with tears". Find out, what solution they ended up implementing!
#ml #batchprocessing
Cloud-Native Applications Design Principles 🍼
Cloud-Native applications is not merely containers with a couple of health probes. They are built with the set of principles in mind, like loosely coupled architecture, API First approach and others. Grab an article from JFrog blog to know 6 principles in details!
#cloud #architecture
Service level objectives 101: Establishing effective SLOs 👷♂️
The article titled "Establishing Service Level Objectives" on Datadog's blog provides a comprehensive guide to help organizations define, measure, and manage their Service Level Objectives (SLOs). The article explains how SLOs can be used to set clear and achievable performance targets for software systems and services, and how they can help organizations improve their overall service reliability and customer satisfaction. The article also provides practical tips on how to establish effective SLOs, including selecting appropriate metrics, defining appropriate thresholds and error budgets, and using feedback loops to continuously improve performance. Additionally, the article highlights the importance of aligning SLOs with business objectives and involving stakeholders in the SLO definition process
#observability
Getting started with small-step operational semantics 🤟
We discussed sagas many times. But now I want to share with you an article by Dominik Tornow about the formal language that helps to describe those. Grab on in the Temporal.io blog.
#saga #distributedtransaction
The cost of Architectural Complexity 🤟
It's pretty obvious that the more complex architecture you have, the more it's operation will cost. However this is not the only penalty you might experience. For example, the number and severity of software bugs increases with the complexity as well. What else? Follow the Abi's review of the paper from 2013 on the cost of architectural complexity.
#architecture #paper
Lambda vs Kappa Data Processing Architecture 🍼
The last one for today is the guide on the 2 approaches to the data processing architecture: Lambda vs Kappa. It has nothing to do with Lambda functions though. So the former is more complex and includes 2 data processing streams for realtime and batch processing; the latter is simplified and trying to do the job with only a single pipeline treating everything as a data stream. Find more details inside.
#data #architecture
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
Media Understanding Platform for ML Innovations 👷♂️
Netflix serves you content, but it also should help you finding it. Previously, the editor had to go through the hours of the shows hosted on the platform to pick up highlights or choose the engaging, story-telling dialogues. So Netflix decided to improve the efficiency of editors' work, introducing a solution to find the scenes in the show by a request, like "red racing car" or "dialogue with tears". Find out, what solution they ended up implementing!
#ml #batchprocessing
Cloud-Native Applications Design Principles 🍼
Cloud-Native applications is not merely containers with a couple of health probes. They are built with the set of principles in mind, like loosely coupled architecture, API First approach and others. Grab an article from JFrog blog to know 6 principles in details!
#cloud #architecture
Service level objectives 101: Establishing effective SLOs 👷♂️
The article titled "Establishing Service Level Objectives" on Datadog's blog provides a comprehensive guide to help organizations define, measure, and manage their Service Level Objectives (SLOs). The article explains how SLOs can be used to set clear and achievable performance targets for software systems and services, and how they can help organizations improve their overall service reliability and customer satisfaction. The article also provides practical tips on how to establish effective SLOs, including selecting appropriate metrics, defining appropriate thresholds and error budgets, and using feedback loops to continuously improve performance. Additionally, the article highlights the importance of aligning SLOs with business objectives and involving stakeholders in the SLO definition process
#observability
Getting started with small-step operational semantics 🤟
We discussed sagas many times. But now I want to share with you an article by Dominik Tornow about the formal language that helps to describe those. Grab on in the Temporal.io blog.
#saga #distributedtransaction
The cost of Architectural Complexity 🤟
It's pretty obvious that the more complex architecture you have, the more it's operation will cost. However this is not the only penalty you might experience. For example, the number and severity of software bugs increases with the complexity as well. What else? Follow the Abi's review of the paper from 2013 on the cost of architectural complexity.
#architecture #paper
Lambda vs Kappa Data Processing Architecture 🍼
The last one for today is the guide on the 2 approaches to the data processing architecture: Lambda vs Kappa. It has nothing to do with Lambda functions though. So the former is more complex and includes 2 data processing streams for realtime and batch processing; the latter is simplified and trying to do the job with only a single pipeline treating everything as a data stream. Find more details inside.
#data #architecture
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
Medium
Building a Media Understanding Platform for ML Innovations
The media understanding platform serves as an abstraction layer between Machine Learning (ML) algos and various applications.
Already tomorrow, at 5PM CET, I am speaking with Baruch Sadogursky on Developer Advocacy, work at Gradle and all that stuff! Book your calendars and of course, subscribe. Link to the stream below 👇
https://youtube.com/live/KXeTwgxs5wU
https://youtube.com/live/KXeTwgxs5wU
YouTube
Developer Advocacy with Baruch Sadogursky
What is the Developer Advocacy? Why it's important for modern companies? What architects and senior engineers should know about it? We are talking about all ...
🔥4
Architecture Weekly #64 Highlights
This week I conducted an interview with Baruch Sadogursky (Principal Developer Advocate at Gradle). We discussed Developer Advocacy, Development Relations and why people are confused with DevOps and DevRels roles. Watch it here!
The end of a myth: Distributed transactions can scale 🤟
I learned recently about RDMA - remote direct memory access, which allows bypassing the CPU while transferring data from one machine to another. And this development actually enables the performant distributed transactions. A paper that depicts the system design for it is being reviewed by Murat. Grab a hard, but insightful read!
#database #distributed #paper
Let's (not) break up the monolith 👷♂️
This week I published a short made out of the interview with Andrey Rebrov about microservices. The short was about should you really necessarily rewrite the monolith though(Damn, watch it already!). Following up the thought, Uwe accidentally started a series of articles on what people really want when they say they would like to break up the monolith into microservices. It appears, they don't want to rewrite it necessarily, but rather have more frequent deployment and more predictable development. Well, you can do that with monolith as well!
#architecture #monolith #microservice
16 System Design Concepts You wish you knew before 👷♂️
Another short guide to help you prepare to the system design interview. This is not like a super-full list but it will do as an introductory checklist.
#systemdesign #interview
This week I conducted an interview with Baruch Sadogursky (Principal Developer Advocate at Gradle). We discussed Developer Advocacy, Development Relations and why people are confused with DevOps and DevRels roles. Watch it here!
The end of a myth: Distributed transactions can scale 🤟
I learned recently about RDMA - remote direct memory access, which allows bypassing the CPU while transferring data from one machine to another. And this development actually enables the performant distributed transactions. A paper that depicts the system design for it is being reviewed by Murat. Grab a hard, but insightful read!
#database #distributed #paper
Let's (not) break up the monolith 👷♂️
This week I published a short made out of the interview with Andrey Rebrov about microservices. The short was about should you really necessarily rewrite the monolith though(Damn, watch it already!). Following up the thought, Uwe accidentally started a series of articles on what people really want when they say they would like to break up the monolith into microservices. It appears, they don't want to rewrite it necessarily, but rather have more frequent deployment and more predictable development. Well, you can do that with monolith as well!
#architecture #monolith #microservice
16 System Design Concepts You wish you knew before 👷♂️
Another short guide to help you prepare to the system design interview. This is not like a super-full list but it will do as an introductory checklist.
#systemdesign #interview
YouTube
Developer Advocacy with Baruch Sadogursky
What is the Developer Advocacy? Why it's important for modern companies? What architects and senior engineers should know about it? We are talking about all of that with the Baruch Sadogursky, Principal Developer Advocate at Gradle!
0:00 Introduction
1:19…
0:00 Introduction
1:19…
🔥1
Introduction to Data Modeling 🍼
We live in a time when it's essential to fast, so everybody is doing MVPs, and once they go live, the engineers never get a chance to rewrite a product, because you know, business needs revenue. The same happens with data: you never get a chance to model the data properly. But if you do, you need to know why it's crucial for your data engineering pipelines and what's there for it. Read a long read by Airbyte.
#data
Zero-ETL, ChatGPT and the Future of Data Engineering 👷♂️
Everybody heard about zero code, but the end of the joke is "zero code - zero jobs". This time we are hearing about Zero-ETL, which was introduced originally by Amazon. So the idea is that you will have only some code doing the job, and the pipeline will be provided for you. But is it really feasible? What are the nuances of such a stack? Some considerations inside.
#data #etl
How to build next-gen serverless applications 🍼
Serverless does indeed have servers in it, but you might not care, because it's a managed service. However, you still need to think about structuring your application, the communication between the apps and the resilience. Grab a short article on some of the considerations on serverless apps.
How to write RFCs and ADRs 🍼
And back to our favorite topic - RFCs and ADRs. I have shared several articles in the previous issues of the newsletter about why we need them in the first place. This time I found a good article on how precisely write them so that they are clear, concise and meaningful. Take a note!
#documentation #rfc #adr #howto
Architecting disaster recovery for cloud infrastructure outages 👷♂️
Cloud is still running on the physicals servers in the data centers. And there are electricity outages, floods, connectivity losses and other disasters. The Cloud systems should tackle this kind of problems still, that's why they put the disaster recovery plans. Read how they do it in Google.
#cloud #gcp #disasterrecovery
Lab Basics 👷♂️
And a little fun. Imagine you're experimenting with some IoT devices or having some working for your SmartHome. The good idea would be to organize a proper network setup to isolate one group of the devices from the another. Ken Moini dropped a nice beginning of the series, where he explains how to runs his own computer lab at home. Grab a read.
#network #casestudy #iot
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
We live in a time when it's essential to fast, so everybody is doing MVPs, and once they go live, the engineers never get a chance to rewrite a product, because you know, business needs revenue. The same happens with data: you never get a chance to model the data properly. But if you do, you need to know why it's crucial for your data engineering pipelines and what's there for it. Read a long read by Airbyte.
#data
Zero-ETL, ChatGPT and the Future of Data Engineering 👷♂️
Everybody heard about zero code, but the end of the joke is "zero code - zero jobs". This time we are hearing about Zero-ETL, which was introduced originally by Amazon. So the idea is that you will have only some code doing the job, and the pipeline will be provided for you. But is it really feasible? What are the nuances of such a stack? Some considerations inside.
#data #etl
How to build next-gen serverless applications 🍼
Serverless does indeed have servers in it, but you might not care, because it's a managed service. However, you still need to think about structuring your application, the communication between the apps and the resilience. Grab a short article on some of the considerations on serverless apps.
How to write RFCs and ADRs 🍼
And back to our favorite topic - RFCs and ADRs. I have shared several articles in the previous issues of the newsletter about why we need them in the first place. This time I found a good article on how precisely write them so that they are clear, concise and meaningful. Take a note!
#documentation #rfc #adr #howto
Architecting disaster recovery for cloud infrastructure outages 👷♂️
Cloud is still running on the physicals servers in the data centers. And there are electricity outages, floods, connectivity losses and other disasters. The Cloud systems should tackle this kind of problems still, that's why they put the disaster recovery plans. Read how they do it in Google.
#cloud #gcp #disasterrecovery
Lab Basics 👷♂️
And a little fun. Imagine you're experimenting with some IoT devices or having some working for your SmartHome. The good idea would be to organize a proper network setup to isolate one group of the devices from the another. Ken Moini dropped a nice beginning of the series, where he explains how to runs his own computer lab at home. Grab a read.
#network #casestudy #iot
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
Airbyte
Data Modeling – The Unsung Hero of Data Engineering: An Introduction to Data Modeling | Airbyte
Dive into data modeling’s importance in data engineering with our series (Part 1). Discover its history, complexity, and the role of business requirements.
❤2👍2
Architecture Weekly History Digest
This time I am sharing you not the new links, but instead some articles and videos from the most liked issues of the newsletter. Give them a thorough read , if you missed them or just remind yourself about their content ;)
How Discord Stores Billions of Messages 👷♂️
Amazing post from the CTO of Discord on storing the messages. You will find a business problem statement, inherited situation, articulated requirements and a detailed solution in the article. A true example of an architectural blog post. Deserves a first line in the newsletter issue!
SQLite Internals: Pages & B-trees 🤟
SQLite is a small database famous for it's reliability, small size, usage in the mobile applications and the fact that the LOCs of tests for it is greater that the lines of actual DB code. In this article you will find how SQLite encodes the rows, why we need pages and how b-trees allow to find data in log(N) time.
#database
Architecture for Flow with Wardley Mapping, DDD, and Team Topologies 👷♂️
With this talk I learned about Wardley Map - a tool to create the strategic design of an application or a service. Having introduced that, Susanne Kaiser shows how to apply it to online learning domain, add a bit of Domain-Driven Design, a grain of Team Topologies to come up with a well-though solution.
#ddd #devops #strategy #video
Reducing Logging Cost by Two Orders of Magnitude using CLP 🤟
The more your business grows the more you need to understand what happens within your system. With Uber and tons of analytical data and therefore log data generated each day, it became an issue. They need to retain a sufficient amount of logs to understand what happens within their Spark jobs and not pay for it too much. So they used a Compressed Log Processor, which was able to reduce the number of writes to SSD and store the logs in a searchable manner but occupying two orders of magnitude less space. More details inside!
#logging #costoptimization #bigdata
Zero Downtime Migration from HBase to Bigtable 👷♂️
Box - a cloud file storage and management system - used to store more than 80TB of user data in on-premise HBase. Last year they decided to migrate it all to fully managed HBase compatible BigTable. The requirement was to make to with zero-downtime, which could be challenging. Read about the PoC they did to choose the storage type, and how they ensured the zero downtime in the post by Axatha Jalimarada.
#migrations #cloud #bigtable
How to choose a database?
Just reminding you that I posted a video of a heading "Architecture Reading". I will go through the books and cover their content. Checkout and leave feedback!
https://youtu.be/M0u3btcLvZo
#video
Aligning organization and architecture with strategic DDD 🍼
Find a deck by Michael Plöd on the application of DDD to organizing teams. As to Convey's law, your architecture will resemble the organization, picking the proper team helps a lot. Pick up the DDD approach, identify contexts, make good boundaries and remember, nothing is perfect. The slide deck below.
#ddd
The security design of the AWS Nitro System 👷♂️
The software manages a lot of PII(Personal Identification Information), PHI(Personal Health Information) and PCI(Personal Commercial Information). Processing this data requires a high level of protection not only at the application but at the VM and hardware as well. AWS published a big material on how Nitro System works and how they provide a secure virtualization platform.
#security #aws
Design Considerations fpr Platform Engineering Teams 🍼
Following up the topic of Platform Teams, please find an article on the teams that might together represent the platform. Platform User Interface, Infrastructure, Services, Support and others inside.
#platform #bestpractices #teams
This time I am sharing you not the new links, but instead some articles and videos from the most liked issues of the newsletter. Give them a thorough read , if you missed them or just remind yourself about their content ;)
How Discord Stores Billions of Messages 👷♂️
Amazing post from the CTO of Discord on storing the messages. You will find a business problem statement, inherited situation, articulated requirements and a detailed solution in the article. A true example of an architectural blog post. Deserves a first line in the newsletter issue!
SQLite Internals: Pages & B-trees 🤟
SQLite is a small database famous for it's reliability, small size, usage in the mobile applications and the fact that the LOCs of tests for it is greater that the lines of actual DB code. In this article you will find how SQLite encodes the rows, why we need pages and how b-trees allow to find data in log(N) time.
#database
Architecture for Flow with Wardley Mapping, DDD, and Team Topologies 👷♂️
With this talk I learned about Wardley Map - a tool to create the strategic design of an application or a service. Having introduced that, Susanne Kaiser shows how to apply it to online learning domain, add a bit of Domain-Driven Design, a grain of Team Topologies to come up with a well-though solution.
#ddd #devops #strategy #video
Reducing Logging Cost by Two Orders of Magnitude using CLP 🤟
The more your business grows the more you need to understand what happens within your system. With Uber and tons of analytical data and therefore log data generated each day, it became an issue. They need to retain a sufficient amount of logs to understand what happens within their Spark jobs and not pay for it too much. So they used a Compressed Log Processor, which was able to reduce the number of writes to SSD and store the logs in a searchable manner but occupying two orders of magnitude less space. More details inside!
#logging #costoptimization #bigdata
Zero Downtime Migration from HBase to Bigtable 👷♂️
Box - a cloud file storage and management system - used to store more than 80TB of user data in on-premise HBase. Last year they decided to migrate it all to fully managed HBase compatible BigTable. The requirement was to make to with zero-downtime, which could be challenging. Read about the PoC they did to choose the storage type, and how they ensured the zero downtime in the post by Axatha Jalimarada.
#migrations #cloud #bigtable
How to choose a database?
Just reminding you that I posted a video of a heading "Architecture Reading". I will go through the books and cover their content. Checkout and leave feedback!
https://youtu.be/M0u3btcLvZo
#video
Aligning organization and architecture with strategic DDD 🍼
Find a deck by Michael Plöd on the application of DDD to organizing teams. As to Convey's law, your architecture will resemble the organization, picking the proper team helps a lot. Pick up the DDD approach, identify contexts, make good boundaries and remember, nothing is perfect. The slide deck below.
#ddd
The security design of the AWS Nitro System 👷♂️
The software manages a lot of PII(Personal Identification Information), PHI(Personal Health Information) and PCI(Personal Commercial Information). Processing this data requires a high level of protection not only at the application but at the VM and hardware as well. AWS published a big material on how Nitro System works and how they provide a secure virtualization platform.
#security #aws
Design Considerations fpr Platform Engineering Teams 🍼
Following up the topic of Platform Teams, please find an article on the teams that might together represent the platform. Platform User Interface, Infrastructure, Services, Support and others inside.
#platform #bestpractices #teams
Discord
How Discord Stores Billions of Messages
We decided early on to store all chat history forever so users can come back at any time and have their data available. This is a lot of data: how do we do it?
👍5❤1