Architecture Weekly
2.99K subscribers
4 photos
725 links
Architecture Weekly newsletter originated at https://blog.vvsevolodovich.dev. ~10 articles or videos on solution architecture and system design every week!.
Download Telegram
Architecture Weekly #57 Follow-Up

Multitask learning at LinkedIn πŸ‘·β€β™‚οΈ
When you have several learning tasks that are somehow similar, it makes sense to leverage multitask learning to increase performance and quality of learning. However, with such an approach new difficulties emerge like a different set of features for tasks, different input datasets etc. LinkedIn shared a post explaining how they handle those issues with their multitask learning framework.

#ml

Types of CQRS πŸ‘·β€β™‚οΈ
When you learn about CQRS, you typically think binary: either you have one, or you don't. Vladimir Khorikov reasons about CQRS in a gradual manner: you can indeed have no CQRS, but then you can have stage 1, 2 and three. Read the article to know the difference and when you need to apply a particular stage.

#cqrs #patterns #architecture

Scalable Annotation Service at Netflix πŸ‘·β€β™‚οΈ
Netflix wanted to store the annotations - some metadata - for any objects of theirs: from whole movies to particular timeframes. They wanted to do that in a unified, performant and scalable fashion. So they created Marken. Read how they employed Cassandra and ElasticSearch to accomplish the goal.

#netflix #microservices #architecture #casestudy

Consumer IoT Device Cybersecurity Standards and Policies 🍼
As you know, "S" in IoT stands for Security. That's why it is so mandatory to bring some clarity to the field and introduce some standardization. Please find a report on the policies, certifications and standartization procedures in different parts of the Globe in IoT field.

#security #iot

Software Build Systems 🍼
This week I took an interview with Artem Zinnatullin about Software Build Systems. We discussed what you should expect from a good build system - performance, scalability, learning curve - and what you, as an architect of a software architect, should know about them. We also covered the evolution of build systems from Ant to Bazel and speculated about their future.

#interview #video #buildsystem

ChatGPT is a Blurry JPEG of the Web 🍼
Did you know that Xerox copying machines are not doing the direct copy? Instead they scan the document, analyze it, compress and then decompress for printing. ChatGPT is simingly doing the same - analyzing and compressing the data from the web and giving us it's own - and blurry - image of the web. But what we prefer better?

#ai
πŸ‘4
Archtiecture Weekly #58 Highlights

Video

Platform Engineering Teams Done Right
πŸ‘·β€β™‚οΈ
Platform Engineering term is on hype now due to the popularity of Team Topologies and overall discussion on how to apply DevOps practices in a right way. Adrian Cockcroft is sharing his ideas about the principles you should build the Platform Teams. And the plural here is no wonder, as it appears you should have several platform teams covering different layers of a platform - and it's no wonder - due to the level of complexity we now have. Find a great article and example of organization structuring in the end of the article.

#platform #devops

Design Considerations for High Throughput Cloud Native Relational Database 🀟
Amazon Aurora is built with an idea of separation compute from storage for the resilience. The way how they commit data in multinode system is through redo log associated witha a database. The paper describes how the Network IO looks like in Aurora, how the traffic goes through it's nodes and finally shares the learning from building a complex cloud based database system.

#aws #cloud #aurora #database

Ongoing Security: a step-by-step guide to a secure app development process πŸ‘·β€β™‚οΈ
Some companies think that they can add security as some step in the CI/CD pipeline, but that obviously does not work. Security impacts the application architecture, the threats have significant impact on the data formats and protocols and many other reasons. This week I published the article explaining those and providing receipts what to do at every step of the development lifecycle. This is also just a first article in the series, so make sure to subscribe!

#security #mobile #ongoingsecurity
πŸ”₯1
Architecture Weekly #58 Follow-Up

Design Considerations fpr Platform Engineering Teams
🍼
Following up the topic of Platform Teams, please find an article on the teams that might together represent the platform. Platform User Interface, Infrastructure, Services, Support and others inside.

#platform #bestpractices #teams

ACID Transactions for Cassandra πŸ‘·β€β™‚οΈ
Cassandra is having lightweight transactions since 2013, but they are not enough as the users would like ot have the performant commits across data centers. Now it becomes available with the new breakthrough called Accord. Now they promise to guarantee an isolation for the transactions through a concensus reached in a single roundtrip. Follow the post for the details.

#databases #distributedsystem #cassandra

Sustainability Cloud Native Way 🍼
Energy consumption can be one of the quality attributes due to the regulations or just because electricity is expensive. RedHat suggests a solution to monitor the power consumption of Kubernetes workloads. Read more inside.

#cloudnative #cloud #sustainability

Secure AWS Organizations with Service Control policies πŸ‘·β€β™‚οΈ
When you start a new account in AWS, you start with the root account which in turns creates the first admin account. But what do you do next? How do you separate dev and prod environment? How do you manage the user permissions? Find the article exactly about that.


#aws #security #cloud

Big Ball of Mud 🍼
The phenomena of big ball of mud is known to everyone: tons of global state, everything talks to everything else, no tests, unmantainable etc. Typically the solution is to impose a design from some kind of an architect and rewrite the whole thing, but it rarely works well, and the question is how to avoid the bbom in the first place. Find a nice article which tries to analyze how we end up with such systems and what to do to make better systems.


#architecture

The Ultimate Guide to Software Architecture Documentation 🍼
And the last one of today is about documenting software architecture. We highlighted the topic many times, and I found a good piece again. They cover the goals of documentation and the reasons why having just code is not enough, like the code won't tell you the non-functional requirements ever. So grab a 12-minute read!

#documentation

And before you go, next week I am interviewing Magda Miu about what an Engineering Manager should know about Software Architecture, so don't forget to join the stream!

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
πŸ‘5
Architecture Weekly #59 Highlights

Uber's Move to the Cloud πŸ‘·β€β™‚οΈ
When you start a new product or a company, you always ask yourself: should I go with the cloud or do something on-prem? Of course, it takes a lot of considerations, starting with a budget and ending with the technologies and available talent. From this perspective, it is fascinating how Uber operating on-prem for more than 12 years decided to move to the cloud. The Pragmatic Engineer newsletter delivers 2 parts of this journey and covers the story of data centres at Uber, the reasoning for the migration and of course the cloud migration problems.

#cloud #migration #onprem

Roblox 3-days long outage 🀟
When a known company's services with millions of users are offline for 15 minutes it is already a tragedy. Facebook was down for several hours last year, and it caused a global panic. Roblox - a gaming company - was down for 3 days which is an infinite timespan. Roblox relies heavily on HashiCorp products like Vault and Consul, and the incident was connected to the usage of those - especially the usage of a new feature in Consul that triggered a pathological issue in its open-source engine - BoltDB. Find a summary on protocol.com, as well as a detailed Post Mortem from Roblox themselves.

#incident #hashicorp

Guide to the system design interview πŸ‘·β€β™‚οΈ
Aaand system design interviews again! This time I've got for you a very detailed guide which covers the structure of system design interview, the expectations from you as a candidate and of course the principles you need to be aware of in the majority of the system design interviews. The ways of thinking, the ways interviewers look at the candidate and real-life stories from FAANG(or MANGA?) - all inside the guide to the system design interview.

And if you need help preparing for one - you can always request a consultation here.

#systemdesign #interview
πŸ‘1πŸ”₯1
Architecture Weekly #59 Follow-Up

11 Reasons to use Lambdas 🍼
There are a lot of talks about serverless nowadays, and AWS Lambda is probably the most known, maybe after Firebase Functions. Is it just a fancy toy or you can really leverage it in the production environment? Find an article from SentinelOne, which explains the advantages behind Lambdas.

#aws #cloud #lambda #serverless

Comparing Security Frameworks 🍼
How do you know which security measures you need to apply in your organization? To answer this question security frameworks exists. Grab an article to understand the detailed comparison of four major cybersecurity frameworks, namely NIST 800-53, ISO 27002, NIST CSF, and SCF. The author highlights the strengths and weaknesses of each framework and explains how they differ in terms of scope, focus, and applicability. The article also offers practical insights and recommendations for organizations looking to adopt or integrate these frameworks into their cybersecurity programs.

#security #standard #csf

The Journey to Server Driven UI at Lyft Rentals πŸ‘·β€β™‚οΈ
One day, a ride-hailing company found that its mobile app's UI was becoming too complex and difficult to maintain, so they decided to adopt a server-driven UI approach to simplify the app and improve the user experience. The article from Lyft's engineering team details their journey towards implementing server-driven UI in their bikes and scooters app, the challenges they faced, and the benefits they achieved in terms of faster development, reduced app size, and improved customer satisfaction.

#mobile #serverdrivenui

Moving back end closer to the users with Edge functions πŸ‘·β€β™‚οΈ
Edge functions allow developers to improve website performance and user experience by moving the back end closer to the front end. This article from the StackOverflow blog explores the concept of edge functions and their ability to move the back end closer to the front end, resulting in improved website performance and user experience. The author highlights the advantages of edge functions, including faster load times, improved scalability, and reduced costs. The article also delves into the use cases of edge functions, such as caching and content delivery, and provides examples of popular edge function platforms, so grab a read!

#serverless #edgefunctions #lambda

Load Balancing Algorithms πŸ‘·β€β™‚οΈ
Large-scale applications and websites often receive a high volume of traffic that cannot be handled by a single server. Load balancing distributes traffic across multiple servers, enabling efficient resource utilization, reducing downtime, and improving application performance. This article discusses various load balancing techniques, such as round-robin, weighted round-robin, and least connections. The author also provides an overview of advanced techniques such as SSL offloading and content-based routing.

#loadbalancing #highload

Anti-Corruption Layer Pattern 🍼
If you read anything about DDD, you might have heard about the Anti-Corruption layer: a pattern useful when you have two subdomains which don't share the same semantics. Basically, you introduce a layer which will translates the messages from one system into the messages readable by the another. Find a detailed description of the pattern it's drawbacks in the article by Microsoft.

#ddd #patterns

Speaking about the DDD - this Wednesday I will be holding an interview the the author of "Learning DDD" - Vlad Khononov! Press the "remind me" button in order not to miss!

And of course, this week I published an interview with Magda Miu about "What should EM know about architecture". Find it here!
πŸ‘3❀1
Architecture Weekly pinned Β«Architecture Weekly #59 Highlights Uber's Move to the Cloud πŸ‘·β€β™‚οΈ When you start a new product or a company, you always ask yourself: should I go with the cloud or do something on-prem? Of course, it takes a lot of considerations, starting with a budget and…»
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them too!
πŸ‘1
Learning DDD, Software Coupling... Who you are thinking about in context of those terms? Yes, Vlad Khononov!

15.03 18.30 GMT+2 we will discuss Software Coupling and what are the biggest problems in DDD adoption. Join the live session to ask questions!

https://youtube.com/live/sCR-5s-ULgA?feature=share
πŸ”₯3
Video

Architecture Weekly #60 Highlights

Cloud-Bound Applications 🍼
If you consider the whole environment of the application running in a modern cloud, be it a container, a VM or a lambda function, you will discover two sets of bindings: compute bindings and integration bindings. While the first set is transparent to developers and basically means a form of delivery(docker container or just a binary), the second one includes all sorts of APIs and protocols to leverage the platform resources and external services. If you add their stateful and asynchronous binding you will get a landscape of a Cloud-Bound application. To know more, follow a long-read which is not poor with the illustrations though.

#cloud #architecture #vision

Container Security Fundamentals part 2 🀟
Containers are used everywhere, and there are tons of best practices around container security. But suppose you need to understand why those practices are in place, or you just want to understand container security better. In that case, I recommend reading a series of articles on container security fundamentals by Datadog. They explain how the containers are mostly Linux processes and continue to show that Linux applies the namespaces across multiple parameters to secure those. Follow the details inside.

#security #linux

Master the Art of Caching for System Design Interviews πŸ‘·β€β™‚οΈ
Caching is an essential technique in system design, which can help improve the performance of applications, reduce latency, and save resources. However, caching can be a double-edged sword, as it can also introduce some challenges, such as consistency and invalidation. This article provides a comprehensive guide on caching, covering everything from the basics to the advanced topics. It starts with an introduction to caching, including its definition, benefits, and types, such as in-memory, disk-based, and distributed caching. It then dives into the different caching strategies and algorithms, such as Least Recently Used (LRU), Least Frequently Used (LFU), and Random Replacement.

#systemdesign #caching #performance
❀1
Architecture Weekly #60 Follow-Up

Database Versioning Best Practices πŸ‘·β€β™‚οΈ
You can think that all the industry switched to versioning of the database schemes long ago and everybody is using something like Flyway or Liquibase for that. The reality is that it is far from true. That's why I decided to highlight a several-years old, but still very actual post by Vladimir Khorikov on Database Versioning. Grab 6 best practices on storing version inside the db itself, having the immutable SQL files, keeping them in the source control and some others.

#database #db #bestpractices

Large Scale Ad Data Systems at Booking.com using GCP 🍼
How much money Booking.com spends on advertising yearly you think? It's almost 5 billions! With such a budget, you definitely want to spend it wisely. In this article in their Engineering Blog, Warren Qi shares how Booking.com is using BigQuery to analyze the efficiency of their ad-campaigns, and how the Spanner is leveraged to combine different data into useful views.

#cloud #gcp #booking

How to do distributed locking 🀟
Locking is a typical approach for consistency: for example, you lock a row in a database so that only one process can modify it. However, in a distributed system it becomes much harder due to network delays. I want to share another old article on how to handle distributed locks by Martin Kleppmann - with an example of Redis. Grab a read.

#distributedsystem #locking

Zachman Framework explained in 12 minutes 🍼
Imagine you're looking into positions of Enterprise Architect, Chief Architect or a CTO. In any case you would have to understand a lot about the company. Zachman matrix is one of the enterprise frameworks that will help you to ask the most important questions on all the levels that matter. Follow the video by Mark Richards - a co-author of "Fundamentals of Software Architecture" and its sequel.

#ea #enterprisearchitecture

Transactional Data Lake with Incremental ETL at Uber πŸ‘·β€β™‚οΈ
Batch processing is a powerful tool able to process tons of data. However, if the major part of this data does not change much, you can lose on the efficiency and the latency. Uber's blog features an article by Sara Wilmes-Reitz on the idea of incremental ETL processes, allowing them to update the information much quicker and with lower latency. Follow the article for details.

#bigdata #etl #datawarehouse

Domain-Driven Design with Vlad Khononov 🍼
And this week I recorded an interview with Vlad Khononov - an author of "Learning Domain-Driven Design" and other O'Reilly-published books. Watch a video to find out how the DDD can help you be a better architect.

#video #ddd #interview

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
πŸ‘7
Architecture Weeekly #61

Video

Highlights
Google Sheets as backend πŸ‘·β€β™‚οΈ
When you start a new project you probably think, hey, I would pick MongoDB, or I want a PostgreSQL installation. But some successful projects start with none other than Google Sheets as their storage. It appears it is scalable, simple, and it's easy to integrate with. Find a great post by Levels.fyi - a site with literally salary bands of IT industry - where they show the evolution of their system design, which started with Google Forms and Google Sheets.

#casestudy

Reality Check Pattern 🍼
Have you ever encountered your navigation software telling you you're in a river while you were driving a highway? Well, Uwe Friedrechson just did, and a resiliency architect, he immediately came up with the pattern which he called "A reality check". It means that your system should check the signals from the outside world, and, if they stop making sense, change the state to something like safe mode, where it could not mislead the user or harm them. Grab more examples and conclusions inside πŸ™‚

#architecture #pattern #resilience

Distributed Machine Learning at Instacart πŸ‘·β€β™‚οΈ
Several issues ago I mentioned the KubeRay - a platform for running ML workloads on Kubernetes. And this week Instacart published a piece on building the distributed ML platform using exactly that. Find out, how they managed to build a scalable, efficient, diverse platform which allows for development and maintenance simplicity, as well as seamless integration with the previous platform called Griffin.

#ml #casestudy #k8s #kubernetes #kuberay
Architecture Weekly #61 Follow Up

AWS Aurora receives Write Through Cache πŸ‘·β€β™‚οΈ
Replication lag appears when the replication stream is generated too fast for a replica to keep up. To improve this situation, Write-Through Cache was introduced which decreases the replication lag by 44%. More details in the article.
#database #cloud #aws

Architects, Anti-Patterns and Organizational Fuckery 🍼
What is the difference between Principal Engineer and Software Architect? Should Architect be a Role or a Title? How to avoid screwing up your organization by introducing the Architects rotting up in their Ivory Tower? The person named mipsytipsi wrote a long read after a discussion in Twitter where the attitude toward the title was collected. Funny, that the post confuses the title and the role... but it's still an entertaining read nonetheless.

#architect #philosofy

Cloud Native Landscape πŸ‘·β€β™‚οΈ
The amount of tools for developing, deploying, running and monitoring cloud-native apps is over 1000. So you face a problem: what tools you would like to solve the expected issues? Worry no more: Cloud-Native Foundation has a tool called landscape, where you can find all the tools conveniently laid out by the area of applicability.

#cloud #cloudnative

Workflow Orchestration πŸ‘·β€β™‚οΈ
The common approach to a distributed transaction is a saga either orchestrated or via choreography. However, sagas compensation can not be always possible or at least expensive, like refunding the money sent for payment due to commissions. One of the ideas to solve the problem is the workflow approach, which camunda or temporal.io is implementing. This way you have a separate component which responsibility is only to manage the workflows and ensure they finish or complete gracefully. Follow the article for details.

#patterns #distributedsystems

Contract-Driven Development πŸ‘·β€β™‚οΈ
The "Contract-Driven Adoption" article on InfoQ explores how contract-driven development can improve the adoption of new technologies and tools in organizations. It discusses the benefits of using contracts to define expected behaviours and outcomes, reduce ambiguity, and enable faster feedback and iteration. The article covers different types of contracts, provides examples of real-world projects, and offers practical tips for using contract-driven development. Overall, it provides a useful perspective on how contract-driven development can facilitate technology adoption and improve collaboration among teams.

#microservices #contract #api

Microservice Dependency Graphs 🀟
The "Characterizing Microservice Dependency Graphs" article on MuratBuffalo's blog discusses a study on analyzing over 100 microservice systems using network science techniques to identify common patterns and properties. The article highlights key findings, such as the prevalence of power-law distributions, clustering, and hierarchical structures in microservice dependency graphs. The authors discuss the implications of these findings on the design and management of microservice architectures, such as the need for monitoring, fault tolerance, and modularization. Overall, the article offers insights for improving the scalability, reliability, and maintainability of microservice architectures.

#microservice #paper #tracing

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
πŸ”₯3
The biggest lie of IT industry
https://youtube.com/shorts/5lAIt_L6Hps
πŸ‘2
Why ChatGPT won't replace developers. Post for patrons and boosty subscribers

Patreon.
Boosty.
πŸ‘Ž4😁3
Architecture Weekly #62 Highlights

This week I decided to give YouTube shorts a try and made a couple of them. Tell me how did you like it? What are the good ideas for shorts?

Capturing a billion Emojis πŸ‘·β€β™‚οΈ
Sports events are crowded - and on the internet even to a bigger degree. The companies who handle the broadcasting are willing to make them interactive so that users can show their reactions to particular events of the show - with emojis of course. To make it happen, there should be some architecture in place. Find a good piece of engineering to solve this peculiar problem.

#highload #casestudy

Uptime Guarantees - A Pragmatic Perspective πŸ‘·β€β™‚οΈ
How frequently do you hear from the business they need 100% availability? More often than me getting incorrectly called Vlad(that's another name!). How do you turn down such requests? Get a pragmatic perspective: how to reason about the actual availability, how much does it cost to come even to 4 nines(99,99%) and what do the cloud guarantees mean?

#cloud #cost

DuckDB Hype πŸ‘·β€β™‚οΈ
A few issues ago I featured a post by Jordan Tigani with the statement that Big Data is Dead. But what is up to the replacement then? EasyData! DuckDB is trying to do the same for data analytics as SQLite doing for transactional processing. Easy, small, embeddable, lightning fast - all of this is about DuckDB. Check out the post by Oliver Molander where he analyzes all the hype around it.

#db #data #duckdb
❀2
Architecture Weekly #62 Follow Up

Vespa vs Elasticsearch vs Solt - Search Engine Comparison 🍼
Picking up a full-text search engine can be tricky, considering the different use cases and the range of capabilities offered by the engines out there. Should you pick Vespa? Maybe Solr will fit your task better? Grab a short article with a comparison table between Vespa, Elasticsearch and Solt.

#searchengine #solr #elasticsearch

Predicting demand at Airports 🀟
The taxi operators at airports face a problem: if there is not enough demand, drivers waste their time. If there are not enough taxis though, the experience for riders is unpleasant. Uber is trying to solve this problem with Machine Learning and predictive models. In this article you will find the formulas and the reference architecture for it, but I still wonder why checking out the arrivals table is not sufficient.

#ml
Operating Lambda: Performance Optimization πŸ‘·β€β™‚οΈ
When you start using Lambdas, the first problem you observe is a long startup time. A typical solution is to schedule an event which will ping your function every once in a while to ensure a warm environment. However, this solution is not universal: it does not help during increased load or starting a Lambda in a different AZ. Grab the read about available performance optimizations for Lambda and the anatomy of Lambda's start.

#cloud #aws #lambda #serverless

The complete guide to System Design in 2023
And another guide to System Design Interviews! This one will get you through all the foundational terms you need to be aware of to pass an interview. It covers storage types, scaling, redundancy, patterns, databases and many more. Follow it on!

#interview #fundamentals

A Simple Framework for Architectural Decisions
It sounds a bit too pretentious, but this article covers some clues you would like to adopt to simplify the technical decision-making in your organization. Those include building a Tech Radar, formulating the best practices like REST API or Observability standards, and documenting the decisions in the form of RFCs and ADRs. Grab more details inside!

#architecture #documentation

Migrating 125.000 users from Auth0 to Supabase
Kevin Peters decided to share the experience of Parqet company of migrating user authentication and authorization from Auth0 to Supabase. The author highlights the reasons for the migration, explains the process of setting up Supabase, and provides an in-depth analysis of the migration process. The article also compares the pricing of Auth0 and Supabase. It serves as a helpful guide for anyone looking to migrate user authentication and authorization from Auth0 to Supabase.

#casestudy #migration #authentication

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course, every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!
πŸ‘4❀2