Architecture Weekly #52 - Follow-Up

Reading DDIA notes 🍼
Designing Data Intensive Applications is a classical book nowadays. Magda Miu started a series of posts where she shares her notes while reading the book. She explains what she got from the book chapters, provides illustrations and complements it with the links for the further readings. Don't hesitate to follow her blog as well!
#ddia #reading #distributedsystem

Introduction to Time-Series Databases 🍼
We face time-series on multiple occasions from monitoring data to IoT and financial events. In order to store this data efficiently, there are special databases: TSDBs. What are there advantages and why they fit well with this type of data, read in the article below.

#database #timeseries

Consider All Microservices Vulnerable 🍼
Microservices usually have one well-defined purpose and narrow API surface. It helps improving the security, but does not obviously eliminate the threat completely. One of the tactics to improve the security of microservices in particular is monitoring their behaviour. If the malicious irefular requests and irregular response times are detected, then we can detect or even prevent an attack at an earlier stage. More details inside.

#k8s #security

Why Bolt use Node.JS? 👷‍♂️
Bolt uses exclusively Node.JS and Typescript for all the backend microservices, except ML and Data Science payloads. This is of course a architecturally significant decision which follows the business motivation of doing more with less. This week my coworker Denys Pysmennyi wrote a piece about this choice, it's consequences and drawbacks.

#backend #technology #nodejs

McDonald's Journey to Event-Driven Architecture 👷‍♂️
McDonald's is not only a set of restaurants for a quick break. It has rich IT infrastructure behind the scene. And the foundation for their systems is Event-Drivent architecture. In this post Derek Comartin shares some details from the McDonalds posts on how do they use Schema Registries, how they validate events and improve the messaging reliability.

#events
Any Decision Record? Architecture, Design and Beyond 🍼
Decision Records again! This time Olaf Zimmermann, the author of “Patterns of API Design", published a 10-minutes read about the applicability of ADRs for capturing not only architecture decisions, but as well as managerial or organizational ones. He also reasons what actually an architecturally significant decision is, and why a programming language choice is a one(mind the previous article). Get a read!

#architecture #adr

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!

Folks! I need a couple of developers with good GoLang experience and good English to interview candidates for a Go Developer position. Compensation is 100$ per interview. Drop me a DM if interested 🙂

Architecture Weekly #53 - Highlights

Video

Is Scalable OLTP in the Cloud a Solved Problem? 🤟
As you folks love papers, I bring a fresh interesting one to the Highlights. Murat Demirbas from AWS posted an overview of a paper called "Is Scalable OLTP in the Cloud a Solved Problem?". The overview and the paper itself describe the shared-nothing, single-writer and multiple-writers designs of a scalable OLTP database and discuss their tradeoffs. Murat also highlights that the paper fails to consider metastable errors(which you can find a paper on here). The second part of the original paper is about a new DB and that's not that interesting, but the first part is gold.

#database #cloud #oltp #paper
Yandex Services Source Code Leak 🍼
The internet blew up this week with a major data source leak from a russian IT giant Yandex. The dump contains 44 GB of source code from their monorepository - which was not a single storage of the code in the company. The dump contains tons of services, configs, user-generated data like prompts for Voice Assistant and even database credentials for insider access. Although there are no trained models though, and you can't really build anything from it, as yandex infrastructure is required, you can still use it to learn how the things are done at a company of that scale.

#security #breach
System Design Interview Survival Guide 🍼
I know you folks read the newsletter partially because you want to be ready for the design interview. And the 3rd highlight today is the survival guide for exactly those. The article goes over the list of concepts to be aware of, the architecture patterns, database types, distribution system problems etc. Use as a nice checklist to prepare for the interview! And of course, you can reach out to @vvsevolodovich if you need a personal consultation 🙂

#systemdesign

Follow-Up
Software Trends Report: What to Watch for in 2023 🍼
As we just wrapped up the 2022, it is good to get back and get an overview of the trends of the last year and see what we can expect in future. In this post at InfoQ they mention the use of ADRs, importance of Senior IC role and the emerge of Platform teams. They also highlight that you would rather treat platforms and data as products and handle those accordingly. As regarding the software architecture I've invited Uwe Friedrechsen for an hour long interview, so book your calendars!

#architecture #review
How to autoscale your SaaS application Infrastructure 👷‍♂️
RedHat has a series on SaaS Architecture Checklist. Recently they shared Chapter 9 which is about autoscaling the infrastructure. Michael Hrivnak explains how autoscaling can be configured using Cluster API, Cluster Autoscaler and Horizontal Pod Autoscaler so that your SaaS resources are truly elastic.

#kubernetes #saas #autoscaling #scalability
Must Read Free Kubernetes Books 🍼
Speaking of Kubernetes, take a look at the set of free books which help you to study Kubernetes itself, it's performance and security. The set also includes books on integration patterns and designing distributed systems.

#kata #books
Solution Architecture Kata Example 👷‍♂️
One of the books I recommended in this post, was "Fundamentals of Software Architecture" by Neil Ford. He also knows as a promoter of "Architecture Kata" exercise and he shows an example of a task for such activity in the aforementioned book. What I want to share with you is solving a problem to that particular Kata in this post on Medium. It includes understanding the context and iterating over the design versions. Take a look.

#architecture #kata
Document your event-driven architecture 🍼
In a loosely coupled, event-driven architecture we still want to know What events/messages is a particular service producing, what events/messages can I consume from the service, what is the message schema and more. This short article shows the 3 ways how you can document the answers to those questions.

#documentation #eda
Metrics for quality 🍼
When you introduce any metric for your software project, team or whatever else, you will always find out that the metric stops being useful due to Goodhart's law. So if you need to optimize your product for quality, what do you do? Well, if you study the quality itself, you will find out the outer, inner and process quality parts. The article though does not answer what metrics to pick - just how to think about the quality itself.

#quality #qa
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!

Time for the announcements! Fasten your seat belts 🙂

1. Software Architecture in 2022 Review with Uwe Friedrechsen happens on Feb, 6, 2023 at 2 PM GMT+2. Broadcast here.

2. Where do microservies go? Interview with Andrey Rebrov about do we go back to a monolith or lambda, what technologies support the microservies and what an ideal ms looks like.

Feb, 9, 2023 at 4 PM GMT+2. Broadcast here.

Make sure to subscribe and click the notify button 🙂

Architecture Weekly #54 - Highlights

Video

MySQL vs PostgreSQL 👷‍♂️
When you ask yourself what relational database you want to pick for a new project, you would probably name MS SQL, Oracle, MySQL, PostgreSQL, and maybe some more. How would you compare the choices? You definitely want to see the data types support, indexing, performance, replication, support for transactions and isolation levels and more. Today I am sharing a good comparison between MySQL and PostgreSQL which includes all of the above. The article also shows that migration is currently in favour of PostgreSQL. Check out the link to understand why.

#mysql #postgresql #database #comparison
Gentle introduction to DDD 🍼
I touched on the Domain-Driven Design topic several times in both the newsletter and the blog itself. If you're new to a topic, want to refresh your knowledge or are just not sold whether you need to read a book, find a lengthy article on the introduction to DDD, which basically covers the content of "Learning the DDD" by Vlad Khononov briefly. But what's good about it - it has a healthy dose of funny memes which improves the reading experience significantly.

#ddd
Asynchronous computing at Meta 👷‍♂️
Meta has tons of data, and they prefer to handle their workloads asynchronously, so they built their own ordering queueing system called FOQS(wonder, what it says) and employed a component called Dispatcher that would manage the ingestion into it. Although it was working fine, the team noted that the Dispatcher incorporated too much of responsibility for consumption, rate limiting, routing the workload and many more. So they decided to improve the async layer architecture and they shared how they did it and why. I understand the problems they face like retries with streams, delays, etc but like really? Why do you reinvent the already existing streaming solutions? Maybe you will find the answers in the article.

#architecture #casestudy #queue

Architecture Weekly #54 - Follow-Up

Unlocking the power of service mesh 👷‍♂️
The service mesh is an idea of adding the features, common to all the microservices not through libraries, but by runtime components running alongside the actual services. But with all the traffic management, routing and security features meshes bring, the complexity of those solutions grows as fast. Please find a 5-part series of articles on the services meshes which guides you through the introduction to service meshes, names a few examples, and demonstrates their internal architecture and working principles.

#servicemesh #patterns
List of books for software design 🍼
Learning and mastering software design comes not only with practice, but also knowing the theory. Grab an article with 8 books you might appreciate to build better applications and more robust systems.

#systemdesign #books
A short note on idempotency 👷‍♂️
Exactly once is a holy grail of events-based systems. You possibly can not have this in general, but you can help the issue if your event handling is idempotent: receiving the event the second time won't affect the state of the system. Get a short note of the explanation.

#idempotency
Actor model explained 👷‍♂️
Find another short note, this time - on the actors. Basically, the author tells the story of transitioning from the arbitrary procedure-style way of implementing a synchronization lock to an actor-based implementation, where you send the events from several computing instances in an async manner and letting another instance handle them thus removing the performance constraints. More details inside 🙂

#async #patterns

Critique of the Spotify Model 🍼
And the last one for today is about team organization which is frequently part of an architect's job as well. You might have heard about Spotify's model with autonomous cross-functional teams and people misunderstanding the principles of those. Grab 2 articles on why it's important to have aligned autonomy, TechRadar, support for Golden Path and more!

Starting in 4 minutes!

https://youtube.com/live/mDDgDbR5kFs

Today I am interviewing Andrey Rebrov - a co-founder & CTO at Scentbird - about microservices. How we came to them, what are the microservices problems, what's the ideal size of a microservice and where do we go with them. Join the broadcast at 4 PM GMT+2! Make sure to attend, as the recording will be taken private for a couple of weeks

https://youtube.com/live/PboH8FFqq5c

Starting in 4 minutes!

https://youtube.com/live/PboH8FFqq5c?feature=share

Architecture Weekly #55

I will publish the video for this issue tomorrow. However, last week I conducted not 1, but 2 interviews. The first one is with Uwe Friedrechsen, where I ask him about where software architecture is going in 2023. The second one is where Andrey Rebrov, a CTO from ScentBird, tells us about the microservices, where we go with them, how to pick the right size for a microservice and how they transitioned from a Grails monolith to Java microservices. Both videos are available for Patrons and Boosty subscribers.

Highlights

Breaking up a Monolith by Khan Academy 👷‍♂️
The question "How to do you transition from a monolith to microservices?" is very real. And fortunate you are, Khan Academy just completed a project of migrating from a Python 2 monolith to a set of Go microservices across the timespan of 3,5 years. Such projects are challenging from so many perspectives: how do you start? Do you apply Agile practices or handle the project as a waterfall one? How do you make the transition: per feature, per domain, per function? How to retain people over this timespan? Khan Academy shared what went well, and what choices they would reconsider. A very insightful article by Gergely Orosz.

#microservices #refactoring #casestudy

How to fix a $1 Million Query 👷‍♂️

Shopify decided to use BigQuery as its Data Warehouse solution. They had to process the data of their customers. But once they estimated the cost of a single query type running for their volumes for a month, they got a walloping value of approx 1 million dollars. That definitely rang a bell, so the engineers went to find a way to optimize that. It appears that a where clause and a cluster feature of BigQuery led them to the solution. Check out which one.

#bigquery #cloud #gcp

Is software getting worse? 🍼
And the last highlight for today is a post in StackOverflow blog on the software getting worse meaning it's heavy, slow and buggy. Ryan Donovan argues that's wrong to blame the programmers: they really want to make fast, compact software; but the market just does not care! The very interesting business reasoning inside.

Architecture Weekly #55 - Follow-Up

Investing Wisely in Architecture Through Refactoring 🍼
Simon Brown said that industry was transitioning from big upfront design to no design at all. The same might happen at an application level: investing in the design too much makes it a bit overengineered; once you learn you will start doing less designing until you find the right balance. J. B. Rainsberger reason this tendency and finds that it is an essential part of learning. You can't predict if you're overdesigning or under-designing, but that's not an issue if you can get out of it via refactoring. And exactly at this point, you might need the architecture patterns.

#architecture #refactoring

Kubernetes Patterns e-book 👷‍♂️
Speaking about the patterns, RedHat just shared an online book published with O'Reilly on the patterns. So you will find foundation patterns like declarative deployment and managed lifecycle, behavioural patterns like batch and periodic jobs, and structural patterns like sidecard and adapter and many more inside.

#patterns #k8s #kubernetes

Designing modern frontend app architecture 👷‍♂️
So you decide you want a new big frontend application - maybe you're building something from scratch, or you're rewriting an old big ball of mud into something more appropriate. How would you approach the front-end architecture? What quality attributes do you need to take into account? Find a well-structured and illustrated article on what you should be aware of to build a good solution.

#architecture #frontend

Modern Cloud for Multiplayer Games 👷‍♂️
I am a personal fan of StarCraft 2 - an online RTS game. But as a software architect I always wondered how you design such a system given the requirement of synchronization of the game data between multiple players simultaneously supporting low latency requirements. This week I stumbled upon an article which digs into the technical challenges of multiplayer games and explains how they can be solved within a cloud environment.

#architecture #gaming #multiplayer

Raft visualized 🤟
Raft is a distributed consensus algorithm. They are typically hard to comprehend, and that's the reason good visualization can be highly valuable. Grab one for Raft and go step by step through the concepts of consensus, leader, follower, leader elections and other related stuff.

#distributedsystem #consensus #raft

Using AI for system design and diagram creation 👷‍♂️
Talks about ChatGPT are all over the internet right now. Some people use them for writing cover letters, some create manuals, and some even suggest building YouTube video scenarios with it. Software architects though are experimenting with building system design with it. Find a receipt for formulating the data model and sequence diagrams for a wealth management system using ChatGPT below.

#systemdesign #ai #chatgpt

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!

Video version!

https://youtu.be/REkLJ-vmMks

Recorded a new video - this time on task delegation. Tried to make it fun, tell me if I managed to do so 🙂 https://youtu.be/_Ca9zHMT8AA

Architecture Weekly #56 Highlights

Will my cloud bill go up with looser coupling? 👷‍♂️
Coupling is considered a bad thing, so architects usually try to decrease it for better flexibility and maintainability. But what about the cost of infrastructure? Gregor Hohpe picks up an example of a Lambda function writing data to DynamoDB and sending an event to EventBridge to see if moving the event handling to DynamoDB streams will increase the cost. But the main point of the article is in its end: whatever the infrastructure increase would be, you will probably save a couple of orders of magnitude in value brought by your programmers.

#architecture #cloud #cost

Big Data is Dead 🍼
Big Data was on the hype for over a decade. The reason for that was the expectation of the neverending growth of data and the growth of the companies handling this data. But it turns out that really large volumes are only handled by a handful of companies, and even for them the workloads are much less than the total amount of data. What does that mean for you? Find out in the article below.

#bigdata

Distributed Reset 🤟
Last time we shared a visualization of Raft consensus protocol. But it is a relatively recent development. But the problems of distributed systems are much older. For example, we had a problem of distributed reset: setting a default value for all the nodes in the system. The original paper was written in 1994, but this week Jesse Jiryu Davis wrote a review for it. The basic idea is close to the Deijksta path search algorithm with the difference that here we need to traverse all the nodes in DAG. Grab the details inside.

#distributedsystem

Architecture Weekly #56 Follow-Up

Evolution of a React application 👷‍♂️
React is usually perceived as a framework, but it's rather a library for building UI. And the application architecture is still up to you. Juntao QIU from ThoughtWorks wrote an article in Martin Fowler's blog on how you can evolve your frontend application which uses React from a single component to a sophisticated multilayer application. Grab a great read.

#frontend #react #architecture

How the Istio Service Mesh became the Critical Infrastructure for CN apps 👷‍♂️
We spoke about service meshes in previous articles; this time I would like to share an article which covers the history of evolution from plain Kubernetes systems to using Services Meshes and the place of Istio there. The article also covers the evolution of proxies from locally configurable to cloud-native solutions at scale.

#servicemesh #istio #cloudnative

DevOps Roadmap 🍼
It's impossible to imagine a high performing organization without DevOps practices nowadays. So you might end up in situation where you need to hire SREs and other relevant specialist or grow them inside the company. The roadmap I found on the internet does a good job depicting the areas of expertise required nowadays from git and programming languages to Clouds, CI/CD and Monitoring and Observability.

#devops

Guide to Projections and Read Models in Event-Driven Architecture
With event sourcing, you typically have a fixation of facts or event happened in the system. However, we frequently need to understand the current state. Projections can help to build up this state from the recorded events. Oskar Dudycz shares the explanations of what exactly projections are in his event-driven architecture blog.

#eventdriven

Postgres WAL Files and Sequence Numbers 🤟
WAL files are used for the recovery and replication purposes. But if you need to dig deep into some issue, you might find useful the understanding of the WAL files naming conventions, log sequence numbers and segment positions. This is why I am sharing an article by Crunchy Data with you.

#postgresql #db #wal

Reference Architecture and Implementation for Deployment Pipelines 👷‍♂️
If you ever needed to build a deployment pipeline, you know how many details are there. Last week AWS published the reference architecture for the deployment pipeline and InfoQ blog has a nice article covering them. Find out how you can build a pipeline using AWS tools and services and how to follow the best practices.

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!

Good news, everyone! Artem Zinatullin agreed to an interview about build systems and what a software architect or staff engineer should know about them. Join this Monday at 6 PM EET. While you wait, the video with Uwe Friedrechsen became publicly available here.

Artem didn't join, will reschedule

New stream today 6 PM EET(GMT+2)! Same time, but today 🙂
https://youtube.com/live/I5qLNIFsywQ?feature=share

Starting in 4 minutes!

https://youtube.com/live/I5qLNIFsywQ?feature=share

Video of "Where do Microservices go?" with Andrey Rebrove is publicly available!
https://youtube.com/live/PboH8FFqq5c