Observability with Charity Majors(CTO at honeycomb.io). This Wednesday, 19.00

https://youtube.com/live/2rSqFCkHdg4

Don't make this mistake of technical leaders

https://youtube.com/shorts/t4ySwZ9GAyE

Have to reschedule, sorry

Live in 90 minutes

Architecture Weekly #63 Highlights

This week I conducted an interview with Charity Majors(CTO at honeycomb.io). She told me so much about state-of-the-art modern observability, and it will be useful for you as well! Watch thoroughly!

Growth of AI Through a Cloud Lens 🍼
Do you remember the start of cloud computing? There was a lot of scepcis, like "real business won't use it", shifting to "Fortune 500 companies won't use it", until everybody was using cloud. Mitchell Hashimoto, a CTO of Hashicorp, recalls this trends and tries to analyze if it applies to the raise of LLMs. How much impractical for business the LLMs are? And how this area of impracticality will shrink in the future? Get a long read!

#ai #cloud

Twitter's Recommendation Algorithm 👷‍♂️
Have you ever ranted about how your tweeter feed contains some weird replies to the people you're not even interested in? Well, last week Twitter shared the source code for their recommendation service and wrote a blog post about it's architecture. Grab the details inside to both get some knowledge and improve your twitter visibility!

#casestudy

Spyware vendors use 0-days against mobile OSs and Chrome 👷‍♂️
Security news here. In Google, there is a Threat Analysis Group(TAG), which tracks malware vendors. Recently they discovered couple of highly impactful campaigns which use 0-days in Android, iOS and Google Chrome. In the article you will find the exploit chains description and what TAG has done about them

#security

Follow Up

Media Understanding Platform for ML Innovations 👷‍♂️
Netflix serves you content, but it also should help you finding it. Previously, the editor had to go through the hours of the shows hosted on the platform to pick up highlights or choose the engaging, story-telling dialogues. So Netflix decided to improve the efficiency of editors' work, introducing a solution to find the scenes in the show by a request, like "red racing car" or "dialogue with tears". Find out, what solution they ended up implementing!

#ml #batchprocessing

Cloud-Native Applications Design Principles 🍼
Cloud-Native applications is not merely containers with a couple of health probes. They are built with the set of principles in mind, like loosely coupled architecture, API First approach and others. Grab an article from JFrog blog to know 6 principles in details!

#cloud #architecture

Service level objectives 101: Establishing effective SLOs 👷‍♂️
The article titled "Establishing Service Level Objectives" on Datadog's blog provides a comprehensive guide to help organizations define, measure, and manage their Service Level Objectives (SLOs). The article explains how SLOs can be used to set clear and achievable performance targets for software systems and services, and how they can help organizations improve their overall service reliability and customer satisfaction. The article also provides practical tips on how to establish effective SLOs, including selecting appropriate metrics, defining appropriate thresholds and error budgets, and using feedback loops to continuously improve performance. Additionally, the article highlights the importance of aligning SLOs with business objectives and involving stakeholders in the SLO definition process

#observability

Getting started with small-step operational semantics 🤟
We discussed sagas many times. But now I want to share with you an article by Dominik Tornow about the formal language that helps to describe those. Grab on in the Temporal.io blog.

#saga #distributedtransaction

The cost of Architectural Complexity 🤟
It's pretty obvious that the more complex architecture you have, the more it's operation will cost. However this is not the only penalty you might experience. For example, the number and severity of software bugs increases with the complexity as well. What else? Follow the Abi's review of the paper from 2013 on the cost of architectural complexity.

#architecture #paper

Lambda vs Kappa Data Processing Architecture 🍼
The last one for today is the guide on the 2 approaches to the data processing architecture: Lambda vs Kappa. It has nothing to do with Lambda functions though. So the former is more complex and includes 2 data processing streams for realtime and batch processing; the latter is simplified and trying to do the job with only a single pipeline treating everything as a data stream. Find more details inside.

#data #architecture

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!

You shouldn't necessarily rewrite your monolith

https://youtube.com/shorts/VyMrurHsVBc?feature=share

Already tomorrow, at 5PM CET, I am speaking with Baruch Sadogursky on Developer Advocacy, work at Gradle and all that stuff! Book your calendars and of course, subscribe. Link to the stream below 👇

https://youtube.com/live/KXeTwgxs5wU

Live!

Architecture Weekly #64 Highlights

This week I conducted an interview with Baruch Sadogursky (Principal Developer Advocate at Gradle). We discussed Developer Advocacy, Development Relations and why people are confused with DevOps and DevRels roles. Watch it here!

The end of a myth: Distributed transactions can scale 🤟
I learned recently about RDMA - remote direct memory access, which allows bypassing the CPU while transferring data from one machine to another. And this development actually enables the performant distributed transactions. A paper that depicts the system design for it is being reviewed by Murat. Grab a hard, but insightful read!

#database #distributed #paper

Let's (not) break up the monolith 👷‍♂️
This week I published a short made out of the interview with Andrey Rebrov about microservices. The short was about should you really necessarily rewrite the monolith though(Damn, watch it already!). Following up the thought, Uwe accidentally started a series of articles on what people really want when they say they would like to break up the monolith into microservices. It appears, they don't want to rewrite it necessarily, but rather have more frequent deployment and more predictable development. Well, you can do that with monolith as well!

#architecture #monolith #microservice

16 System Design Concepts You wish you knew before 👷‍♂️
Another short guide to help you prepare to the system design interview. This is not like a super-full list but it will do as an introductory checklist.

#systemdesign #interview

Introduction to Data Modeling 🍼
We live in a time when it's essential to fast, so everybody is doing MVPs, and once they go live, the engineers never get a chance to rewrite a product, because you know, business needs revenue. The same happens with data: you never get a chance to model the data properly. But if you do, you need to know why it's crucial for your data engineering pipelines and what's there for it. Read a long read by Airbyte.

#data

Zero-ETL, ChatGPT and the Future of Data Engineering 👷‍♂️
Everybody heard about zero code, but the end of the joke is "zero code - zero jobs". This time we are hearing about Zero-ETL, which was introduced originally by Amazon. So the idea is that you will have only some code doing the job, and the pipeline will be provided for you. But is it really feasible? What are the nuances of such a stack? Some considerations inside.

#data #etl

How to build next-gen serverless applications 🍼
Serverless does indeed have servers in it, but you might not care, because it's a managed service. However, you still need to think about structuring your application, the communication between the apps and the resilience. Grab a short article on some of the considerations on serverless apps.

How to write RFCs and ADRs 🍼
And back to our favorite topic - RFCs and ADRs. I have shared several articles in the previous issues of the newsletter about why we need them in the first place. This time I found a good article on how precisely write them so that they are clear, concise and meaningful. Take a note!

#documentation #rfc #adr #howto

Architecting disaster recovery for cloud infrastructure outages 👷‍♂️
Cloud is still running on the physicals servers in the data centers. And there are electricity outages, floods, connectivity losses and other disasters. The Cloud systems should tackle this kind of problems still, that's why they put the disaster recovery plans. Read how they do it in Google.

#cloud #gcp #disasterrecovery

Lab Basics 👷‍♂️
And a little fun. Imagine you're experimenting with some IoT devices or having some working for your SmartHome. The good idea would be to organize a proper network setup to isolate one group of the devices from the another. Ken Moini dropped a nice beginning of the series, where he explains how to runs his own computer lab at home. Grab a read.

#network #casestudy #iot

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!

Architecture Weekly History Digest

This time I am sharing you not the new links, but instead some articles and videos from the most liked issues of the newsletter. Give them a thorough read , if you missed them or just remind yourself about their content ;)

How Discord Stores Billions of Messages 👷‍♂️
Amazing post from the CTO of Discord on storing the messages. You will find a business problem statement, inherited situation, articulated requirements and a detailed solution in the article. A true example of an architectural blog post. Deserves a first line in the newsletter issue!


SQLite Internals: Pages & B-trees 🤟
SQLite is a small database famous for it's reliability, small size, usage in the mobile applications and the fact that the LOCs of tests for it is greater that the lines of actual DB code. In this article you will find how SQLite encodes the rows, why we need pages and how b-trees allow to find data in log(N) time.

#database

Architecture for Flow with Wardley Mapping, DDD, and Team Topologies 👷‍♂️
With this talk I learned about Wardley Map - a tool to create the strategic design of an application or a service. Having introduced that, Susanne Kaiser shows how to apply it to online learning domain, add a bit of Domain-Driven Design, a grain of Team Topologies to come up with a well-though solution.

#ddd #devops #strategy #video

Reducing Logging Cost by Two Orders of Magnitude using CLP 🤟
The more your business grows the more you need to understand what happens within your system. With Uber and tons of analytical data and therefore log data generated each day, it became an issue. They need to retain a sufficient amount of logs to understand what happens within their Spark jobs and not pay for it too much. So they used a Compressed Log Processor, which was able to reduce the number of writes to SSD and store the logs in a searchable manner but occupying two orders of magnitude less space. More details inside!

#logging #costoptimization #bigdata

Zero Downtime Migration from HBase to Bigtable 👷‍♂️
Box - a cloud file storage and management system - used to store more than 80TB of user data in on-premise HBase. Last year they decided to migrate it all to fully managed HBase compatible BigTable. The requirement was to make to with zero-downtime, which could be challenging. Read about the PoC they did to choose the storage type, and how they ensured the zero downtime in the post by Axatha Jalimarada.

#migrations #cloud #bigtable

How to choose a database?

Just reminding you that I posted a video of a heading "Architecture Reading". I will go through the books and cover their content. Checkout and leave feedback!

https://youtu.be/M0u3btcLvZo

#video

Aligning organization and architecture with strategic DDD 🍼
Find a deck by Michael Plöd on the application of DDD to organizing teams. As to Convey's law, your architecture will resemble the organization, picking the proper team helps a lot. Pick up the DDD approach, identify contexts, make good boundaries and remember, nothing is perfect. The slide deck below.

#ddd

The security design of the AWS Nitro System 👷‍♂️
The software manages a lot of PII(Personal Identification Information), PHI(Personal Health Information) and PCI(Personal Commercial Information). Processing this data requires a high level of protection not only at the application but at the VM and hardware as well. AWS published a big material on how Nitro System works and how they provide a secure virtualization platform.

#security #aws

Design Considerations fpr Platform Engineering Teams 🍼
Following up the topic of Platform Teams, please find an article on the teams that might together represent the platform. Platform User Interface, Infrastructure, Services, Support and others inside.

#platform #bestpractices #teams

Через 4 минуты будут проводить System Design интервью)

Подключайтесь на встречу “Владимир Иванов, Август Вилакия: System Design Doodle”, к YouTube трансляции или в Zoom

We exceeded 1024 subscribers! Thanks everyone for the following 🙂 If you like the work I am doing, feel free to support here or here 🙂 Also, I got a couple of awesome people scheduled for the interviews the following weeks, so stay tuned for the updates!

Architecture Weekly #65 - Highlights

Tech Radar by Thoughtworks 👷‍♂️
The new issue of the Tech Radar by Thoughtworks. This time they highlight the rise of AI Tools for Software Development, marking GitHub Copilot for "Assess", highlighting the ease of adding accessibility to web and mobile applications and warning about the proper use of Lambda functions. See more details there!

#techradar

IaaS pricing patterns and Trends 2022 👷‍♂️
A major part of choosing a platform to host the solution is the cost. That's why it is useful to understand the cloud vendor's landscape regarding the price. Here you can find a report on pricing patterns and trends with data gathered at the end of 2022. Unfortunately, the vendors make it impossible to compare the VMs offerings, but the report tries to do it best. Looks like VMs are the most expensive at Amazon, but the disks there are the cheapest... but there is a lot of parameters inside.

#cloud #cost

The Fractal Geometry of Software Design 🍼
Vlad Khononov is one of my favourite authors on DDD. This time I am sharing his talk about energy transfer systems, which software design can relate to. And as such systems happen to be fractals, software design has the same patters. Sounds intriguing? Watch the video!

Architecture Weekly #65 - Follow Up

Intro to AWS Well-Architected Framework 🍼
If the organization is new to the cloud of need a substantial architecture change, AWS can support it with it's Well-Architected Framework: the collection of articles and papers on how to do things right, including Operational Excellence, Security, Performance and other important aspects called pillars. See more details in the article by "A Cloud Guru".

#aws #cloud #architecture

A brief history of high availability 👷‍♂️
What does high availability mean in particular? What's the difference between fault tolerance and high availability? How we came up with multi-active availability for databases? The answers to those questions you will find in the article in the Cockroach Labs Blog.

#db #availability #replication

How RocksDB works 🤟
Previously we already considered how some of the database engines could work, for example how B-trees can be leveraged for reading and writing data. I stumbled upon the article by Artem Krylysov where he explains how the RocksDB work - an embeddable database that powers Yugabyte, TitaniumDB and others - which happen to use LSM - a log-structured merge tree. Follow the article for the details!

#db #lsm

Cross-shard transactions at 10 million requests per second
Two-phase commits is a long-known technique for distributed transactions. Dropbox blog contains an article with the story of the implementation of this easy-on-paper protocol within a cluster of thousands of MySQL databases handling petabytes of metadata to support user-facing features. Follow the fascinating journey!

#databases #2pc

Replacing RabbitMQ with the Postgres Queue 👷‍♂️
Usually, when you need some queue processing you switch from a database implementation to something like Kafka. But Prequel did the opposite: having RabbitMQ to manage task queues they decided to switch to a PostgreSQL table because it is impossible to disable message prefetching in Rabbit. Have more details inside!

Grokking Scalability for System Design 👷‍♂️
And another primer on system design! This time on scalability. Find out the 2 types of scalability, the tactics for it and overall approaches in the Grokking Scalability article.

Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman and Egor for already supporting the newsletter. Join them as well!

This Wednesday we are talking about automated mobile testing with Marathon test runner author! Book your calendars :)

https://youtube.com/live/S-4XbJpwaaI?feature=share

Imagine your entire production environment going down or losing the database disks. What about a datacenter going off? What strategies should we plan to employ? How to execute the recovery efficiently? Figure out with Mikhail Druzhinin - an EM at DataDog with more than 500 hours of incident resolution time!

Broadcast

11.05.2023 18:00 GMT+3