Architecture Readings. Database Internals: Part 2. Binary Search Trees and B-Trees
https://www.youtube.com/watch?v=LolbCFY1gpg

Architecture Weekly N43

Video

Highlights of the Week

The industry comes through a tough period: Big Tech is laying off people; those who don't - freeze the hire. According to layoffs.fyi, the Bay Area got ~70,000 engineers freed of their jobs. However, Adyen keeps hiring to make a day positive.

This week I would like to bring your attention to several articles about data management and the future of Frontend in the highlight section. Another seven awaiting you in the follow-up section.

Making Schema Changes Safe with Raft 🤟
Schema change is adding or deleting a column, changing column types etc. ScyllaDB is a distributed database, and it should apply the changes across the cluster. But this is a hard task and bugs are being reported over and over again. So they applied Raft to solve those problems. From the video, I discovered the prevoting procedure; pretty sure you will find a lot of interesting for you as well!

#database #raft #distributedconsensus

The future history of Data Engineering 🍼
Matt Aderne makes an exercise of predicting the future of Data Engineering. He extrapolates that with the adoption of cloud services and the rise of Data Processing as a Service(Tech abstraction aaS) the demand for data engineering will drop focusing on tooling and orchestration instead of ETLs. Debatable? Indeed. Interesting nonetheless.

#dataengineering #forecast

The state of Frontend in 2022 👷‍♂️
Gergely Orosz published the state of Frontend report. Technology-wise: TypeScript is at 80% of adoption. Surprisingly, 20% of developers are not using CI/CD despite the impression that this is a standard nowadays. More insights inside!

#frontend #statereport

Follow Up

Aligning organization and architecture with strategic DDD 🍼
Find a deck by Michael Plöd on the application of DDD to organizing teams. As to Convey's law, your architecture will resemble the organization, picking the proper team helps a lot. Pick up the DDD approach, identify contexts, make good boundaries and remember, nothing is perfect. The slide deck below.

#ddd #teamtopologies #accelerate

Accelerating the LinkedIn experience with Azure Front Door 👷‍♂️
Another excellent article from the LinkedIn engineering blog. Serving data to far locations affects latency significantly and worsens the user experience. Points of Presence help with it by heavy lifting the HTTPS and TCP/IP connections between the end user and itself, while maintaining the connection to LI servers. Brilliant illustrations and lessons learned inside.

#cloud #edge

Machine Learning for Fraud Detection in Streaming Services 👷‍♂️
Streaming platforms use limitations for the content from the perspective of the number of devices, accessing particular content etc. The implementation of those limitations can be tricky. Netflix heavily uses machine learning based on anomaly detection to detect fraudsters. More details in their tech blog.

#ml #fraud

Scaling customer review system for peak traffic 👷‍♂️
Fresh post from the Booking.com tech blog on the system scaling for customer reviews. To prepare for picks in traffic they have the load testing with predicted data, have observability in place and employ a consistent hashing algorithm to distribute the load. Details inside!

#performance #scalability #consistenthashing

Lyfts' SafeMode 🍼
I remember when you roll out a new feature in a mobile application, and it starts crashing right from the first screen. It is unacceptable with any scale of a mobile app, but the complexity of applications is extremely high. In order to safeguard Lyft introduced SafeMode: an approach to detect a crash, disable a feature flag and report the issue. Read how it helped Lyft!

#scalability #resiliency

Is Enterprise Architecture Dead? 🍼
Short note on Enterprise Architecture continuing the topic of Road-way architecture. As we remember the global rearchitecting never completes; instead you need to be flexible and extendable. Navdeep Singh shares the lesson learned.

#enterprisearchitecture #ea

Why B-trees are efficient? 🤟
I am continuing reading the "Database Internals" book. This time it's chapter number 2 where I cover the binary search trees and B-trees. Find out why the latter are used in databases!

#video #database #btree

Like the newsletter? Consider helping to run it at Patreon or Boosty. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel and Robert for already supporting the newsletter.

Made a short post on the examples of bad articles with falsely promising titles which I will definitely not include in the newsletter. Find it on Patreon(EN) and Boosty(RU).

Archtitecture Weekly #44

Video

Highlights

This week Spring announced the general availability of version 6.0 bringing baseline to Java 17, support of Java EE 9 and, what's more important, support for AOT Spring Application Context and enabling of Graal VM support for Sprint Boot 3. Aside from that, I want to share a couple of great decks and articles with you in the highlights, and all the rest reside in the
Follow-Up section.

The Distributed Computing Manifesto 🍼
This week the CTO of Amazon - Wener Vogels - published a Distributed Computing Manifesto dated 1998. In this manifest Amazon states that support order of magnitude bigger amount of orders they need to shift both the system approach as well as development culture, separating data from applications and introducing strict business checks. A fascinating read which still resonates.

#distributed #aws

Apache Pulsar, Splunk and 10 Petabytes of data daily 👷‍♂️
Apache Pulsar is a Kafka competitor being a streaming platform. Splunk showcased how they solved a problem of ingesting and transforming 120Gbps of logs for several of their clients. Calculations, Design decisions and optimizations inside!

#pulsar #streaming

Managing Your Data Lifecycle with Time-To-Live Tables 👷‍♂️
Time to live is a common capability. For example, DynamoDB has it. If you implement the OTPs on your own, you will find that useful. TiDB has it as well, and automatically removing data can also help with the cost of your storage. Read how TiDB supports TTL on row and partition level and what issued they faced.

#database #tidb

Follow-Up Section down below 🔽

Architecture Weekly N44 - Follow-Up

Using the cloud to scale Etsy 🍼
Please welcome a story from Etsy - a unique art items marketplace - on how they migrated to the cloud in expectation of 3times growth. This is the first article in the series. Still, it already covers the principles of migration and the challenges like transitioning the system from a LAMP monolith to a cloud-native application and migrating the data lake to BigQuery.

#casestudy #scalability #cloud #migration

API Design for teams 👷‍♂️
It's relatively easy to design an API with a single team. However, once you have more than 3, it becomes a problem to align them. Nobody wants having to save an entity in API 1 via POST method, and API 2 uses PUT method and different body format. In order to align the design you to approach API from a perspective of use cases, requirements, common tools and agreements. More details inside.

#api #openapi

Strategies and Tools for performing migrations on platform 🍼

Spotify shared a bunch of receipts on strategies for platform migrations. They speak about mobile, but it does not really matter: changing platform faces similar challenges no matter the technology. Key takeaways: focus on stakeholder, communicate extensively, make PoC and allow teams to help you. More details in the article.

#strategy #spotify #platform #migration

Lesson learned on Migrating to AWS Aurora 👷‍♂️
Kimberly from Gridium shares a case study how they migrated from Postgres RDS to AWS Aurora. The reason was as the storage in a cluster came with a cost despite not being fully used, and they hoped to reduce it by using Aurora. But it rendered to be even more costly! So the team had to put some effort to analyze what causing the biggest part of the bill and then optimize it. Chances are the migration was not required in a first place given the same optimization. Figure out the details!

#aws #cloud #aurora #database #migration

Event-Driven Architecture - 5 Pitfalls to Avoid 👷‍♂️
Event Driven architecture comes with benefits like loose coupling, ease of debugging by replaying the events, etc. However, it is also easy to make trivial mistakes like forgetting atomicity in issuing an event alongside writing to the db, overusing event sourcing, large payloads in the events, etc. Natan from Wix shares the remedies for those mistakes.

#eventdriven #patterns

Introduction to Chaos Engineering 🍼
Chaos Engineering is a practice of checking how your system behaves if something really bad happens, like a whole datacenter going offline. It is tempting to just simulate the event and then trying to fix the discovered problems, but you can do it more effectively, if you follow the best practices like formulating hypothesis, testing in production and others, listed in the Harness.io blog.

#chaos #testing

Enterprise Architecture Tools 🍼

What the hell does Enterprise Architect do? Well, strategic decisions and technology landscape management are among the position roles. In order to do that the EA should understand the capabilities, what applications support them and how well do they do. Tools like Archimate, Red-Green-Amber analysis and other can help a lot. If not - you can always build one with Excel!

#ea

Like the newsletter? Consider helping to run it at Patreon or Boosty. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert and Roman for already supporting the newsletter.

Two announcements! This monday I will do 2 public events online. First is a showcase how to come up with a topic for a public talk. Right on the session we will figure out what I can share with an audience and how to build a story on that - in English. Will be hosted on Architecture Weekly YouTube Channel. Link is here.

Another one in russian - I am participating in a book club. We are discussing the first chapter of “Fundamentals of Software Architecture”. Here is the announcement.

Architecture Weekly N45 - State of AI 2022, Policy in CD, Vulnerability Management at Lyft and many more!

Video

Highlights

State of AI 2022
AI made a boom in social networks with text-to-picture engines, and - to a less extent - with Github Co-pilot. But the most interesting things happen below the level of public vision. State of AI Report highlights the amazing breakthroughs from the DeepMind team and other researchers. It also covers the regulations, hardware markets for AI, AI safety concerns and makes predictions for the next year. 100 pages of a report, but worth to click through.
#ai #ml

Policy in Continuous Delivery
This week Continuous Delivery Foundation introduced a new term - "Policy". With the growing complexity of software and delivery pipelines, some aspects can be done ad-hoc: like requiring a static code analysis step or forbidding rolling out changes on Christmas. In order to govern such restrictions CD pipelines would benefit from a single source of policies. Follow the article for the first tools which can provide such capability.

#cd #policy

Vulnerability Management at Lyft and fix cascade
Suppose, you find a vulnerability in a service; how to understand its impact? How to find out how it was introduced? How to fix it efficiently? Lyft shares its approach with cartography: mapping the connections between different entities in the first part of the article. The second part is the propagation of fixes across Docker images. Read the instruction inside!

#security

Architecture Weekly N45 - Follow-Up

The security design of the AWS Nitro System 👷‍♂️
The software manages a lot of PII(Personal Identification Information), PHI(Personal Health Information) and PCI(Personal Commercial Information). Processing this data requires a high level of protection not only at the application but at the VM and hardware as well. AWS published a big material on how Nitro System works and how they provide a secure virtualization platform.

#aws #security
Google Cloud Platform security best practices 🍼
Once you setup a new project within a cloud provider, it's crucial to follow the best practices regarding security, like not doing anything with your root account, configure the rotation of the keys, enabling some defaults etc. In this post you fill find a long checklist of things you need to do in your GCP account before you deploy anything.

#gcp #cloud #security
Building a JSON CRDT 🤟
Paxos, Raft and other algorithms are a way to establish a consensus in a distributed system meaning there is only one true state of the data. What if we shift our understanding and say that every node of the system - including mobile and desktop clients - can have their own state? And that's why, if we demand that the state will eventually match if every node receives the same update. This is the idea behind CRDT - data types which allow building such a system. Read a long blog post which gives more details and shows how you can build such data type yourself.

#distributedsystem #crdt
Designing Embedded Software 🍼
Embedded Solutions bring new unobvious challenges to software architecture. Portability, Performance and Security are among them. Surprisingly, separation of concern works really well. Read the 2 articles from the series at embedded.com.
#architecture #portability #embedded

12 Factor App Revisited 🍼
Heroku was a popular platform to deploy apps to in the early 2010s. The founder of Heroku formulated 12 factors for the successful development of software as a service, from code repositories to managing logs. In this note, Mahdi Yusuf from "Architecture Notes" checks what changed in those 12 factors. Spoiler: the majority is still there, but an understanding of working with repositories and code storage definitely evolved.

#architecture #bestpractice

What is an Internal Developer Platform? 🍼
DevOps was a great idea: let's use some code to manage the infrastructure and run our services. However, the growing complexity of cloud services, technologies and approaches imposes a high cognitive load on developers. That's why the companies transition to the Developer Platform or Platform Engineering. This is what we do in Bolt as well: the developer only focuses on the business logic; the whole runtime is handled for him by a builder and runtime solution. Read, why this happens and why it's not a necessarily bad thing below.

#devops

5 diagramming tools for enterprise and software architects 🍼
This week I wrote a short article on diagramming tools. It is similar to what I wrote a year or so ago about 5 diagrams you only need to describe software architecture. But the peculiarity of this one is that it was a guest post on RedHat's blog. Read it too 🙂

#documentation

Have an article to suggest? Drop us a message at architectureweeklynewsletter@gmail.com!

Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert and Roman for already supporting the newsletter.

If you have an article you would like to be included in the newsletter, you can drop a message with a link to architectureweeklynewsletter@gmail.com .

Architecture Weekly N46

Video

Highlights

The biggest event of this week is definitely Amazon re:Invent and so far the most exciting announcement is the Amazon Lambda cold start improvement by 90%. This was the major showstopper for many services, and now no more. Another item worth noting is the start of the Advent of Code - everyday programming tasks for the whole of December.

Synthesized.io and TestContainers integration 🍼
We all love integration tests, and TestContainer is an excellent tool to set up them. However, in order to have valuable tests we want to run them against the close-to-production data. This is where a Synthesized solution comes in handy. And the good news you can now use both products combined. Check out more details in the video.


Improving Kafka client performance by 40% by cleaning up unused Topic Metadata By LinkedIn 👷‍♂️
You might wonder how empty topics can slow down a Kafka Cluster, but they actually do: metadata is still stored in ZooKeeper, the partitions polls it etc. At scale it can be significant. LinkedIn Engineering Team came up with a TopicGC service, which will monitor for empty topics and drop them. Get more details inside.

#kafka

Preparing for the Systems Design Interview 🍼
System Design interview is a common step in the hiring process in many companies. Gergely Orosz made a cool write-up half a year ago on the books, courses, prep materials and mock interview resources which can help you to prepare. I can personally help with them too 🙂

#systemdesign #course

Architecture Weekly N46 Follow-Up

How do One-Time passwords work? 🤟
We all use the OTP for 2FA with Google, Github and other services. But how do they work? And who to implement the same scheme for your service? Read a short note below.

#security #encryption

Create Your Distributed Database on Kubernetes with Existing Monolithic Databases 👷‍♂️
Kubernetes gained huge success in container orchestration. But using databases in with it is still not a completely solved problem. The article describes how you can upgrade your database to be distributed and pair it with your K8s workloads through Apache ShardingSphere that we have already spoke(in #34) before.

#kubernetes #k8s #distributedsystem

Discovering performance issues at Netflix 🤟
Netflix managed to triple it's RPS per VM Node. However in order to do that, they had to go way down to the JDK implementation and patch the two problems: "False Sharing" and immediately after that "True Sharing", which happened for super-class cache. A complex, but fascinating read about JVM optimization. If it was a bit hard to understand, there is an explanatory article here.

#performance #jvm #vm

Event-Driven Architecture Patterns by Wix 🍼
Wix is a Saas, which allows you to create and host a website in minutes. Natan who is leading their data platform shares 6 receipts for different use cases like separating reads and writes for multiple services, providing real-time updates to a web-page and others - and all using Kafka!

Short ML Course on GANs 🤟
Machine Learning is everywhere. I am sharing a short course from Google, which explains how GAN - Generative adversarial networks - works. Check out the relations between a generator, discriminator and loss functions there!

#ml #course

In search of a robust model
The only constant in IT we have - is that everything changes. Technologies come and go, architecture approaches transform, the way we work change as well. Humans need some basis to rely on; Uwe suggests that a robust change model can save our psyche. He says, let's embrace the constant change and have a process of adopting new things. The full thought journey below.

#vision

Folks! I am taking a short vacation next week, so see you in your inboxes 18th of December! Have a rest yourself or go through some previous issues.

Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert and Roman for already supporting the newsletter.

Architecture Weekly #47 - Highlights

Video
Hey, folks! I got a bit sick, so the video is coming earlier next week. Sorry for that.

Highlights

This week highlights are all about databases. First of all, Bolt and PingCAP held a technical meetup covering the TitaniumDB architecture and the necessity of migration from MySQL to TitaniumDB for Bolt. Next I include an article about DB sharding following it up with a practical example of balancing the shards and moving data by Shopify. Find it all below!

TitaniumDB and Bolt 👷‍♂️

On 6th of December there was a first technical meetup of Bolt and PingCAP combined. There we discussed how Bolt used to have MySQL as a primary database, what were the issues, it's journey to Vitesse and finally to TitaniumDB. A fascinating journey I would say!

#database #titaniumdb

Database Sharding Explained 🍼

Database sharding is a technique used to horizontally partition a database into smaller, faster, more easily managed parts called shards. There are several approaches to sharding, including range-based, hash-based, and directory-based sharding. Sharding is a complex process that requires careful planning and is not suitable for all databases or workloads. However, for large, high-performance databases that need to scale, sharding can be an effective way to improve performance, scalability, and availability. Find a guide on the sharding strategies below.

#database #sharding

Shard Balancing with confidence 👷‍♂️

Continuing the topic of sharding, I want to share an astonishing article by Shopify. First of all, I like the approach of picking up a sharding strategy: they create a hypothesis and then test it with real data in the simulated environment, thus being able to pick a best strategy. Secondly, the article contains the real example of moving a high traffic shop from one shard to another keeping the integrity in place and avoiding downtime. I am impressed!

Follow-Up section is down below!

Architecture Weekly #47 - Follow-Up

Diagrams as Code 🍼
Creating the diagrams with code is almost mainstream now with Mermaid, PlantUML and their integration into GitHub, Confluence and other tools. Today we share a new project which allows doing the same - with Python Code now.

#diagrams #documentation

A curated list for Awesome K8s Security 👷‍♂️
Kubernetes provides many capabilities, but with great power comes great responsibility. In the K8s case - it's the cluster security. Grab a very nice list of blogs, videos, books and CVEs on Kubertentes Security.

#security #k8s #kubernetes

Abstraction is Expensive 🤟
We build complex systems via abstractions of different levels: be it the network, files, virtual machines etc. If the abstraction is misaligned you end up with a cost: let's say you pick a wrong schema for a DB and instead of a single look up spend time doing scans. In the blog post Nima Badizadegan from Google shares his thoughts on why abstractions are expensive and gives some great examples even with TCP or memory allocators.

#abstraction #vision

Essential features of an Event Store 🍼
Whatever you use for event store: a custom built piece, a library or a complete solution, you should have some expectations from it like being append-only, supporting event streams, event stream versioning, and others. Find the full list in the article down below.

#eventsourcing #eventstore

Designing an Engineering Strategy. Part 1 🍼
Part of the Enterprise Architect, Director of Engineering and other senior roles is designing and executing an engineering strategy, which is way to solve a big business problem from the engineering perspective. Sounds interesting? Start with a first post from a series of designing an engineering strategy.

#leadership #strategy #ea

What is a service mesh? 🍼
Mesh term is used so frequently: data mesh, network mesh, service mesh. Folks from Koyeb blog decided to explain what is a service mesh, what it typically consist of, and how it can help to solve the problems for the developers of microservices.

#servicemesh #microservice

Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman and Andrei for already supporting the newsletter.

Architecture Weekly #48 - Highlights

Video

Accelerating Code Delivery by 97% 👷‍♂️
LinkedIn made a journey from a single repo for frontend to multiple repos reaching 70 repositories for a single product. It affected the lead time which became unacceptable. Read how LinkedIn managed to migrate back to a monorepo per product approach reducing the lead time by 97% to a matter of hours instead of days!

Mobile App Development Frameworks - a security guide 🍼
This week I finally published a long article on the mobile development security guide. If you choose a technology for your brand new mobile application, it will help understand what are the security concerns for each of the technology starting with native Kotlin/Swift development through React-Native and Kotlin to Hybrid HTML-based apps. In the end, you will find a comparison table with the severity of common mobile threats which will guide your technology choice.

Istio & Service Mesh explained in 15 minutes 👷‍♂️
Having multiple microservices comes with a cost: you need to implement the API exposure, interservice communication, security, resilience patterns, and more. Instead of doing it manually in each service, we can offload all of it in a sidecar - a separate unit running alongside each service. This set of sidecars is called service mesh, and Istio implements it. Find out the video by Nina on the details of services meshes, Istio, and how to configure it within your Kubernetes cluster.

Find Follow-Up below!

Architecture Weekly #48 - Follow-Up

Application Resiliency by Uncovering Hidden Issues with Chaos Testing 🍼
We already covered the overviews of Chaos Testing but repetition makes perfect. Find a new article on how Chaos Engineering helped identified an obscure issue in an online bank and what are the general principles and ideas of this valuable practice.

#resilience

The evolution of state transfer - ElectricSQL 🍼
Web and mobile clients use different techniques to send information to a server, starting from submitting an HTML form through POST request to making GraphQL queries. However in a world with multiple devices per user and a requirement of real-time collaboration between users there should be a better tool. And there is - a local-first state. What's that? Follow the article below.

#web #concept

Kubernetes for Data Science practice 🍼
A short note an how the Data Science people often need to run big workloads but lack tools to do that. One of the answer is using an open-source Kubeflow to run ML and other data processing on Kubernetes.

#kubernetes #k8s #ml #dataengineering

Difference between a 2PC and a saga 👷‍♂️
Making changes in a distributed systems can be done through a distributed locking - what is essentially 2-Phase-Commit is. If you want to avoid locks you can go with sagas - a way to have a coordinated long-live-transaction by combining several short ones. Need more details? Follow the article inside!

#distributedsystems #saga #2pc

Markdown Any Decision Record Explained 🍼
I am a fan of Architecture Decition Record - a short note which clarifes the decision to address architecturelly significant non-functional requirement. The Version 3.0 of the markdown template for such document got released recently, and the respective article was written to explain it. Follow for the details :)

#adr #documentation

Deep Dive into Architecture Decision Records 🍼
Continuing the topic of ADRs, I want to share an article which explains how to provide better context, and make the decision records more clear and consice.

#adr #documentation #bestpractices

How to give feedback to your coworkers? 🍼
The most people in the IT field regulary provide feedback during performance reviews and 360 feedback rounds. But from what I see, a major part does not know how to give good, actionable feedback. I decided to help the issue and recorded a video how to do that properly. But hey, I bet you already saw it on the Architecture Weekly channel, didn't you?

#management #video

Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, August and Andrey for already supporting the newsletter.

Hello, everyone! The New Year is approaching really soon. Trying to sum it up, I would like to say that this year was the most difficult so far with the war, relocation, new job and making this newsletter. However, it came with some achievements as well. I posted 49 issues; the audience of the channel grew from 0 to 727 people. The blog subscribers went from 300 people to more than a thousand. I also launched a video channel with more than 335 followers. I appreciate everyone who joined the Telegram and YouTube channels as well as blog followers!

I want to thank you for following it all - means to me a ton and motivates to continue the work. I would also like to wish you a safe, peaceful New Year. Let it bring new, exclusively professional challenges to your life.

If you want to help the channel you can take the following actions: 1) tell your colleagues about the channel. Let our community grow! 2) subscribe to a YouTube channel if you still haven't. 3) Subscribe to Patreon or Boosty - the videos obviously require some audio improvements which comes with a cost of a hardware or software filters. You would also get access to some videos and blog posts. Tomorrow's issue is already there btw :)

And of course, don't hesitate to leave any kind of the feedback - I am always open to it.

If you're interested in my own results of the year - follow my blog post open to everyone.

Thank you again and continue your growth!

Architecture Weekly #49
Video

Highlights
LastPass is a popular password management tool that has been plagued by security issues for years. Despite its widespread use, the latest security breach of LastPass should serve as a wake-up call for users to consider alternative options. In this article, Jeremy delves into the history of LastPass' security problems and explains why it is time for users to consider moving on to more secure password management solutions.

#security
The complete guide to protecting your APIs with OAuth2(part 1) 👷‍♂️
I doubt there are many IT specialists who never heard the term JWT, standing for JSON Web Token or OAuth. The standard helps manage access to the major part of the Internet resources, thus it's important to understand not only how to use it in the first place, but also how to do that securely. Stackoverflow blog got a long article explaining the OAuth Standard and security best practices for it.

#security #api
Observability - State of Art 22' 👷‍♂️
There is no need to sell observability to anyone in 2022: now we treat it as an essential part of our systems and a natural requirement. But what observability consists of? What types of signals and data do we want to monitor? Let's figure it out with an article by Daniel.

#observability

Follow-Up Below 👇🏽

Architecture Weekly #49 - Follow-Up

PostgreSQL + TimescaleDB: 1,000x Faster Queries and 90% Data compression for time-series data 🍼
TimescaleDB is a solution on top of PostgreSQL for time-series data. Focusing on a single problem allows for providing incredible optimisations like several orders of magnitude faster queries and an order of magnitude less storage requirement. Read an article about those advantages in the Timescale blog.

#postgresql #timeseries #db

How to learn Software Design and Architecture 🍼
Going from a software developer intern to a Solution Architect took me 10 years. I would definitely appreciate any kind of a roadmap to help make this journey a bit shorter time-wise. Khalil Stemmler wrote a detailed map on mastering software design. I believe it lacks 2 or 3 stages, but it is a very good attempt to map the knowledge required. Find it below.

#systemdesign #learning

Why not DNS? 👷‍♂️
Searching for a proper pod in a K8s cluster requires name resolution. And hopefully, DNS would solve that problem for us. However, microservice instances come and go significantly faster than website A-records. That's why we need KubeProxy. A detailed explanation inside.

Microservices integration Using Contract-Driven Development
Integration tests are good, but they can catch a problem only when the integration is done. In order to have a shorter feedback loop, we can test the API of each service in isolation against the commonly shared contract, which is API description in a form of lets OpenAPI. Checkout the strategies and a suggested tool to do so below.

#microservices #contracttesting #api

Kafka Stream Transformation Patterns and Message Reprocessing 👷‍♂️
Several months ago my team decided that we need a refactoring for a fiscalization solution: it was based on crons and virtualized queues in a DB and was a source of several painful problems linked to reprocessing the invalid records. We created a design using Kafka, and now I found a cool article that addresses pretty much the same problem: what do you do if you don't want to stop the processing of an event stream in case of a message with incomplete or wrong info? Follow it for details.

#kafka

Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, Andrey and Lidia for already supporting the newsletter.

Architecture Weekly #50 🔥

Video

Highlights
CircleCI breach 🍼
The CI/CD pipelines literally have 2 jobs: build and stay secure. Obviously, CircleCI failed to fill in the second part with an announcement that the secrets stored in all the pipelines can be compromised. So if you're a client - first of all do so, and if it's not yet done - implement a secret rotation policy. The news came in on the 4th of December with an email to the clients. Gergely Orosz has made an overview of the incident and CircleCI's communication of it, grab it here.
#cicd #security #incident

GitHub solution for converting the columns to ActiveRecord's encrypted 👷‍♂️
As we just learned storing data is better encrypted. But we don't always implement this strategy right from the get-go. And once we want to do that, a challenge arises, how to convert the plain text fields to encrypted ones without downtime. And if some fields were encrypted how to upgrade them to use better protection? Follow the GitHub blog on how they did it with their data. Multiple encryption keys, decypher rounds and error handling inside.

#security #encryption

How CloudFlare run their Kafka clusters with more than a trillion messages a day 👷‍♂️
Kafka has been used at Cloudflare for 8 years already and processed over 1 trillion messages in a general-purpose cluster. They built several internal tools, sunset part of them for better versions, and decided to share their journey in a detailed post. They show how their Connectors framework allows declaring data transformation with a single configuration file connecting multiple systems without repeating the same code over and over again. Nice one here.

Follow-up section is down below 👇🏽