Architecture Weekly pinned Β«If you have an article you would like to be included in the newsletter, you can drop a message with a link to architectureweeklynewsletter@gmail.com .Β»
Architecture Weekly #47 - Highlights
Video
Hey, folks! I got a bit sick, so the video is coming earlier next week. Sorry for that.
Highlights
This week highlights are all about databases. First of all, Bolt and PingCAP held a technical meetup covering the TitaniumDB architecture and the necessity of migration from MySQL to TitaniumDB for Bolt. Next I include an article about DB sharding following it up with a practical example of balancing the shards and moving data by Shopify. Find it all below!
TitaniumDB and Bolt π·ββοΈ
On 6th of December there was a first technical meetup of Bolt and PingCAP combined. There we discussed how Bolt used to have MySQL as a primary database, what were the issues, it's journey to Vitesse and finally to TitaniumDB. A fascinating journey I would say!
#database #titaniumdb
Database Sharding Explained πΌ
Database sharding is a technique used to horizontally partition a database into smaller, faster, more easily managed parts called shards. There are several approaches to sharding, including range-based, hash-based, and directory-based sharding. Sharding is a complex process that requires careful planning and is not suitable for all databases or workloads. However, for large, high-performance databases that need to scale, sharding can be an effective way to improve performance, scalability, and availability. Find a guide on the sharding strategies below.
#database #sharding
Shard Balancing with confidence π·ββοΈ
Continuing the topic of sharding, I want to share an astonishing article by Shopify. First of all, I like the approach of picking up a sharding strategy: they create a hypothesis and then test it with real data in the simulated environment, thus being able to pick a best strategy. Secondly, the article contains the real example of moving a high traffic shop from one shard to another keeping the integrity in place and avoiding downtime. I am impressed!
Follow-Up section is down below!
Video
Hey, folks! I got a bit sick, so the video is coming earlier next week. Sorry for that.
Highlights
This week highlights are all about databases. First of all, Bolt and PingCAP held a technical meetup covering the TitaniumDB architecture and the necessity of migration from MySQL to TitaniumDB for Bolt. Next I include an article about DB sharding following it up with a practical example of balancing the shards and moving data by Shopify. Find it all below!
TitaniumDB and Bolt π·ββοΈ
On 6th of December there was a first technical meetup of Bolt and PingCAP combined. There we discussed how Bolt used to have MySQL as a primary database, what were the issues, it's journey to Vitesse and finally to TitaniumDB. A fascinating journey I would say!
#database #titaniumdb
Database Sharding Explained πΌ
Database sharding is a technique used to horizontally partition a database into smaller, faster, more easily managed parts called shards. There are several approaches to sharding, including range-based, hash-based, and directory-based sharding. Sharding is a complex process that requires careful planning and is not suitable for all databases or workloads. However, for large, high-performance databases that need to scale, sharding can be an effective way to improve performance, scalability, and availability. Find a guide on the sharding strategies below.
#database #sharding
Shard Balancing with confidence π·ββοΈ
Continuing the topic of sharding, I want to share an astonishing article by Shopify. First of all, I like the approach of picking up a sharding strategy: they create a hypothesis and then test it with real data in the simulated environment, thus being able to pick a best strategy. Secondly, the article contains the real example of moving a high traffic shop from one shard to another keeping the integrity in place and avoiding downtime. I am impressed!
Follow-Up section is down below!
YouTube
Bolt/PingCAP meetup Tallinn 6.12
βββββββββ About PingCAP βββββββββ
PingCAP is dedicated to building #TiDB, an #OpenSource, #CloudNative, distributed, MySQL compatible #database for elastic scale and real-time analytics.
βββββββββ Resource Links βββββββββ
π«Quick start with TiDB: httβ¦
PingCAP is dedicated to building #TiDB, an #OpenSource, #CloudNative, distributed, MySQL compatible #database for elastic scale and real-time analytics.
βββββββββ Resource Links βββββββββ
π«Quick start with TiDB: httβ¦
π3
Architecture Weekly #47 - Follow-Up
Diagrams as Code πΌ
Creating the diagrams with code is almost mainstream now with Mermaid, PlantUML and their integration into GitHub, Confluence and other tools. Today we share a new project which allows doing the same - with Python Code now.
#diagrams #documentation
A curated list for Awesome K8s Security π·ββοΈ
Kubernetes provides many capabilities, but with great power comes great responsibility. In the K8s case - it's the cluster security. Grab a very nice list of blogs, videos, books and CVEs on Kubertentes Security.
#security #k8s #kubernetes
Abstraction is Expensive π€
We build complex systems via abstractions of different levels: be it the network, files, virtual machines etc. If the abstraction is misaligned you end up with a cost: let's say you pick a wrong schema for a DB and instead of a single look up spend time doing scans. In the blog post Nima Badizadegan from Google shares his thoughts on why abstractions are expensive and gives some great examples even with TCP or memory allocators.
#abstraction #vision
Essential features of an Event Store πΌ
Whatever you use for event store: a custom built piece, a library or a complete solution, you should have some expectations from it like being append-only, supporting event streams, event stream versioning, and others. Find the full list in the article down below.
#eventsourcing #eventstore
Designing an Engineering Strategy. Part 1 πΌ
Part of the Enterprise Architect, Director of Engineering and other senior roles is designing and executing an engineering strategy, which is way to solve a big business problem from the engineering perspective. Sounds interesting? Start with a first post from a series of designing an engineering strategy.
#leadership #strategy #ea
What is a service mesh? πΌ
Mesh term is used so frequently: data mesh, network mesh, service mesh. Folks from Koyeb blog decided to explain what is a service mesh, what it typically consist of, and how it can help to solve the problems for the developers of microservices.
#servicemesh #microservice
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman and Andrei for already supporting the newsletter.
Diagrams as Code πΌ
Creating the diagrams with code is almost mainstream now with Mermaid, PlantUML and their integration into GitHub, Confluence and other tools. Today we share a new project which allows doing the same - with Python Code now.
#diagrams #documentation
A curated list for Awesome K8s Security π·ββοΈ
Kubernetes provides many capabilities, but with great power comes great responsibility. In the K8s case - it's the cluster security. Grab a very nice list of blogs, videos, books and CVEs on Kubertentes Security.
#security #k8s #kubernetes
Abstraction is Expensive π€
We build complex systems via abstractions of different levels: be it the network, files, virtual machines etc. If the abstraction is misaligned you end up with a cost: let's say you pick a wrong schema for a DB and instead of a single look up spend time doing scans. In the blog post Nima Badizadegan from Google shares his thoughts on why abstractions are expensive and gives some great examples even with TCP or memory allocators.
#abstraction #vision
Essential features of an Event Store πΌ
Whatever you use for event store: a custom built piece, a library or a complete solution, you should have some expectations from it like being append-only, supporting event streams, event stream versioning, and others. Find the full list in the article down below.
#eventsourcing #eventstore
Designing an Engineering Strategy. Part 1 πΌ
Part of the Enterprise Architect, Director of Engineering and other senior roles is designing and executing an engineering strategy, which is way to solve a big business problem from the engineering perspective. Sounds interesting? Start with a first post from a series of designing an engineering strategy.
#leadership #strategy #ea
What is a service mesh? πΌ
Mesh term is used so frequently: data mesh, network mesh, service mesh. Folks from Koyeb blog decided to explain what is a service mesh, what it typically consist of, and how it can help to solve the problems for the developers of microservices.
#servicemesh #microservice
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman and Andrei for already supporting the newsletter.
Mingrammer
Diagrams Β· Diagram as Code
π4
Architecture Weekly #48 - Highlights
Video
Accelerating Code Delivery by 97% π·ββοΈ
LinkedIn made a journey from a single repo for frontend to multiple repos reaching 70 repositories for a single product. It affected the lead time which became unacceptable. Read how LinkedIn managed to migrate back to a monorepo per product approach reducing the lead time by 97% to a matter of hours instead of days!
Mobile App Development Frameworks - a security guide πΌ
This week I finally published a long article on the mobile development security guide. If you choose a technology for your brand new mobile application, it will help understand what are the security concerns for each of the technology starting with native Kotlin/Swift development through React-Native and Kotlin to Hybrid HTML-based apps. In the end, you will find a comparison table with the severity of common mobile threats which will guide your technology choice.
Istio & Service Mesh explained in 15 minutes π·ββοΈ
Having multiple microservices comes with a cost: you need to implement the API exposure, interservice communication, security, resilience patterns, and more. Instead of doing it manually in each service, we can offload all of it in a sidecar - a separate unit running alongside each service. This set of sidecars is called service mesh, and Istio implements it. Find out the video by Nina on the details of services meshes, Istio, and how to configure it within your Kubernetes cluster.
Find Follow-Up below!
Video
Accelerating Code Delivery by 97% π·ββοΈ
LinkedIn made a journey from a single repo for frontend to multiple repos reaching 70 repositories for a single product. It affected the lead time which became unacceptable. Read how LinkedIn managed to migrate back to a monorepo per product approach reducing the lead time by 97% to a matter of hours instead of days!
Mobile App Development Frameworks - a security guide πΌ
This week I finally published a long article on the mobile development security guide. If you choose a technology for your brand new mobile application, it will help understand what are the security concerns for each of the technology starting with native Kotlin/Swift development through React-Native and Kotlin to Hybrid HTML-based apps. In the end, you will find a comparison table with the severity of common mobile threats which will guide your technology choice.
Istio & Service Mesh explained in 15 minutes π·ββοΈ
Having multiple microservices comes with a cost: you need to implement the API exposure, interservice communication, security, resilience patterns, and more. Instead of doing it manually in each service, we can offload all of it in a sidecar - a separate unit running alongside each service. This set of sidecars is called service mesh, and Istio implements it. Find out the video by Nina on the details of services meshes, Istio, and how to configure it within your Kubernetes cluster.
Find Follow-Up below!
YouTube
How to reduce TTM by 97%? What are security implications of mobile tech? What is a service mesh?
Architecture Weekly #48
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-48/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Start
0:16β¦
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-48/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Start
0:16β¦
π3
Architecture Weekly #48 - Follow-Up
Application Resiliency by Uncovering Hidden Issues with Chaos Testing πΌ
We already covered the overviews of Chaos Testing but repetition makes perfect. Find a new article on how Chaos Engineering helped identified an obscure issue in an online bank and what are the general principles and ideas of this valuable practice.
#resilience
The evolution of state transfer - ElectricSQL πΌ
Web and mobile clients use different techniques to send information to a server, starting from submitting an HTML form through POST request to making GraphQL queries. However in a world with multiple devices per user and a requirement of real-time collaboration between users there should be a better tool. And there is - a local-first state. What's that? Follow the article below.
#web #concept
Kubernetes for Data Science practice πΌ
A short note an how the Data Science people often need to run big workloads but lack tools to do that. One of the answer is using an open-source Kubeflow to run ML and other data processing on Kubernetes.
#kubernetes #k8s #ml #dataengineering
Difference between a 2PC and a saga π·ββοΈ
Making changes in a distributed systems can be done through a distributed locking - what is essentially 2-Phase-Commit is. If you want to avoid locks you can go with sagas - a way to have a coordinated long-live-transaction by combining several short ones. Need more details? Follow the article inside!
#distributedsystems #saga #2pc
Markdown Any Decision Record Explained πΌ
I am a fan of Architecture Decition Record - a short note which clarifes the decision to address architecturelly significant non-functional requirement. The Version 3.0 of the markdown template for such document got released recently, and the respective article was written to explain it. Follow for the details :)
#adr #documentation
Deep Dive into Architecture Decision Records πΌ
Continuing the topic of ADRs, I want to share an article which explains how to provide better context, and make the decision records more clear and consice.
#adr #documentation #bestpractices
How to give feedback to your coworkers? πΌ
The most people in the IT field regulary provide feedback during performance reviews and 360 feedback rounds. But from what I see, a major part does not know how to give good, actionable feedback. I decided to help the issue and recorded a video how to do that properly. But hey, I bet you already saw it on the Architecture Weekly channel, didn't you?
#management #video
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, August and Andrey for already supporting the newsletter.
Application Resiliency by Uncovering Hidden Issues with Chaos Testing πΌ
We already covered the overviews of Chaos Testing but repetition makes perfect. Find a new article on how Chaos Engineering helped identified an obscure issue in an online bank and what are the general principles and ideas of this valuable practice.
#resilience
The evolution of state transfer - ElectricSQL πΌ
Web and mobile clients use different techniques to send information to a server, starting from submitting an HTML form through POST request to making GraphQL queries. However in a world with multiple devices per user and a requirement of real-time collaboration between users there should be a better tool. And there is - a local-first state. What's that? Follow the article below.
#web #concept
Kubernetes for Data Science practice πΌ
A short note an how the Data Science people often need to run big workloads but lack tools to do that. One of the answer is using an open-source Kubeflow to run ML and other data processing on Kubernetes.
#kubernetes #k8s #ml #dataengineering
Difference between a 2PC and a saga π·ββοΈ
Making changes in a distributed systems can be done through a distributed locking - what is essentially 2-Phase-Commit is. If you want to avoid locks you can go with sagas - a way to have a coordinated long-live-transaction by combining several short ones. Need more details? Follow the article inside!
#distributedsystems #saga #2pc
Markdown Any Decision Record Explained πΌ
I am a fan of Architecture Decition Record - a short note which clarifes the decision to address architecturelly significant non-functional requirement. The Version 3.0 of the markdown template for such document got released recently, and the respective article was written to explain it. Follow for the details :)
#adr #documentation
Deep Dive into Architecture Decision Records πΌ
Continuing the topic of ADRs, I want to share an article which explains how to provide better context, and make the decision records more clear and consice.
#adr #documentation #bestpractices
How to give feedback to your coworkers? πΌ
The most people in the IT field regulary provide feedback during performance reviews and 360 feedback rounds. But from what I see, a major part does not know how to give good, actionable feedback. I decided to help the issue and recorded a video how to do that properly. But hey, I bet you already saw it on the Architecture Weekly channel, didn't you?
#management #video
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, August and Andrey for already supporting the newsletter.
InfoQ
How We Improved Applicationβs Resiliency by Uncovering Our Hidden Issues Using Chaos Testing
This article lists the chaos testing principles which are outlined by Netflix. The readers should be able to understand the advantages and disadvantages that chaos testing offers. This will help them to decide whether they want to perform it or not. The articleβ¦
π₯3
Hello, everyone! The New Year is approaching really soon. Trying to sum it up, I would like to say that this year was the most difficult so far with the war, relocation, new job and making this newsletter. However, it came with some achievements as well. I posted 49 issues; the audience of the channel grew from 0 to 727 people. The blog subscribers went from 300 people to more than a thousand. I also launched a video channel with more than 335 followers. I appreciate everyone who joined the Telegram and YouTube channels as well as blog followers!
I want to thank you for following it all - means to me a ton and motivates to continue the work. I would also like to wish you a safe, peaceful New Year. Let it bring new, exclusively professional challenges to your life.
If you want to help the channel you can take the following actions: 1) tell your colleagues about the channel. Let our community grow! 2) subscribe to a YouTube channel if you still haven't. 3) Subscribe to Patreon or Boosty - the videos obviously require some audio improvements which comes with a cost of a hardware or software filters. You would also get access to some videos and blog posts. Tomorrow's issue is already there btw :)
And of course, don't hesitate to leave any kind of the feedback - I am always open to it.
If you're interested in my own results of the year - follow my blog post open to everyone.
Thank you again and continue your growth!
I want to thank you for following it all - means to me a ton and motivates to continue the work. I would also like to wish you a safe, peaceful New Year. Let it bring new, exclusively professional challenges to your life.
If you want to help the channel you can take the following actions: 1) tell your colleagues about the channel. Let our community grow! 2) subscribe to a YouTube channel if you still haven't. 3) Subscribe to Patreon or Boosty - the videos obviously require some audio improvements which comes with a cost of a hardware or software filters. You would also get access to some videos and blog posts. Tomorrow's issue is already there btw :)
And of course, don't hesitate to leave any kind of the feedback - I am always open to it.
If you're interested in my own results of the year - follow my blog post open to everyone.
Thank you again and continue your growth!
Patreon
Get more from Vladimir Ivanov on Patreon
ΡΠΎΠ·Π΄Π°ΡΡ Architecture Weekly Newsletter
π11
Architecture Weekly #49
Video
Highlights
LastPass is a popular password management tool that has been plagued by security issues for years. Despite its widespread use, the latest security breach of LastPass should serve as a wake-up call for users to consider alternative options. In this article, Jeremy delves into the history of LastPass' security problems and explains why it is time for users to consider moving on to more secure password management solutions.
#security
The complete guide to protecting your APIs with OAuth2(part 1) π·ββοΈ
I doubt there are many IT specialists who never heard the term JWT, standing for JSON Web Token or OAuth. The standard helps manage access to the major part of the Internet resources, thus it's important to understand not only how to use it in the first place, but also how to do that securely. Stackoverflow blog got a long article explaining the OAuth Standard and security best practices for it.
#security #api
Observability - State of Art 22' π·ββοΈ
There is no need to sell observability to anyone in 2022: now we treat it as an essential part of our systems and a natural requirement. But what observability consists of? What types of signals and data do we want to monitor? Let's figure it out with an article by Daniel.
#observability
Follow-Up Below ππ½
Video
Highlights
LastPass is a popular password management tool that has been plagued by security issues for years. Despite its widespread use, the latest security breach of LastPass should serve as a wake-up call for users to consider alternative options. In this article, Jeremy delves into the history of LastPass' security problems and explains why it is time for users to consider moving on to more secure password management solutions.
#security
The complete guide to protecting your APIs with OAuth2(part 1) π·ββοΈ
I doubt there are many IT specialists who never heard the term JWT, standing for JSON Web Token or OAuth. The standard helps manage access to the major part of the Internet resources, thus it's important to understand not only how to use it in the first place, but also how to do that securely. Stackoverflow blog got a long article explaining the OAuth Standard and security best practices for it.
#security #api
Observability - State of Art 22' π·ββοΈ
There is no need to sell observability to anyone in 2022: now we treat it as an essential part of our systems and a natural requirement. But what observability consists of? What types of signals and data do we want to monitor? Let's figure it out with an article by Daniel.
#observability
Follow-Up Below ππ½
Architecture Weekly #49 - Follow-Up
PostgreSQL + TimescaleDB: 1,000x Faster Queries and 90% Data compression for time-series data πΌ
TimescaleDB is a solution on top of PostgreSQL for time-series data. Focusing on a single problem allows for providing incredible optimisations like several orders of magnitude faster queries and an order of magnitude less storage requirement. Read an article about those advantages in the Timescale blog.
#postgresql #timeseries #db
How to learn Software Design and Architecture πΌ
Going from a software developer intern to a Solution Architect took me 10 years. I would definitely appreciate any kind of a roadmap to help make this journey a bit shorter time-wise. Khalil Stemmler wrote a detailed map on mastering software design. I believe it lacks 2 or 3 stages, but it is a very good attempt to map the knowledge required. Find it below.
#systemdesign #learning
Why not DNS? π·ββοΈ
Searching for a proper pod in a K8s cluster requires name resolution. And hopefully, DNS would solve that problem for us. However, microservice instances come and go significantly faster than website A-records. That's why we need KubeProxy. A detailed explanation inside.
Microservices integration Using Contract-Driven Development
Integration tests are good, but they can catch a problem only when the integration is done. In order to have a shorter feedback loop, we can test the API of each service in isolation against the commonly shared contract, which is API description in a form of lets OpenAPI. Checkout the strategies and a suggested tool to do so below.
#microservices #contracttesting #api
Kafka Stream Transformation Patterns and Message Reprocessing π·ββοΈ
Several months ago my team decided that we need a refactoring for a fiscalization solution: it was based on crons and virtualized queues in a DB and was a source of several painful problems linked to reprocessing the invalid records. We created a design using Kafka, and now I found a cool article that addresses pretty much the same problem: what do you do if you don't want to stop the processing of an event stream in case of a message with incomplete or wrong info? Follow it for details.
#kafka
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, Andrey and Lidia for already supporting the newsletter.
PostgreSQL + TimescaleDB: 1,000x Faster Queries and 90% Data compression for time-series data πΌ
TimescaleDB is a solution on top of PostgreSQL for time-series data. Focusing on a single problem allows for providing incredible optimisations like several orders of magnitude faster queries and an order of magnitude less storage requirement. Read an article about those advantages in the Timescale blog.
#postgresql #timeseries #db
How to learn Software Design and Architecture πΌ
Going from a software developer intern to a Solution Architect took me 10 years. I would definitely appreciate any kind of a roadmap to help make this journey a bit shorter time-wise. Khalil Stemmler wrote a detailed map on mastering software design. I believe it lacks 2 or 3 stages, but it is a very good attempt to map the knowledge required. Find it below.
#systemdesign #learning
Why not DNS? π·ββοΈ
Searching for a proper pod in a K8s cluster requires name resolution. And hopefully, DNS would solve that problem for us. However, microservice instances come and go significantly faster than website A-records. That's why we need KubeProxy. A detailed explanation inside.
Microservices integration Using Contract-Driven Development
Integration tests are good, but they can catch a problem only when the integration is done. In order to have a shorter feedback loop, we can test the API of each service in isolation against the commonly shared contract, which is API description in a form of lets OpenAPI. Checkout the strategies and a suggested tool to do so below.
#microservices #contracttesting #api
Kafka Stream Transformation Patterns and Message Reprocessing π·ββοΈ
Several months ago my team decided that we need a refactoring for a fiscalization solution: it was based on crons and virtualized queues in a DB and was a source of several painful problems linked to reprocessing the invalid records. We created a design using Kafka, and now I found a cool article that addresses pretty much the same problem: what do you do if you don't want to stop the processing of an event stream in case of a message with incomplete or wrong info? Follow it for details.
#kafka
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel, Robert, Roman, Andrey and Lidia for already supporting the newsletter.
TigerData Blog
PostgreSQL + TimescaleDB: 1,000x Faster Queries, 90 % Data Compression, and Much More | TigerData
TimescaleDB expands PostgreSQL query performance by 1000x, reduces storage utilization by 90%, and provides time-saving features for time-series and analytical applicationsβwhile still being 100% Postgres.
π4
Architecture Weekly #50 π₯
Video
Highlights
CircleCI breach πΌ
The CI/CD pipelines literally have 2 jobs: build and stay secure. Obviously, CircleCI failed to fill in the second part with an announcement that the secrets stored in all the pipelines can be compromised. So if you're a client - first of all do so, and if it's not yet done - implement a secret rotation policy. The news came in on the 4th of December with an email to the clients. Gergely Orosz has made an overview of the incident and CircleCI's communication of it, grab it here.
#cicd #security #incident
GitHub solution for converting the columns to ActiveRecord's encrypted π·ββοΈ
As we just learned storing data is better encrypted. But we don't always implement this strategy right from the get-go. And once we want to do that, a challenge arises, how to convert the plain text fields to encrypted ones without downtime. And if some fields were encrypted how to upgrade them to use better protection? Follow the GitHub blog on how they did it with their data. Multiple encryption keys, decypher rounds and error handling inside.
#security #encryption
How CloudFlare run their Kafka clusters with more than a trillion messages a day π·ββοΈ
Kafka has been used at Cloudflare for 8 years already and processed over 1 trillion messages in a general-purpose cluster. They built several internal tools, sunset part of them for better versions, and decided to share their journey in a detailed post. They show how their Connectors framework allows declaring data transformation with a single configuration file connecting multiple systems without repeating the same code over and over again. Nice one here.
Follow-up section is down below ππ½
Video
Highlights
CircleCI breach πΌ
The CI/CD pipelines literally have 2 jobs: build and stay secure. Obviously, CircleCI failed to fill in the second part with an announcement that the secrets stored in all the pipelines can be compromised. So if you're a client - first of all do so, and if it's not yet done - implement a secret rotation policy. The news came in on the 4th of December with an email to the clients. Gergely Orosz has made an overview of the incident and CircleCI's communication of it, grab it here.
#cicd #security #incident
GitHub solution for converting the columns to ActiveRecord's encrypted π·ββοΈ
As we just learned storing data is better encrypted. But we don't always implement this strategy right from the get-go. And once we want to do that, a challenge arises, how to convert the plain text fields to encrypted ones without downtime. And if some fields were encrypted how to upgrade them to use better protection? Follow the GitHub blog on how they did it with their data. Multiple encryption keys, decypher rounds and error handling inside.
#security #encryption
How CloudFlare run their Kafka clusters with more than a trillion messages a day π·ββοΈ
Kafka has been used at Cloudflare for 8 years already and processed over 1 trillion messages in a general-purpose cluster. They built several internal tools, sunset part of them for better versions, and decided to share their journey in a detailed post. They show how their Connectors framework allows declaring data transformation with a single configuration file connecting multiple systems without repeating the same code over and over again. Nice one here.
Follow-up section is down below ππ½
YouTube
CircleCI breach, 1 trillion messages in Kafka by CloudFlare and column db encryption at GitHub
Architecture Weekly N50 π₯
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-50/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Startβ¦
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-50/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
0:00 Startβ¦
π4
Architecture Weekly #50 Follow-Up
Highlights are up above ππ½
The lost art of Software Design πΌ
Do you do a big upfront design? Or do you do the architecture design phase ever? Simon Brown's talk on the lost art of Software Design explains why there is a good middle-ground between the two and spices it up with multiple software design kata examples.
#architecture #systemdesign
You should be reading academic papers πΌ
For a long time, I was only reading blog posts. But once I started doing some system design I quickly realized that blogs are not enough: you need to understand the underlying problems. That's what academic papers are good for. StackOverflow Blog includes a short article on further motivation to do so.
#reading
Time, clocks, and event ordering in distributed systems π€
As long as we are talking about the papers, I want to remind you of a foundation paper on time, logical and physical clocks, and events ordering in distributed systems by Lesly Lamport. Have some fun with mathematical proofs!
#distributedsystem #paper
Leveraging CDC for real-time Inventory Data Processing π·ββοΈ
Proper inventory allows for having the appropriate amount of goods to sell. This is crucial for any e-shop and of course for grocery delivery. One option to know about the inventory updates is to make the notifications by the code which modifies the related tables. But if there are many code paths it can be a problem. DoorDash went with a different solution: having a Change-Data-Capture enables the near-real-time update. Check out the details inside!
#kafka #cdc
Enterprise Workloads at Scale with a Next-Gen IaaC Platform πΌ
Intuit is a U.S.-based company doing tax reporting automation. They are also a partner of AWS. Intuit was an early adopter of many cloud technologies, and CloudFormation was one of them. But running it at scale brought its own challenges, like a long feedback loop and the difficulty of finding and fixing the error. In this post, they tell the story of how they contributed to several open-source and AWS Cloud Development Kit as well.
Configuring the Vacuum process in the PostgreSQL π·ββοΈ
PostgreSQL leverages MVCC - multi-version concurrency control, which is a mechanism for ensuring consistency. Basically, a new version of a row does not overwrite the old one, but gets appended, but with a higher transaction id. With time the old records add up to occupied space, and VACUUM is the operation to remove them. Read here for more about it and how to configure it.
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, and Roman for already supporting the newsletter. Join them as well!
Highlights are up above ππ½
The lost art of Software Design πΌ
Do you do a big upfront design? Or do you do the architecture design phase ever? Simon Brown's talk on the lost art of Software Design explains why there is a good middle-ground between the two and spices it up with multiple software design kata examples.
#architecture #systemdesign
You should be reading academic papers πΌ
For a long time, I was only reading blog posts. But once I started doing some system design I quickly realized that blogs are not enough: you need to understand the underlying problems. That's what academic papers are good for. StackOverflow Blog includes a short article on further motivation to do so.
#reading
Time, clocks, and event ordering in distributed systems π€
As long as we are talking about the papers, I want to remind you of a foundation paper on time, logical and physical clocks, and events ordering in distributed systems by Lesly Lamport. Have some fun with mathematical proofs!
#distributedsystem #paper
Leveraging CDC for real-time Inventory Data Processing π·ββοΈ
Proper inventory allows for having the appropriate amount of goods to sell. This is crucial for any e-shop and of course for grocery delivery. One option to know about the inventory updates is to make the notifications by the code which modifies the related tables. But if there are many code paths it can be a problem. DoorDash went with a different solution: having a Change-Data-Capture enables the near-real-time update. Check out the details inside!
#kafka #cdc
Enterprise Workloads at Scale with a Next-Gen IaaC Platform πΌ
Intuit is a U.S.-based company doing tax reporting automation. They are also a partner of AWS. Intuit was an early adopter of many cloud technologies, and CloudFormation was one of them. But running it at scale brought its own challenges, like a long feedback loop and the difficulty of finding and fixing the error. In this post, they tell the story of how they contributed to several open-source and AWS Cloud Development Kit as well.
Configuring the Vacuum process in the PostgreSQL π·ββοΈ
PostgreSQL leverages MVCC - multi-version concurrency control, which is a mechanism for ensuring consistency. Basically, a new version of a row does not overwrite the old one, but gets appended, but with a higher transaction id. With time the old records add up to occupied space, and VACUUM is the operation to remove them. Read here for more about it and how to configure it.
Like the newsletter? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, and Roman for already supporting the newsletter. Join them as well!
YouTube
The Lost Art of Software Design β’ Simon Brown β’ YOW! 2019
This presentation was recorded at YOW! 2019. #GOTOcon #YOW
https://yowcon.com
Simon Brown - Author of "Software Architecture for Developers" & Creator of the C4 Software @simonbrown4821
ABSTRACT
"Big design up front is dumb. Doing no design up front isβ¦
https://yowcon.com
Simon Brown - Author of "Software Architecture for Developers" & Creator of the C4 Software @simonbrown4821
ABSTRACT
"Big design up front is dumb. Doing no design up front isβ¦
π3
Unfortunately, the sending functionality on the blog is not working now, the ghost instance can't reach the mailgun servers(DNS issue). Figuring out the exact problem.
π10π3
How do computers encode information and how databases store data on disk? New video!
https://youtu.be/haz2h7_xFDk
https://youtu.be/haz2h7_xFDk
YouTube
How databases store data on disk?
Architecture Readings #3 - Database Internals. Slotted Pages
0:00 Start
0:23 Encoding data with binary calculation system
3:19 Storing strings
4:00 Structure of a database file
4:56 Fixed-size page layout
6:13 Variable-size pages: Slotted pages
8:00 Cellβ¦
0:00 Start
0:23 Encoding data with binary calculation system
3:19 Storing strings
4:00 Structure of a database file
4:56 Fixed-size page layout
6:13 Variable-size pages: Slotted pages
8:00 Cellβ¦
π₯1
Folks! I want to make a session on the summing up the architecture trends of 2022. Wanna invite Gregor Hohpe for that. Would you mind retweeting my post about it?
https://twitter.com/vvsevolodovich/status/1613117466217189376?s=20&t=GsVmFKvicXh4mGT5ksObpQ
https://twitter.com/vvsevolodovich/status/1613117466217189376?s=20&t=GsVmFKvicXh4mGT5ksObpQ
π3
Architecture Weekly Issue #51.
Video
Highlights
Should architects code? πΌ
The common problem for architects is closing themselves up in an ivory tower - getting far from the actual code, making abstract decisions and imposing them on a team. This is a recognized antipattern, but the discussion is about the remedy. Gregor Hohpe wrote an article explaining how lines of code written by an architect do not bring much value; but the insights in the process do. So instead of coding you might be doing debugging instead. Follow Gregor on the new post in the Architect Elevator.
6 steps to autoscaling on Kubernetes π·ββοΈ
Everybody wants a performant, yet cost-efficient solution with any container orchestrator. Although Kubernetes can definitely give you that with autoscaling, just enabling Horizontal Pod Autoscaler won't cut it. You need to better understand your workload, performance baseline and some nuances about autoscaling like rightsizing the pods and performance baselines - like how many users a pod can handle before degradation occurs. This and actually many more in the Perficient Blog - a fascinating read indeed.
The Art of Writing Amazing REST APIs πΌ
Every second API on the Internet is a REST API. But many of them are poorly designed: they are inconsistent, they have bad errors and they hard to understand. Joy Ebertz wrote a good, long blog post what to keep in mind when you design a REST API. I think she also could have included some advice on the API versioning, but it can done as a separate article indeed.
Follow-Up is down below!
Video
Highlights
Should architects code? πΌ
The common problem for architects is closing themselves up in an ivory tower - getting far from the actual code, making abstract decisions and imposing them on a team. This is a recognized antipattern, but the discussion is about the remedy. Gregor Hohpe wrote an article explaining how lines of code written by an architect do not bring much value; but the insights in the process do. So instead of coding you might be doing debugging instead. Follow Gregor on the new post in the Architect Elevator.
6 steps to autoscaling on Kubernetes π·ββοΈ
Everybody wants a performant, yet cost-efficient solution with any container orchestrator. Although Kubernetes can definitely give you that with autoscaling, just enabling Horizontal Pod Autoscaler won't cut it. You need to better understand your workload, performance baseline and some nuances about autoscaling like rightsizing the pods and performance baselines - like how many users a pod can handle before degradation occurs. This and actually many more in the Perficient Blog - a fascinating read indeed.
The Art of Writing Amazing REST APIs πΌ
Every second API on the Internet is a REST API. But many of them are poorly designed: they are inconsistent, they have bad errors and they hard to understand. Joy Ebertz wrote a good, long blog post what to keep in mind when you design a REST API. I think she also could have included some advice on the API versioning, but it can done as a separate article indeed.
Follow-Up is down below!
π4
Architecture Weekly #51 Follow-Up
Multi-region deployment with AWS Whitepaper π·ββοΈ
How many availability 9s you need for your application? If it's 99.99%, it will be nearly impossible to satisfy within a single cloud region. This week AWS published a guide how to understand if you need a multi-region deployment, why you need to think about failover, data consistency and operations. Grab it here!
#aws #resilience #cloud
Introduction to IaaC on AWS πΌ
Continuiung AWS topic in this newsletter and the history of IaaC in previous one, I am sharing an introduction to the Iaac. This is a light article which covers the history of deployment and 4 different approaches: manual, scripted, declarative and component based.
#aws #cloud #iaac
Data Ingestion challenges πΌ
Tons of data nowadays is a norm - user requests, telemetry, IoT, business analytics and many more. The question which still is open how to process it efficiently. Grab an article on how to do it and what challenges you are going to face.
Modules, not microservices πΌ
Microservices promised us velocity, granularity, low coupling, quality and many more. But what we found were problems of distributed systems. In the short article Ted Neward reasoning that we wanted was actually modules with clear boundaries. Examples inside.
Troubleshooting Kafka for 2000 microservices π€
As an illustration of a problem with microservices, check out an article from Wix. The backbone of the product is tons of microservices communication through Kafka. And debugging a particular problem just reported from production could take long time. Then they introduced a lot of helpful tools like sidecars for request tracing, consumer lag monitoring and many more. Follow the article for details.
#kafka #microservice #observability
Real-time integration of PostgreSQL and Kafka π€
If you need to put data to several locations, like the database, index and some other storage the first idea which comes to a mind is to do a write right-away to those systems. Although being simple solution, it also bring a problem with consistency - what would do if a write to one of those system fails? Confluent blog suggests to write to the DB first, then capture the change with Kafka and connect it to the target location. Enjoy a nice article with bottled water analogy.
#kafka
Distributed State π€
Can't leave you without a paper to read. This time it's about distributed state. The paper covers the benefits of distributed state, but states that those are potential benefits and are hardly achieved due to consistency problems and performance drawbacks.
#distributedsystem
How databases store data on disk? π€
This week I publicly shared the video on how the dbs store data on disk in the form of files, pages and how to handle variable-size data. Checkout the video!
#databases #video
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
Multi-region deployment with AWS Whitepaper π·ββοΈ
How many availability 9s you need for your application? If it's 99.99%, it will be nearly impossible to satisfy within a single cloud region. This week AWS published a guide how to understand if you need a multi-region deployment, why you need to think about failover, data consistency and operations. Grab it here!
#aws #resilience #cloud
Introduction to IaaC on AWS πΌ
Continuiung AWS topic in this newsletter and the history of IaaC in previous one, I am sharing an introduction to the Iaac. This is a light article which covers the history of deployment and 4 different approaches: manual, scripted, declarative and component based.
#aws #cloud #iaac
Data Ingestion challenges πΌ
Tons of data nowadays is a norm - user requests, telemetry, IoT, business analytics and many more. The question which still is open how to process it efficiently. Grab an article on how to do it and what challenges you are going to face.
Modules, not microservices πΌ
Microservices promised us velocity, granularity, low coupling, quality and many more. But what we found were problems of distributed systems. In the short article Ted Neward reasoning that we wanted was actually modules with clear boundaries. Examples inside.
Troubleshooting Kafka for 2000 microservices π€
As an illustration of a problem with microservices, check out an article from Wix. The backbone of the product is tons of microservices communication through Kafka. And debugging a particular problem just reported from production could take long time. Then they introduced a lot of helpful tools like sidecars for request tracing, consumer lag monitoring and many more. Follow the article for details.
#kafka #microservice #observability
Real-time integration of PostgreSQL and Kafka π€
If you need to put data to several locations, like the database, index and some other storage the first idea which comes to a mind is to do a write right-away to those systems. Although being simple solution, it also bring a problem with consistency - what would do if a write to one of those system fails? Confluent blog suggests to write to the DB first, then capture the change with Kafka and connect it to the target location. Enjoy a nice article with bottled water analogy.
#kafka
Distributed State π€
Can't leave you without a paper to read. This time it's about distributed state. The paper covers the benefits of distributed state, but states that those are potential benefits and are hardly achieved due to consistency problems and performance drawbacks.
#distributedsystem
How databases store data on disk? π€
This week I publicly shared the video on how the dbs store data on disk in the form of files, pages and how to handle variable-size data. Checkout the video!
#databases #video
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
π₯5π1
Architecture Weekly #52 - Highlights
Video
Highlights
Architect's Intro to Data Science and Machine Learning πΌ
I personally kinda afraid of Data Science and ML, because I think it requires a lot of math I am not really good at. However, you will hardly build a complex enough system without either of those. So you'd better understand the terms and underlying principles at least at a high level. Redhat wrote a post in their "Enable Architect" blog with what you as an architect should know about the Data Science and Machine Learning. The list includes courses and learning materials, data manipulation tools, programming languages, data pipelines, types of machine learning and more.
#datascience #ml
Relational Databases Explained π€
We use RDBMS almost every day in our day to day jobs, but do we know enough? Do we know how do indexes work? Or how do dbs store data on the disk? What are the transactions and levels of isolations? Architecture Notes got you back with answers on those questions. You would be also appreciate my own Architecture Readings series on the YouTube channel as well.
#database #db
Continuous Architecture Manifesto πΌ
As we mentioned previously, architecture activity evolved from big upfront design to no design at all. Although the spectre has long range, we can find ourselves in another point of it - the place where we do architecture continuously. It means targeting a long-term product vision, and not a short-term project; performing it by the whole team; using holistic approach and more!
#architecture
Video
Highlights
Architect's Intro to Data Science and Machine Learning πΌ
I personally kinda afraid of Data Science and ML, because I think it requires a lot of math I am not really good at. However, you will hardly build a complex enough system without either of those. So you'd better understand the terms and underlying principles at least at a high level. Redhat wrote a post in their "Enable Architect" blog with what you as an architect should know about the Data Science and Machine Learning. The list includes courses and learning materials, data manipulation tools, programming languages, data pipelines, types of machine learning and more.
#datascience #ml
Relational Databases Explained π€
We use RDBMS almost every day in our day to day jobs, but do we know enough? Do we know how do indexes work? Or how do dbs store data on the disk? What are the transactions and levels of isolations? Architecture Notes got you back with answers on those questions. You would be also appreciate my own Architecture Readings series on the YouTube channel as well.
#database #db
Continuous Architecture Manifesto πΌ
As we mentioned previously, architecture activity evolved from big upfront design to no design at all. Although the spectre has long range, we can find ourselves in another point of it - the place where we do architecture continuously. It means targeting a long-term product vision, and not a short-term project; performing it by the whole team; using holistic approach and more!
#architecture
YouTube
Continuous Architecture and Relational Databases explained
Continuous Architecture Manifesto, Relational Databases explained, Data Science and Machine Learning for Software Architects and more in Architecture Weekly #52
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-52/
Telegram channel:β¦
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-52/
Telegram channel:β¦
π2
Architecture Weekly #52 - Follow-Up
Reading DDIA notes πΌ
Designing Data Intensive Applications is a classical book nowadays. Magda Miu started a series of posts where she shares her notes while reading the book. She explains what she got from the book chapters, provides illustrations and complements it with the links for the further readings. Don't hesitate to follow her blog as well!
#ddia #reading #distributedsystem
Introduction to Time-Series Databases πΌ
We face time-series on multiple occasions from monitoring data to IoT and financial events. In order to store this data efficiently, there are special databases: TSDBs. What are there advantages and why they fit well with this type of data, read in the article below.
#database #timeseries
Consider All Microservices Vulnerable πΌ
Microservices usually have one well-defined purpose and narrow API surface. It helps improving the security, but does not obviously eliminate the threat completely. One of the tactics to improve the security of microservices in particular is monitoring their behaviour. If the malicious irefular requests and irregular response times are detected, then we can detect or even prevent an attack at an earlier stage. More details inside.
#k8s #security
Why Bolt use Node.JS? π·ββοΈ
Bolt uses exclusively Node.JS and Typescript for all the backend microservices, except ML and Data Science payloads. This is of course a architecturally significant decision which follows the business motivation of doing more with less. This week my coworker Denys Pysmennyi wrote a piece about this choice, it's consequences and drawbacks.
#backend #technology #nodejs
McDonald's Journey to Event-Driven Architecture π·ββοΈ
McDonald's is not only a set of restaurants for a quick break. It has rich IT infrastructure behind the scene. And the foundation for their systems is Event-Drivent architecture. In this post Derek Comartin shares some details from the McDonalds posts on how do they use Schema Registries, how they validate events and improve the messaging reliability.
#events
Any Decision Record? Architecture, Design and Beyond πΌ
Decision Records again! This time Olaf Zimmermann, the author of βPatterns of API Design", published a 10-minutes read about the applicability of ADRs for capturing not only architecture decisions, but as well as managerial or organizational ones. He also reasons what actually an architecturally significant decision is, and why a programming language choice is a one(mind the previous article). Get a read!
#architecture #adr
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
Reading DDIA notes πΌ
Designing Data Intensive Applications is a classical book nowadays. Magda Miu started a series of posts where she shares her notes while reading the book. She explains what she got from the book chapters, provides illustrations and complements it with the links for the further readings. Don't hesitate to follow her blog as well!
#ddia #reading #distributedsystem
Introduction to Time-Series Databases πΌ
We face time-series on multiple occasions from monitoring data to IoT and financial events. In order to store this data efficiently, there are special databases: TSDBs. What are there advantages and why they fit well with this type of data, read in the article below.
#database #timeseries
Consider All Microservices Vulnerable πΌ
Microservices usually have one well-defined purpose and narrow API surface. It helps improving the security, but does not obviously eliminate the threat completely. One of the tactics to improve the security of microservices in particular is monitoring their behaviour. If the malicious irefular requests and irregular response times are detected, then we can detect or even prevent an attack at an earlier stage. More details inside.
#k8s #security
Why Bolt use Node.JS? π·ββοΈ
Bolt uses exclusively Node.JS and Typescript for all the backend microservices, except ML and Data Science payloads. This is of course a architecturally significant decision which follows the business motivation of doing more with less. This week my coworker Denys Pysmennyi wrote a piece about this choice, it's consequences and drawbacks.
#backend #technology #nodejs
McDonald's Journey to Event-Driven Architecture π·ββοΈ
McDonald's is not only a set of restaurants for a quick break. It has rich IT infrastructure behind the scene. And the foundation for their systems is Event-Drivent architecture. In this post Derek Comartin shares some details from the McDonalds posts on how do they use Schema Registries, how they validate events and improve the messaging reliability.
#events
Any Decision Record? Architecture, Design and Beyond πΌ
Decision Records again! This time Olaf Zimmermann, the author of βPatterns of API Design", published a 10-minutes read about the applicability of ADRs for capturing not only architecture decisions, but as well as managerial or organizational ones. He also reasons what actually an architecturally significant decision is, and why a programming language choice is a one(mind the previous article). Get a read!
#architecture #adr
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
Magda Miu
Reliability, Scalability & Maintainability
When designing systems/services is about combining functional requirements (what the system should do) with non-functional requirements (quality attributes or ilities). The challenge here is to devβ¦
π1
Folks! I need a couple of developers with good GoLang experience and good English to interview candidates for a Go Developer position. Compensation is 100$ per interview. Drop me a DM if interested π
π€‘4π1π1
Architecture Weekly #53 - Highlights
Video
Is Scalable OLTP in the Cloud a Solved Problem? π€
As you folks love papers, I bring a fresh interesting one to the Highlights. Murat Demirbas from AWS posted an overview of a paper called "Is Scalable OLTP in the Cloud a Solved Problem?". The overview and the paper itself describe the shared-nothing, single-writer and multiple-writers designs of a scalable OLTP database and discuss their tradeoffs. Murat also highlights that the paper fails to consider metastable errors(which you can find a paper on here). The second part of the original paper is about a new DB and that's not that interesting, but the first part is gold.
#database #cloud #oltp #paper
Yandex Services Source Code Leak πΌ
The internet blew up this week with a major data source leak from a russian IT giant Yandex. The dump contains 44 GB of source code from their monorepository - which was not a single storage of the code in the company. The dump contains tons of services, configs, user-generated data like prompts for Voice Assistant and even database credentials for insider access. Although there are no trained models though, and you can't really build anything from it, as yandex infrastructure is required, you can still use it to learn how the things are done at a company of that scale.
#security #breach
System Design Interview Survival Guide πΌ
I know you folks read the newsletter partially because you want to be ready for the design interview. And the 3rd highlight today is the survival guide for exactly those. The article goes over the list of concepts to be aware of, the architecture patterns, database types, distribution system problems etc. Use as a nice checklist to prepare for the interview! And of course, you can reach out to @vvsevolodovich if you need a personal consultation π
#systemdesign
Video
Is Scalable OLTP in the Cloud a Solved Problem? π€
As you folks love papers, I bring a fresh interesting one to the Highlights. Murat Demirbas from AWS posted an overview of a paper called "Is Scalable OLTP in the Cloud a Solved Problem?". The overview and the paper itself describe the shared-nothing, single-writer and multiple-writers designs of a scalable OLTP database and discuss their tradeoffs. Murat also highlights that the paper fails to consider metastable errors(which you can find a paper on here). The second part of the original paper is about a new DB and that's not that interesting, but the first part is gold.
#database #cloud #oltp #paper
Yandex Services Source Code Leak πΌ
The internet blew up this week with a major data source leak from a russian IT giant Yandex. The dump contains 44 GB of source code from their monorepository - which was not a single storage of the code in the company. The dump contains tons of services, configs, user-generated data like prompts for Voice Assistant and even database credentials for insider access. Although there are no trained models though, and you can't really build anything from it, as yandex infrastructure is required, you can still use it to learn how the things are done at a company of that scale.
#security #breach
System Design Interview Survival Guide πΌ
I know you folks read the newsletter partially because you want to be ready for the design interview. And the 3rd highlight today is the survival guide for exactly those. The article goes over the list of concepts to be aware of, the architecture patterns, database types, distribution system problems etc. Use as a nice checklist to prepare for the interview! And of course, you can reach out to @vvsevolodovich if you need a personal consultation π
#systemdesign
YouTube
Scalable OLTP as a solved problem, Yandex Source Code leak and System Design Interview Guide
Architecture Weekly #53
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-53/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
Consultationsβ¦
Text version of the issue: https://vvsevolodovich.dev/architecture-weekly-53/
Telegram channel: https://t.me/architectureweekly
Patreon: https://www.patreon.com/vvsevolodovich
Boosty: https://boosty.to/vvsevolodovich
Consultationsβ¦
π₯3
Follow-Up
Software Trends Report: What to Watch for in 2023 πΌ
As we just wrapped up the 2022, it is good to get back and get an overview of the trends of the last year and see what we can expect in future. In this post at InfoQ they mention the use of ADRs, importance of Senior IC role and the emerge of Platform teams. They also highlight that you would rather treat platforms and data as products and handle those accordingly. As regarding the software architecture I've invited Uwe Friedrechsen for an hour long interview, so book your calendars!
#architecture #review
How to autoscale your SaaS application Infrastructure π·ββοΈ
RedHat has a series on SaaS Architecture Checklist. Recently they shared Chapter 9 which is about autoscaling the infrastructure. Michael Hrivnak explains how autoscaling can be configured using Cluster API, Cluster Autoscaler and Horizontal Pod Autoscaler so that your SaaS resources are truly elastic.
#kubernetes #saas #autoscaling #scalability
Must Read Free Kubernetes Books πΌ
Speaking of Kubernetes, take a look at the set of free books which help you to study Kubernetes itself, it's performance and security. The set also includes books on integration patterns and designing distributed systems.
#kata #books
Solution Architecture Kata Example π·ββοΈ
One of the books I recommended in this post, was "Fundamentals of Software Architecture" by Neil Ford. He also knows as a promoter of "Architecture Kata" exercise and he shows an example of a task for such activity in the aforementioned book. What I want to share with you is solving a problem to that particular Kata in this post on Medium. It includes understanding the context and iterating over the design versions. Take a look.
#architecture #kata
Document your event-driven architecture πΌ
In a loosely coupled, event-driven architecture we still want to know What events/messages is a particular service producing, what events/messages can I consume from the service, what is the message schema and more. This short article shows the 3 ways how you can document the answers to those questions.
#documentation #eda
Metrics for quality πΌ
When you introduce any metric for your software project, team or whatever else, you will always find out that the metric stops being useful due to Goodhart's law. So if you need to optimize your product for quality, what do you do? Well, if you study the quality itself, you will find out the outer, inner and process quality parts. The article though does not answer what metrics to pick - just how to think about the quality itself.
#quality #qa
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
Software Trends Report: What to Watch for in 2023 πΌ
As we just wrapped up the 2022, it is good to get back and get an overview of the trends of the last year and see what we can expect in future. In this post at InfoQ they mention the use of ADRs, importance of Senior IC role and the emerge of Platform teams. They also highlight that you would rather treat platforms and data as products and handle those accordingly. As regarding the software architecture I've invited Uwe Friedrechsen for an hour long interview, so book your calendars!
#architecture #review
How to autoscale your SaaS application Infrastructure π·ββοΈ
RedHat has a series on SaaS Architecture Checklist. Recently they shared Chapter 9 which is about autoscaling the infrastructure. Michael Hrivnak explains how autoscaling can be configured using Cluster API, Cluster Autoscaler and Horizontal Pod Autoscaler so that your SaaS resources are truly elastic.
#kubernetes #saas #autoscaling #scalability
Must Read Free Kubernetes Books πΌ
Speaking of Kubernetes, take a look at the set of free books which help you to study Kubernetes itself, it's performance and security. The set also includes books on integration patterns and designing distributed systems.
#kata #books
Solution Architecture Kata Example π·ββοΈ
One of the books I recommended in this post, was "Fundamentals of Software Architecture" by Neil Ford. He also knows as a promoter of "Architecture Kata" exercise and he shows an example of a task for such activity in the aforementioned book. What I want to share with you is solving a problem to that particular Kata in this post on Medium. It includes understanding the context and iterating over the design versions. Take a look.
#architecture #kata
Document your event-driven architecture πΌ
In a loosely coupled, event-driven architecture we still want to know What events/messages is a particular service producing, what events/messages can I consume from the service, what is the message schema and more. This short article shows the 3 ways how you can document the answers to those questions.
#documentation #eda
Metrics for quality πΌ
When you introduce any metric for your software project, team or whatever else, you will always find out that the metric stops being useful due to Goodhart's law. So if you need to optimize your product for quality, what do you do? Well, if you study the quality itself, you will find out the outer, inner and process quality parts. The article though does not answer what metrics to pick - just how to think about the quality itself.
#quality #qa
Like the newsletter? Wanna receive new content earlier, than everybody else? Consider helping to run it at Patreon or Boosty. The funds go to pay for the hosting and some software like a Camo Studio license. Patrons and Boosty subscribers of a certain level also get access to a private Architecture Community and of course every supporter gets early access. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August and Roman for already supporting the newsletter. Join them as well!
InfoQ
InfoQ Software Trends Report: Major Trends in 2022 and What to Watch for in 2023
2022 was another year of significant technological innovations and trends in the software industry and communities. The InfoQ podcast co-hosts met last month to discuss the major trends from 2022, and what to watch for in 2023. This article is a summary ofβ¦
π4π₯1
Time for the announcements! Fasten your seat belts π
1. Software Architecture in 2022 Review with Uwe Friedrechsen happens on Feb, 6, 2023 at 2 PM GMT+2. Broadcast here.
2. Where do microservies go? Interview with Andrey Rebrov about do we go back to a monolith or lambda, what technologies support the microservies and what an ideal ms looks like.
Feb, 9, 2023 at 4 PM GMT+2. Broadcast here.
Make sure to subscribe and click the notify button π
1. Software Architecture in 2022 Review with Uwe Friedrechsen happens on Feb, 6, 2023 at 2 PM GMT+2. Broadcast here.
2. Where do microservies go? Interview with Andrey Rebrov about do we go back to a monolith or lambda, what technologies support the microservies and what an ideal ms looks like.
Feb, 9, 2023 at 4 PM GMT+2. Broadcast here.
Make sure to subscribe and click the notify button π
YouTube
Architecture in 2022 in Review with Uwe Friedrechsen
I am interviewing a Software Architect and frequent speaker Uwe Friedrechsen on where we find ourselves with Platform Teams, Services Meshes, Observability, Testing for Distributed Systems and many more!
Release It! Design and Deploy Production-Ready Software:β¦
Release It! Design and Deploy Production-Ready Software:β¦
π₯8