Если вдруг кто не знал, последнюю версию "Distributed systems: principles and paradigms" 3d edition (3.03 от 2020 года) by Andrew S. Tanenbaum, Maarten Van Steen можно скачать бесплатно на официальном сайте:

https://www.distributed-systems.net/index.php/books/ds3/

Пусть вас не смущает, что на сайте указан 2017й год. На почту придет книга с редакцией за 2020.

🔷 "Diagramming to Communicate" by Vaughn Vernon
- Статья про UML, EventStorming, способы выражения SAGA в EventStorming, ну и конечно же про разрабатываемый им сервис для моделирования EventStorming https://domorobo.to/ с генерацией кода посредством XOOM Designer.
💬 "Kalele is building the modeling tool DomoRoboto. It hosts seven model types, including EventStorming, DDD Context Maps that are auto-derived from EventStorming models, Topo Architecture (a blend of EventStorming results with Context Maps), Mind Maps, Impact Maps, Specification by Example (BDD), and User Stories."

#DDD #SoftwareArchitecture #EventStorming

🔷 "Event Sourcing: Why Kafka is not suitable as an Event Store" - Don’t believe everything a company or some consultants want to sell you!
by Anton Stöckl - автор проекта go-iddd - Golang DDD CQRS/ES Reference Application

#DDD #CQRS #EventSourcing #SoftwareDesign

Последняя трактовка принципов SOLID by Robert C. Martin от 2022-07-06 (ничего нового, но снижает вероятность ложного восприятия):

💬 "The Single Responsibility Principle (SRP):

Gather together those things that change for the same reasons and at the same times. Separate those things that change for different reasons or at different times."

💬 "The Open-Closed Principle (OCP):

Separate modules that frequently change from modules that change less frequently with a layer of abstraction."

💬 "The Liskov Substitution Principle (LSP):

The implementation of an interface must never violate the contract between that interface and its users."

💬 "The Interface Segregation Principle (ISP):

Don't depend on things you don’t need."

💬 "The Dependency Inversion Principle (DIP):

Lower level policies should depend on higher level policies."

#SoftwareDesign

🔷 Хотя сайт Alistair Cockburn все еще не работоспособен, оригинальная статья "Hexagonal architecture" доступна для просмотра.

Anton Stöckl не так давно опубликовал очередную статью из цикла, посвященного его проекту go-iddd - Golang DDD CQRS/ES Reference Application:
🔷 "Hexagonal Architecture: Structuring a project and the influence of granularity" - About the granularity of ports and adapters and how it influences the code structure.

Предыдущая статья этого цикла, посвященная Hexagonal Architecture:
🔷 "Implementing Domain-Driven Design and Hexagonal Architecture with Go (3)" - Part 3 — How I structure my application according to Hexagonal Architecture aka. Ports&Adapters

#DDD #SoftwareDesign #SoftwareArchitecture

Выступление одного из учередителей нашего Объединения на ArchDays Recap https://www.youtube.com/watch?v=NSN-NXfbEqM

🔷 "Read models in event-sourced systems" by Alexey Zimarev
Статья является частью документации Eventuous.

#DDD #CQRS #EventSourcing #SoftwareDesign

Видео от Mark Richards о том, как формировать собственные знания, имеет особое значение для нашего канала, поскольку пересекается с его целями:

🔷 "Soft Skills: Gaining Technical Breadth" by Mark Richards

#SoftSkills

Есть ли альтернатива Transactional Outbox pattern? В последнее время этот вопрос часто появляется в пабликах.

Говорят, что "Transactional Outbox" под названием "Local Messaging" впервые был опубликован в статье "BASE: An Acid Alternative: In partitioned databases, trading some consistency for availability can lead to dramatic improvements in scalability" to ACM by ebay architect Dan Pritchett in 2008.

Как говорил Alexey Zimarev:

💬 "если же это первое сообщение в цепочке - то уже есть смысл смотреть на transactional outbox".

Имеется ввиду в случае отсутствия "Front-Door Queue".

Глубокое понимание этой проблемы и способов ее решения дается в главе "10.Messaging Endpoints :: Transactional Client" книги "Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions" by Gregor Hohpe, Bobby Woolf.

А также эта тема затрагивается в главе "Chapter 9. Message Endpoints :: Transactional Client/Actor" книги "Reactive Messaging Patterns with the Actor Model: Applications and Integration in Scala and Akka" by Vaughn Vernon.

В приведенных главах подробно раскрывается утверждение of Alexey Zimarev.

Vaughn Vernon посвящает этой проблеме главу "8 Domain Events :: Spreading the News to Remote Bounded Contexts :: Messaging Infrastructure Consistency" книги "Implementing Domain-Driven Design".

Ссылки по теме:

🔷 "Subscribing to events :: Publishing events through the event bus :: Designing atomicity and resiliency when publishing to the event bus" by Cesar de la Torre, Bill Wagner, Mike Rousos;
🔷 "Subscribing to events :: Resiliently publishing to the event bus" by Cesar de la Torre, Bill Wagner, Mike Rousos;
🔷 "Transactional messaging" by Chris Richardson
🔷 "3.3.7 Transactional messaging" by Chris Richardson
🔷 "The Outbox Pattern" by Kamil Grzybek
🔷 "Achieving reliable dual writes in distributed systems" by Rishabh Gupta

🔷 "Transaction log mining"
🔷 "Transaction log tailing"

💬 "Both a sender and a receiver can be transactional. With a sender, the message isn’t “really” added to the channel until the sender commits the transaction. With a receiver, the message isn’t “really” removed from the channel until the receiver commits the transaction. A sender that uses explicit transactions can be used with a receiver that uses implicit transactions, and vise versa. A single channel might have a combination of implicitly and explicitly transactional senders; it could also have a combination of receivers.

With a transactional receiver, an application can receive a message without actually removing the message from the queue. At this point, if the application crashed, when it recovered, the message would still be on the queue; the message would not be lost. Having received the message, the application can then process it. Once the application is finished with the message and is certain it wants to consume it, the application commits the transaction, which (if successful) removes the message from the channel. At this point, if the application crashed, when it recovered, the message would no longer be on the channel, so the application had better truly be finished with the message.

How does controlling a messaging system’s transactions externally help an application coordinate several tasks? Here’s what the application would do in the scenarios described above:"
-- "Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions" by Gregor Hohpe, Bobby Woolf, Chapter "Transactional Client"

#Microservices #DistributedSystems #Integration #DDD

Можно ли изменять несколько агрегатов одной транзакцией? В профессиональной среде существует популярное мнение что нельзя. Этот вопрос затрагивает Vladimir Khorikov в статье "Crossing aggregate boundaries", и говорит, что:

> I hear this guideline a lot as well.

Тоже слышал эту фразу много раз, как правило от тех, кто эту книгу сам не читал. Именно поэтому важно получать информацию из первоисточника, где V.Vernon сопровождает эту информацию еще целым рядом дополнительной информации, в которой он не только допускает, но и рекомендует в некоторых случаях фиксировать несколько агрегатов одной транзакцией. Более того, он раскрывает причины такого правила, а знание причин освобождает от догматизма и опасности впадения в Cargo Cult.

Но, ок, а как все-таки обеспечить соблюдение инвариантов в условиях отсутствия поддержки строгой транзакционной согласованности?

Недавно, как раз обсуждал этот вопрос в одной из заметок.

Одним из решений является "Data, context, and interaction (DCI)" (от авторов MVC).
Подробное описание приводится в главе "Chapter 9. Coding it Up: The DCI Architecture" книги "Lean Architecture: for Agile Software Development" 1st edition by James O. Coplien, Gertrud Bjørnvig.
Можно посмотреть на примере Golang-реализации перевода денежных средств с одного счета на другой счет, с использованием DDD & DCI (доступна также python-версия).

Также стоит выделить в этом вопросе интервью "Modeling Uncertainty with Reactive DDD" by Vaughn Vernon reviewed by Thomas Betts, где решение подобных задач возлагается на Process Manager Pattern.

Другие варианты приводятся в заметке.

#DDD #DistributedSystems

Какие выгоды даёт использование FP? Вот что говорит автор CQS, добавивший функциональщины в OOP, объясняя причины своего решения:

💬 "Referential transparency

Why should we be concerned about side effects in functions? After all it is in the nature of software execution to change things.

The problem is that if we allow functions to change things as well as commands, we lose many of the simple mathematical properties that enable us to reason about our software. As noted in the discussion of abstract data types, when we first encountered the distinction between the applicative and the imperative, mathematics is change-free: it talks about abstract objects and defines operations on these objects, but the operations do not change the objects. (Computing square root of 2 does not change the number two.) This immutability is the principal difference between the worlds of mathematics and computer software.

Some approaches to programming seek to retain the immutability of mathematics: Lisp in its so-called "pure" form, "Functional Programming" languages such as Backus's FP, and other applicative languages shun change. But they have not caught on for practical software development, suggesting that change is a fundamental property of software.

The object immutability of mathematics has an important practical consequence known as referential transparency, a property defined as follows:

Definition: referential transparency

An expression e is referentially transparent if it is possible to exchange any subexpression with its value without changing the value of e.

If x has value three, we can use x instead of 3, or conversely, in any part of a referentially transparent expression. (Only Swift's Laputa academicians were willing to pay the true price of renouncing referential transparency: always carrying around all the things you will ever want to talk about.) As a consequence of the definition, if we know that x and y have the same value, we can use one interchangeably with the other. For that reason referential transparency is also called "substitutivity of equals for equals".

With side-effect-producing functions, referential transparency disappears. Assume a class contains the attribute and the function

attr: INTEGER

sneaky: INTEGER is do attr := attr + 1 end

Then the value of sneaky (meaning: of a call to that function) is always 0; but you cannot use 0 and sneaky interchangeably, since an extract of the form

attr := 0; if attr /= 0 then print ("Something bizarre!") end

will print nothing, but would print Something bizarre! if you replaced 0 by sneaky.

Maintaining referential transparency in expressions is important to enable us to reason about our software. One of the central issues of software construction, analyzed clearly by Dijkstra many years ago, is the difficulty of getting a clear picture of the dynamic behavior (the myriad possible executions of even a simple software element) from its static description (the text of the element). In this effort it is essential to be able to rely on the proven form of reasoning, provided by mathematics. With the demise of referential transparency, however, we lose basic properties of mathematics, so deeply rooted in our practice that we may not even be aware of them. For example, it is no longer true that n + n is the same thing as 2 ∗ n if n is the sneaky-like function

n: INTEGER is do attr := attr + 1; Result := attr end 

since, with attr initially zero, 2 ∗ n will return 2 whereas n + n will return 3.

By limiting ourselves to functions that do not produce side effects, we will ensure that talking about "functions" in software ceases to betray the meaning of this term in ordinary mathematics. We will maintain a clear distinction between commands, whichchange objects but do not directly return results, and queries, which provide information about objects but do not change them. Another way to express this rule informally is to state that asking a question should not change the answer."
-- "Object-Oriented Software Construction" 2nd edition by Bertrand Meyer

Немного определений к предыдущему сообщению:

🔷 Definition: correctness

Correctness is the ability of software products to perform their exact tasks, as defined by their specification.

🔷 Definition: robustness

Robustness is the ability of software systems to react appropriately to abnormal conditions.

-- "Object-Oriented Software Construction" 2nd edition by Bertrand Meyer

Тут стоит заметить, что Bertrand Meyer выделяет два типа side-effect:

🔷 Definition: concrete side effect

A function produces a concrete side effect if its body contains any of the following:
- An assignment, assignment attempt or creation instruction whose target is an attribute.
- A procedure call.

🔷 Definition: abstract side effect

An abstract side effect is a concrete side effect that can change the value of a non-secret query.

-- "Object-Oriented Software Construction" 2nd edition by Bertrand Meyer

CQS накладывает ограничение только на abstract side effect (который часто называют observable), т.е. наблюдаемый клиентами класса.
Дело в том, что concrete side effect ("only affecting properties that are not visible to clients") может быть "easily detect", а потому - "some concrete side effects are not only harmless but necessary."

💬 "This is the notion used by the Command-Query Separation principle — the principle The principle that prohibits abstract side effects in functions."
-- "Object-Oriented Software Construction" 2nd edition by Bertrand Meyer

Конечно, эти краткие цитаты не передают всей глубины сути, так что цель этого поста не передать какое-то знание, а привлечь к нему внимание. В первоисточнике приводятся примеры. Чуть больше информации доступно в этой заметке.

💬 "I/O is inherently impure: input operations undermine referential transparency, and output operations create side effects. Nevertheless, there is a sense in which function can perform input or output and still be pure, if the sequence of operations on the relevant I/O devices is modeled explicitly as both an argument and a result, and I/O operations are taken to fail when the input sequence does not describe the operations actually taken since the program began execution.
The second point ensures that the only sequence usable as an argument must change with each I/O action; the first allows different calls to an I/O-performing function to return different results on account of the sequence arguments having changed.[5][6]
The I/O monad is a programming idiom typically used to perform I/O in pure functional languages."

5. Peyton Jones, Simon L. (2003). Haskell 98 Language and Libraries: The Revised Report (PDF). Cambridge, United Kingdom: Cambridge University Press. p. 95. ISBN 0-521 826144. Retrieved 17 July 2014.
6. Hanus, Michael. "Curry: An Integrated Functional Logic Language" (PDF). www-ps.informatik.uni-kiel.de. Institut für Informatik, Christian-Albrechts-Universität zu Kiel. p. 33. Archived from the original (PDF) on 25 July 2014. Retrieved 17 July 2014.

— https://en.m.wikipedia.org/wiki/Pure_function

Наконец, подбираемся к Event Sourcing:

💬 "Что особенно важно, никакая информация не удаляется из такого хранилища и не изменяется. Как следствие, от набора CRUD-операций в приложениях остаются только CR. Также отсутствие операций изменения и/или удаления с хранилищем устраняет любые проблемы конкурирующих обновлений.

Обладая хранилищем достаточного объема и достаточной вычислительной мощностью, мы можем сделать свои приложения полностью неизменяемыми — и, как следствие, полностью функциональными.

Если это все еще кажется вам абсурдным, вспомните, как работают системы управления версиями исходного кода.

More importantly, nothing ever gets deleted or updated from such a data store. As a consequence, our applications are not CRUD; they are just CR. Also, because neither updates nor deletions occur in the data store, there cannot be any concurrent update issues.

If we have enough storage and enough processor power, we can make our applications entirely immutable—and, therefore, entirely functional.

If this still sounds absurd, it might help if you remembered that this is precisely the way your source code control system works."
— "Clean Architecture: A Craftsman’s Guide to Software Structure and Design" by Robert C. Martin

💬 "Event Sourcing is naturally functional. It's an append only log of facts that have happened in the past. You can say that any projection any state is a left fold over your previous history."
— Greg Young, "A Decade of DDD, CQRS, Event Sourcing" at 16:44

💬 "It's actually functional."
— Greg Young, "Event Sourcing is actually just functional code" at 34:49

💬 "I have always said that Event Sourcing is "Functional Data Storage". In this talk we will try migrating to a idiomatic functional way of looking at Event Sourcing. Come and watch all the code disappear! By the time you leave you will never want an "Event Sourcing Framework (TM)" ever again!
— Greg Young, "Functional Data", NDC Conferences

Дело в том, что для одних и тех же входных параметров (критериев выборки), Event Sourced Storage возвращает всегда один и тот же результат, т.е. обладает признаками ссылочной прозрачности.

Рост спроса на FP обусловлен не только простотой доказательства корректности, но и ростом широты использования распределенных систем:

💬 "Все состояния гонки (race condition), взаимоблокировки (deadlocks) и проблемы параллельного обновления обусловлены изменяемостью переменных. Если в программе нет изменяемых переменных, она никогда не окажется в состоянии гонки и никогда не столкнется с проблемами одновременного изменения. В отсутствие изменяемых блокировок программа не может попасть в состояние взаимоблокировки.

All race conditions, deadlock conditions, and concurrent update problems are due to mutable variables. You cannot have a race condition or a concurrent update problem if no variable is ever updated. You cannot have deadlocks without mutable locks."
— "Clean Architecture: A Craftsman’s Guide to Software Structure and Design" by Robert C. Martin

💬 "But doesn't lazy loading affect domain model purity?"

Vladimir Khorikov в статье "Domain model purity and lazy loading" приходит к следующему выводу:

💬 "A pure domain model has the following attributes:

- Its domain classes don’t explicitly refer to out-of-process dependencies or classes from the application services layer. In other words, domain classes should only depend on other domain classes or the framework’s built-in primitive types.

- All inputs to the domain model are referentially transparent, except for calls to the domain classes themselves. Referential transparency means that you can replace a method call or an expression with the output of that method call or expression, and it will not change the code’s behavior.

- All side effects are limited to the domain model — Domain classes should only modify themselves, not out-of-process dependencies or classes from the application services layer. This attribute flows from the first attribute but is still worth stating explicitly.

This attribute, along with the exception in the second attribute ("expect for calls to the domain classes themselves") is what differentiates a pure domain model from pure code in a functional programming sense. In functional programming, a pure code is always referentially transparent (which also means it works with immutable data only and doesn't incur side effects), while in DDD, the notion of purity isn't as strict."
— "Domain model purity and lazy loading" by Vladimir Khorikov

PostgresPro представил четыре книги для разных уровней подготовленности читателей, от совершенно неосведомленного человека до разработчика баз данных. Книги дают комплексные знания в лаконичной форме. Все книги доступны для скачивания в свободном доступе:

1) "Postgres: первое знакомство" / П.В. Лузанов, Е.В. Рогов, И.В. Лёвшин
2) "PostgreSQL изнутри" / Е.В. Рогов — М.: ДМК Пресс, 2022. — 660 с.
3) "PostgreSQL. Основы языка SQL: учеб. пособие" / Е.П. Моргунов; под ред. Е.В. Рогова, П.В. Лузанова.
4) "Основы технологий баз данных: учеб. пособие" / Б.А. Новиков, Е.А. Горшкова, Н.Г. Графеева; под ред. Е.В. Рогова.

В них есть все: согласованность, реляционная алгебра, формы нормализации, архитектура, структуры данных и алгоритмы, оптимизация, транзакции, надежность, безопасность, администрирование, масштабирование, и т.п.

Так же доступны учебные материалы курсов: слайды, видео, руководства. Скачать можно все материалы каждого курса одним архивом.

Видеозаписи курсов.

Превосходная подборка статей с фундаментальной информацией простым языком о внутреннем устройстве PostgreSQL, от разработчиков PostgresPro:

1) "MVCC-1. Изоляция"
2) "WAL в PostgreSQL: 1. Буферный кеш"

Ну и пара превосходных книг от разработчика PostgreSQL, но уже не в свободном доступе:

- "Mastering PostgreSQL In Application Development" by Dimitri Fontaine

- "The Art of PostgreSQL" 2nd edition by Dimitri Fontaine - is the new title of "Mastering PostgreSQL in Application Development"

#Database #PostgreSQL

Презентация с выступления.

Raft - Understandable Distributed Consensus
- https://thesecretlivesofdata.com/raft/

- простая и понятная интерактивная визуализация алгоритма.

[UPDATE]: еще одна (thanks to @AlexKirillov ):
- https://raft.github.io/

#DistributedSystems

Сравнение алгоритмов достижения консенсуса:
🔷 "Paxos, Raft, EPaxos: How Has Distributed Consensus Technology Evolved?" by Yan Xiangguang
Thanks to @doonto !

#DistributedSystems