Blockchain and Web3: just released this new episode!

• https://tinyurl.com/sdm-blockchain
• https://tinyurl.com/sdm-blockchain-slides

The history and the state of the art of blockchain and Web3 from first principles.

One of the ideas we are flirting with is to sketch out some “open source design” of a nontrivial architectural component.

If done well, not only it could serve as a reference document on how to prepare such designs, but also a blueprint based on which we could implement the prototype.

I think there’s room for such a design out there today, from contracts and protocol specifications (HTTP JSON to gRPC to binary) all the way to components’ implementation and unit/integration/load testing.

As an example problem I keep coming back to either the Total Order Broadcast engine or a simple De-Duplicator.

The former can be viewed as an extension of a service mesh proxy, which inserts a sequence ID into every requests, guaranteeing that each and every ID would be unique.

The latter can be viewed as something similar, but less so on the transport level, focusing on respecting the logic of idempotency tokens instead.

Do you think this would be a good idea? Would you be interested in participating in parts of such “open designs” one way or another? Let me know pls.

https://adamwalker.github.io/Building-FPGA-KVS/ -- there's always something fun to be done with FPGAs!

In other news, I've created a Github repo dedicated to our meetup.

Besides all the relevant links, there is now an open pull request into it, in which I outline, in an RFC form, how I envision the future "open design" work.

It should expand the markdown nicely if you click "Display the rich diff" next to the only changed file on Github UI, or try this link to get to the respective branch and read it in rich format right away.

Comments are more than welcome, please and thank you. And cheers to a good start, I sure hope this opens the next big chapter in this venture we are enjoying together!

Upd: The PR with Principles is merged.

The next step is to decide what problem is worthy of working on it first.

My major idea remains the Total Order Broadcast Engine. But it may well be too ambitious for the first shot, and we'll settle for something smaller to start from.

Comments and suggests are more than welcome.

PS: On an unofficial and off the record event yesterday we spoke of Kubernetes vs. Borg vs. Mesos vs. Odin, and overall about the future of scheduling short-term and evergreen jobs in a constellation of datacenters scattered around the globe. Fun times!

Publishing Enriched Data Update Events
"When caching and "poor man's replication" play well together."

I've put together quite a few thoughts, and would appreciate external wisdom at how to approach them from the larger eng org standpoint.

https://dimakorolev.substack.com/p/publishing-enriched-data-update-events

The next — Educational — episode of the meetup is up.

Never confuse functional, integration, regression, and end-to-end tests again!

Moreover, don't take this very question seriously. For if you want to set things straight, offense is the best defense!

Video: https://tinyurl.com/sdm-testing-video
Slides: https://tinyurl.com/sdm-testing-slides

https://www.youtube.com/watch?v=y8OnoxKotPQ -- a really well done pun, many thanks to the kind soul in our Slack who shared it!

Sharing my longread on Event Sourcing here as well.

(Likely a topic for one of the next on-air meetups; we'll chat about it off the record as well, that's for sure.)

If my writing is too dense -- I'm working on style every day! -- this post on building real-time exchanges is very good too; I could have linked to it from my post, but, admittedly, forgot to.

I've published a longread about Event Sourcing, in the "Gentle Introduction" way.

The introduction is gentle both from the engineering standpoint and from the grounds of convincing the business to begin making the first steps.

If you struggle with justifying it to the stakeholders that the company needs to upgrade its data stack, or if you already have a soft buy-in but are unsure how to best proceed -- I hope you'd find this post helpful and insightful.

Yours truly designing Uber. Good to be on the other end from time to time, many thanks to Tom for interviewing me.

Also, I gotta work on being less confusing. Although think the conversation went quite well.

https://open.substack.com/pub/dimakorolev/p/the-oltp-language -- collected my thoughts on how can we accelerate the retirement of SQL as the meant to implement OLTP transactions.

Just clicked "Publish" on two posts that were in the works for well over a month. Behold:

• Distributed Stateful Workflows, and
• Stateful Orchestration Engines.

Hope you enjoy reading them as much as I enjoyed writing them. And spread the word!

The episode on Stateful Orchestration is out!

In reality this episode sets the table for proper introduction into the domain of stateful orchestration. I talk in detail about many SysDesign problems leading to formulating the problem, from the idea of serialization and vector clocks, through decoupling of producers and consumers, and all the way to Cadence and Temporal as generic mechanisms to enable SAGAs and more sophisticated patterns at scale.

Durable Execution, a.k.a. Stateful Orchestration 2.0.

The closing meetup episode of 2024 is now released.

https://tinyurl.com/sdm-durable-execution
https://tinyurl.com/sdm-durable-execution-slides

Would be great to build this together in 2025 and beyond!

#Milestone

I guess ... thank you — and looking forward to much more!

Folks, Maxim Fateev, the founder of Temporal has a super discount code — CSDM80 — for those of us who want to attend Replay 2025, https://replay.temporal.io/.

It's literally 80% off. Definitely good enough in my book to share with the community. Hope to see you there!

A very good catalogue, via @vanadium23.

https://muratbuffalo.blogspot.com/2021/02/foundational-distributed-systems-papers.html