https://adamwalker.github.io/Building-FPGA-KVS/ -- there's always something fun to be done with FPGAs!
adamwalker.github.io
Building a Networked Key-Value-Store on an FPGA
This post is about building an FPGA based networked key-value-store in the functional hardware description language Clash. I’ll describe my FPGA-tailored hashtable design, the testing framework I built for it, and its performance and resource usage.
In other news, I've created a Github repo dedicated to our meetup.
Besides all the relevant links, there is now an open pull request into it, in which I outline, in an RFC form, how I envision the future "open design" work.
It should expand the markdown nicely if you click "Display the rich diff" next to the only changed file on Github UI, or try this link to get to the respective branch and read it in rich format right away.
Comments are more than welcome, please and thank you. And cheers to a good start, I sure hope this opens the next big chapter in this venture we are enjoying together!
Besides all the relevant links, there is now an open pull request into it, in which I outline, in an RFC form, how I envision the future "open design" work.
It should expand the markdown nicely if you click "Display the rich diff" next to the only changed file on Github UI, or try this link to get to the respective branch and read it in rich format right away.
Comments are more than welcome, please and thank you. And cheers to a good start, I sure hope this opens the next big chapter in this venture we are enjoying together!
GitHub
GitHub - SysDesignMeetup/sdm: Everything about the SysDesign Meetup by @dkorolev.
Everything about the SysDesign Meetup by @dkorolev. - SysDesignMeetup/sdm
👍3
Upd: The PR with Principles is merged.
The next step is to decide what problem is worthy of working on it first.
My major idea remains the Total Order Broadcast Engine. But it may well be too ambitious for the first shot, and we'll settle for something smaller to start from.
Comments and suggests are more than welcome.
PS: On an unofficial and off the record event yesterday we spoke of Kubernetes vs. Borg vs. Mesos vs. Odin, and overall about the future of scheduling short-term and evergreen jobs in a constellation of datacenters scattered around the globe. Fun times!
The next step is to decide what problem is worthy of working on it first.
My major idea remains the Total Order Broadcast Engine. But it may well be too ambitious for the first shot, and we'll settle for something smaller to start from.
Comments and suggests are more than welcome.
PS: On an unofficial and off the record event yesterday we spoke of Kubernetes vs. Borg vs. Mesos vs. Odin, and overall about the future of scheduling short-term and evergreen jobs in a constellation of datacenters scattered around the globe. Fun times!
Publishing Enriched Data Update Events
"When caching and "poor man's replication" play well together."
I've put together quite a few thoughts, and would appreciate external wisdom at how to approach them from the larger eng org standpoint.
https://dimakorolev.substack.com/p/publishing-enriched-data-update-events
"When caching and "poor man's replication" play well together."
I've put together quite a few thoughts, and would appreciate external wisdom at how to approach them from the larger eng org standpoint.
https://dimakorolev.substack.com/p/publishing-enriched-data-update-events
Dima Korolev
Publishing Enriched Data Update Events
When caching and "poor man's replication" play well together.
The next — Educational — episode of the meetup is up.
Never confuse functional, integration, regression, and end-to-end tests again!
Moreover, don't take this very question seriously. For if you want to set things straight, offense is the best defense!
Video: https://tinyurl.com/sdm-testing-video
Slides: https://tinyurl.com/sdm-testing-slides
Never confuse functional, integration, regression, and end-to-end tests again!
Moreover, don't take this very question seriously. For if you want to set things straight, offense is the best defense!
Video: https://tinyurl.com/sdm-testing-video
Slides: https://tinyurl.com/sdm-testing-slides
YouTube
Testing:: Educational SysDesign Meetup :: 2023-August-26
Today, in this educational episode, we talk about testing software.
Timekeys:
0:00 Intro
0:39 Goals
4:32 ChatGPT screenshots
5:33 A test that is failing is a good test!
6:56 Types of tests
8:34 Dimensions to the types of tests
13:24 Who is the stakeholder…
Timekeys:
0:00 Intro
0:39 Goals
4:32 ChatGPT screenshots
5:33 A test that is failing is a good test!
6:56 Types of tests
8:34 Dimensions to the types of tests
13:24 Who is the stakeholder…
👍5
https://www.youtube.com/watch?v=y8OnoxKotPQ -- a really well done pun, many thanks to the kind soul in our Slack who shared it!
YouTube
Microservices
it's because of the way our backend works
// more krazam scenes: https://www.patreon.com/KRAZAM
// merch: https://merch.krazam.tv
// https://www.instagram.com/krazam.tv
// https://twitter.com/krazamtv
// more krazam scenes: https://www.patreon.com/KRAZAM
// merch: https://merch.krazam.tv
// https://www.instagram.com/krazam.tv
// https://twitter.com/krazamtv
Sharing my longread on Event Sourcing here as well.
(Likely a topic for one of the next on-air meetups; we'll chat about it off the record as well, that's for sure.)
If my writing is too dense -- I'm working on style every day! -- this post on building real-time exchanges is very good too; I could have linked to it from my post, but, admittedly, forgot to.
(Likely a topic for one of the next on-air meetups; we'll chat about it off the record as well, that's for sure.)
If my writing is too dense -- I'm working on style every day! -- this post on building real-time exchanges is very good too; I could have linked to it from my post, but, admittedly, forgot to.
Adaptive - Trading Technology, Built for You.
Building fault-tolerant, low-latency exchanges | Blog | WeAreAdaptive.com
Before we discuss how we build bespoke marketplaces at Adaptive, let’s consider an alternative: buying an off-the-shelf solution.
Forwarded from Not boring, and a bit of a condescending prick
I've published a longread about Event Sourcing, in the "Gentle Introduction" way.
The introduction is gentle both from the engineering standpoint and from the grounds of convincing the business to begin making the first steps.
If you struggle with justifying it to the stakeholders that the company needs to upgrade its data stack, or if you already have a soft buy-in but are unsure how to best proceed -- I hope you'd find this post helpful and insightful.
The introduction is gentle both from the engineering standpoint and from the grounds of convincing the business to begin making the first steps.
If you struggle with justifying it to the stakeholders that the company needs to upgrade its data stack, or if you already have a soft buy-in but are unsure how to best proceed -- I hope you'd find this post helpful and insightful.
Substack
Event Sourcing V1
Gentle introduction to the Event Sourcing paradigm.
Yours truly designing Uber. Good to be on the other end from time to time, many thanks to Tom for interviewing me.
Also, I gotta work on being less confusing. Although think the conversation went quite well.
Also, I gotta work on being less confusing. Although think the conversation went quite well.
YouTube
Uber system design: mock interview walk-through with Dima Korolev (ex-Google)
GET 1-to-1 COACHING for system design interviews: https://igotanoffer.com/en/interview-coaching/type/system-design-interview?utm_source=Youtube&utm_medium=eng-video&utm_campaign=dima-uber
Today's mock interview: "Design Uber".
Our special guest candidate:…
Today's mock interview: "Design Uber".
Our special guest candidate:…
👍3❤1
https://open.substack.com/pub/dimakorolev/p/the-oltp-language -- collected my thoughts on how can we accelerate the retirement of SQL as the meant to implement OLTP transactions.
Dima Korolev
The OLTP Language
Time to deprecate SQL as the means to implement business logic.
👍2
Forwarded from Not boring, and a bit of a condescending prick
Just clicked "Publish" on two posts that were in the works for well over a month. Behold:
• Distributed Stateful Workflows, and
• Stateful Orchestration Engines.
Hope you enjoy reading them as much as I enjoyed writing them. And spread the word!
• Distributed Stateful Workflows, and
• Stateful Orchestration Engines.
Hope you enjoy reading them as much as I enjoyed writing them. And spread the word!
Substack
Distributed Stateful Workflows
The theory and the concepts behind modern-day distributed software.
❤4
The episode on Stateful Orchestration is out!
In reality this episode sets the table for proper introduction into the domain of stateful orchestration. I talk in detail about many SysDesign problems leading to formulating the problem, from the idea of serialization and vector clocks, through decoupling of producers and consumers, and all the way to Cadence and Temporal as generic mechanisms to enable SAGAs and more sophisticated patterns at scale.
In reality this episode sets the table for proper introduction into the domain of stateful orchestration. I talk in detail about many SysDesign problems leading to formulating the problem, from the idea of serialization and vector clocks, through decoupling of producers and consumers, and all the way to Cadence and Temporal as generic mechanisms to enable SAGAs and more sophisticated patterns at scale.
YouTube
Stateful Orchestration :: SysDesign Meetup :: 2024-Sep-21
Resuming after a long pause. The topic is Stateful Orchestration, although in reality I just talk through a lot of modern-day SysDesign concepts to set the table for the episodes to come!
Errata sheet: Temporal itself is open source, released under the MIT…
Errata sheet: Temporal itself is open source, released under the MIT…
👍6
Durable Execution, a.k.a. Stateful Orchestration 2.0.
The closing meetup episode of 2024 is now released.
https://tinyurl.com/sdm-durable-execution
https://tinyurl.com/sdm-durable-execution-slides
Would be great to build this together in 2025 and beyond!
The closing meetup episode of 2024 is now released.
https://tinyurl.com/sdm-durable-execution
https://tinyurl.com/sdm-durable-execution-slides
Would be great to build this together in 2025 and beyond!
🔥3❤1❤🔥1
Folks, Maxim Fateev, the founder of Temporal has a super discount code —
It's literally 80% off. Definitely good enough in my book to share with the community. Hope to see you there!
CSDM80 — for those of us who want to attend Replay 2025, https://replay.temporal.io/.It's literally 80% off. Definitely good enough in my book to share with the community. Hope to see you there!
🔥1
A very good catalogue, via @vanadium23.
https://muratbuffalo.blogspot.com/2021/02/foundational-distributed-systems-papers.html
https://muratbuffalo.blogspot.com/2021/02/foundational-distributed-systems-papers.html
Blogspot
Foundational distributed systems papers
I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the d...
❤3🔥2
Realistically, it looks like we'd do well to just build an open-source comparison tool for a few most popular cluster (i.e. decentralized) databases.
The list so far is quite large and growing:
⒈ FoundationDB
⒉ Yugabyte
⒊ CockroachDB
⒋ YDB
The test may well be the trivial "Alice pays Bob, no over-spending" app.
The cluster DB should run on three/five/seven nodes, an odd number.
Client nodes would be sending traffic either to all nodes at random, or to just one of them.
Failing nodes should also be tested, as well as slow/deteriorating network.
All can be done with docker-compose.
Invaluable experience, useful insights, open source code, a meetup episode or two of making this happen. All the good stuff.
Best way to build this, I think, would be to have think wrappers for each DB to a) start it, and b) provide some unified API layer for the above operations, so that each DB node is one-to-one paired with a thin API node.
So that the tester code can be independent of what DB is being used. Including sending the requests, verifying correctness of responses, measuring throughput/latency, and turning off the nodes and/or slowing down traffic between them at random.
Who's excited?
The list so far is quite large and growing:
⒈ FoundationDB
⒉ Yugabyte
⒊ CockroachDB
⒋ YDB
The test may well be the trivial "Alice pays Bob, no over-spending" app.
The cluster DB should run on three/five/seven nodes, an odd number.
Client nodes would be sending traffic either to all nodes at random, or to just one of them.
Failing nodes should also be tested, as well as slow/deteriorating network.
All can be done with docker-compose.
Invaluable experience, useful insights, open source code, a meetup episode or two of making this happen. All the good stuff.
Best way to build this, I think, would be to have think wrappers for each DB to a) start it, and b) provide some unified API layer for the above operations, so that each DB node is one-to-one paired with a thin API node.
So that the tester code can be independent of what DB is being used. Including sending the requests, verifying correctness of responses, measuring throughput/latency, and turning off the nodes and/or slowing down traffic between them at random.
Who's excited?
👍3
The lightning talks event from this week's SysDesignMeetup is out: https://tinyurl.com/sdm-lightningtalks-feb2025
It has three lightning talks.
• By David Archuleta, on Wasm and perfomant code in both the browser and node.js,
• By me, on Source-of-Truth, and how it relates to "learning to count", and
• By Alex Kantsevoi, on using
I'd say it's good content, although I'm still getting used to using the Mac and recording the screen from it. Enjoy!
It has three lightning talks.
• By David Archuleta, on Wasm and perfomant code in both the browser and node.js,
• By me, on Source-of-Truth, and how it relates to "learning to count", and
• By Alex Kantsevoi, on using
etcd to perform leader elections and achieve consensus.I'd say it's good content, although I'm still getting used to using the Mac and recording the screen from it. Enjoy!
YouTube
Lightning Talks :: SysDesign Meetup :: 2025-Feb-20
Today's episode is three lightning talks.
• By David Archuleta, on Wasm and perfomant code in both the browser and node.js,
• By Dima Korolev, on Source-of-Truth, and how it relates to "learning to count", and
• By Alex Kantsevoi, on using `etcd` to perform…
• By David Archuleta, on Wasm and perfomant code in both the browser and node.js,
• By Dima Korolev, on Source-of-Truth, and how it relates to "learning to count", and
• By Alex Kantsevoi, on using `etcd` to perform…
👍5🔥2
I normally don't share upcoming off-the-record events of the SysDesignMeetup, but we had a break of a few months, and thus some extra visibility is warranted.
This Saturday. If you'd like to join, vote for the best time slot with a Slack reacc here.
To get to Slack, https://tinyurl.com/sdm-slack-invite should still work.
More details and links at http://github.com/sysdesignmeetup/sdm.
Hope to see you soon!
This Saturday. If you'd like to join, vote for the best time slot with a Slack reacc here.
To get to Slack, https://tinyurl.com/sdm-slack-invite should still work.
More details and links at http://github.com/sysdesignmeetup/sdm.
Hope to see you soon!
❤2