We really need some torrent-style worldwide Docker and Ubuntu mesh network of distribution.

Just unthinkable how much traffic is wasted daily by moving the very same bits over and over again.

Not intending to start a holy war [probably], but I'm finding myself comparing programming languages to one another again.

To set the tables straight, I'm talking about backend development, where high throughput, low latency, and high reliability are essential.

In addition to the above, I am at the stage of my career where "design for refactoring" is a big theme. So just "we can hack this up quickly" and/or "we can make it work super-quickly in just a week of work" are not strong enough arguments, because what would the company do after you've shipped it and left?

The above immediately disqualifies languages such as JavaScript and Python. Too slow, and pretty much impossible to refactor, or even maintain.

The pros and cons of the "standard" languages are sort of well known. Let me summarize them quickly; again, this is not intending to start a pointless discussion, and I do not claim to be unbiased, just setting the tables to actually ask what I want to ask about.

• C++. Pros: The code would be effective, and, if the team is strong and C++-focused, the code would be maintainable, and relatively error-free. Cons: You pretty much have to be a C++ company, or, at least, you need to have a task that is so niche and so performance-critical, that it's okay for this particular piece to be done in C++.

• Java and other JVM languages, most notably Kotlin. Pros: Easy to find people who speak the language. Cons: I personally am terrified by a) code bloat, b) garbage collection that kills latency at high percentiles, and c) slow warm-ups, so that autoscaling becomes a pain in and of itself.

• C# and the .NET family. Pros: About the same as Java, plus, personally, I'm more comfortable with .NET runtime vs. the JVM one, even on Linux. Cons: Same as with JVM.

Now, the question. In the setup above, how should one gauge Rust and Golang?

My view is rather straightforward so far.

• Rust: Still young. Unclear if it delivers enough to win over C++; i.e., even if support- and refactoring-wise it turns out to be better than C++ for "simple" services in the long run, Rust people are still rather difficult to find.

• Golang. Here, I'm on the fence. There's a large crowd of people who would argue Go is unfit for large projects, mostly due to lack of proper type safety. There's also a large crowd of people who would argue Go is quite efficient for its simple syntax.

Ultimately, my question then is: should Go be taken seriously in 2023?

Is it worth my time to learn it and [help] build a production service in it?

Or would this knowledge be rather useless, given I speak C++ well, and for "simple" low-latency high-throughput gRPC services, it likely is and will remain a better choice?

If the latter is true, I should not really invest much into Golang, and wait for an opportune moment to educate myself on Rust. If the former is true, an opportune moment is now.

Advice welcome -- and many thanks in advance!

In the meantime, we've recorded another educational interview episode, thanks Jos!

Folks, any experience with the "best practice" of different `USER`s in Docker?

Via this link -- https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user -- the recommended way is:

RUN groupadd -r postgres && useradd --no-log-init -r -g postgres postgres

I struggle to understand:

a) Why change the user at all? Isn't Docker secure enough so that "just" using the default, "root", user is good enough for most intents and purposes?

b) If some service would rather not be run as root -- which I admire -- why not build everything from under root from an AS builder container, and then only change the user into some app from within the latest, runner, container?

c) Why would the Docker team build this feature in such a way that works very differently on Linux and on macOS, so that cross-religion dev teams have to struggle?

d) On a related note, why doesn't the Linux binary of Docker offer the host.network.internal hostname by default?

Edit: To clarify, postgres is actually a great example of an application that I believe should be run from under its own user! But when it comes to building & running custom Golang apps, for instance, my original question holds.

Edit 2, via @Komzpa: root is not safe because root is really something like if (!uid) {} in many paths in kernel. if you manage to create a device file in your namespace you can remount it and there's nothing you can do about it as you're root. All the security enclosures only work if you're not root, there's no "root of this namespace".

"Dima, I worked in many companies that were at risk of downsizing and/or closing down. The lesson I thus learned early on is to open source more, sooner."

Put together some thoughts about Docker[files and Make[files]: https://dimakorolev.substack.com/p/docker-and-make

If the topic gains interest, we might even consider a meetup episode about it. A tad too hands on and low-level for a SysDesign episode, but if it helps us, The Engineers -- so be it.

Our SysDesign guide with interviewing.io is out!

A lot of work, hope it pays off. The content is really freaking awesome there, easily the best material on the subject available out there, and likely for quite some time.

Enjoy responsibly!

Wrote a piece with some ideas on the long term future of Schema Registries.

https://dimakorolev.substack.com/p/beyond-the-schema-registry

Depending on a number of factors (mostly on whether you agree with my views), this can be a vision roadmap for the future or a long-ish rant about the sad state of the affairs.

But it's something I'm passionate about, so public these thoughts go. Enjoy responsibly!

A follow-up to my previous Substack post.

It's not really about schema registries and API contracts. It's about cutting slack from our software development practices.

And, as Jos points out too, the easiest way to cut this slack is to stop viewing "software development processes" as necessary guardrails, but to critically examine each and every one of them. Switching to "approve" rather than "need more proof" by default is a heuristic he suggests. It's a good one, because, in my experience, good high-level engineers do tend to be the people who ask for forgiveness, not permission.

My view is probably even bolder. Demand cost-benefit analysis for each "best practice", and aggressively cut the ones that are a) not on the critical path, and b) do not deliver the benefits that we should demand given the costs.

In fact, I'm quite comfortable making an even stronger statement.

We have over-stigmatized "ninja engineers" in favor of "team players" who are perfectly fungible. And these "perfectly fungible" engineers, in fact, do get the job done. But the pace is 10x slower, while the rate of missteps is ... well, most certainly not lower.

So it's time we take a sober look and get back to basics. No company would depend on one or two key people who know everything; that's too low of a bus factor. But three key people plus another five solid performers is _probably_ a far better solution for many, if not most, tasks.

We do, after all, have successful large products used by millions, and even billions, of people, and maintained by just a few dozen. That's the spirit to get back to.

Docker is deleting open source orgs.

Can't believe the Docker folks didn't realize name-squatting would be a real issue in the very immediate future. At least, freeze those names ffs. Terrible move, IMHO.

During the golden times of open source someone would have built a Torrent plugin for those free images. So that Docker would not be able to plausibly claim they are cutting costs, while the developer community would keep thriving, albeit with somewhat slower docker pull times.

H/T @Max_Losev for the link.

I asked my colleagues today, in all seriousness, and now I want to ask you.

Does StackOverflow improve your effective coding speed?

In my case, the answer is definitively no.

Because if I need a quick answer, I'll find it without StackOverflow, and thus the value added by SO is negligibly low.

And if there's a deep questoin to ponder on that is bugging me, StackOverflow is just a time sink. Sure, there is gold to be found there, especially in the comments, but I would need to literally sift through tens of questions that are described with approximately the same words, but are totally unrelated to what I'm struggling with.

(Those questions I would need to sift through are also, quite often, much easier, if not outright trivial. To the degree that it makes me wonder why do people even need comment threads where the answer is literally something along the lines of chmod -x or --no-cache or export this or "flip the order of these two command line arguments".)

A notable exception would be when it really helps to know some industry-standard two-words or three-words term for the problem that I need to solve. Or for a similar problem that I do not need to solve, because it's a totally different one. But then we are not talking about the positive value of SO, but rather about making the value of SO less negative.

Am I the only one who's having these thoughts from time to time, or does this curse affect many experienced engineers?

Any OS / kernel experts here?

Could you fill me in on io_uring and [e]BPF when it comes to application-level networking?

On the one hand, I'm inclined to say that gRPC/protobuf would be notoriously slow here, while, if trivial batching + deserialization happens at the kernel level, we'll get far higher throughput, together with lower latency.

On the other hand, I'd assume problems that are "RPC-bound" do not use gRPC these days already, or would that be a wrong assumption?

As someone who, to a decent degree, identifies as a [high-performance] infrastructure engineer, in addition to being a [realtime] data architect, I'm very curious what are the modern-day bleeding edge problems in the field.

Also, maybe there are use cases for the above in the space of blockchains?

Benchmarking Production Systems.

A new @SysDesignMeetup episode is out.

There is much more to load testing than just figuring out how many requests can a service handle per second.

Video: https://tinyurl.com/sdm-benchmarking
Slides: https://tinyurl.com/sdm-benchmarking-slides

Our guide is out in full: https://interviewing.io/guides/system-design-interview/part-four

Part four is probably more advanced than others (i.e. is more for seniors++ than for mid-levels), but it’s a good piece overall, imho, likely best prep material available online.

Enjoy responsibly :-)

Riding the wave further, the SysDesign Meetup event, dedicated to interviewing.io's SysDesign guide is happening soon!

San Francisco, USA Fri, 7 Apr 2023 at 11:00 PDT
New York, USA Fri, 7 Apr 2023 at 14:00 EDT
London, United Kingdom Fri, 7 Apr 2023 at 19:00 BST
Amsterdam, Netherlands Fri, 7 Apr 2023 at 20:00 CEST
Tbilisi, Georgia Fri, 7 Apr 2023 at 22:00 GET
Kuala Lumpur, Malaysia Sat, 8 Apr 2023 at 02:00 MYT

Pls join #iio-sd-guide on our Slack if you're interested, and please ask your questions in Slido, regardless of whether you plan to be there live. And here's a Google Calendar link.

Very much looking forward to!

A semi-crazy idea: a decentralized mesh blockchain run on the hardware of pet lovers, powered by their pets.

Say, you have a hamster, or a few to officially indulge the use of the pronoun of the year. You feed them well, and treat them well; please stop reading here if it's not the case.

Now, you'd love your hamsters to play and you get them a running wheel.

Here's the thing: at this point, modern technology can run a proper blockchain that would only require less computational power than your hamster generates.

First of all, this already is a wonderful plot, because we're saving trees and overall care for the environment.

Second, there are tons of cases for blockchain that are general-purpose, and yet widely under-utilized.

My favorite example would be a blockchain Twitter. Everybody can post a short message, and everybody else can read this message. This short message can be encrypted, so that only those with the key can decipher it. And the original author can publish the key later, granting the public access to whatever they have written before, with accurate timestamps in the past.

Imagine a journalist reporting from a dangerous area. The area is so dangerous that they can be lost in action there, while their material may well be of supreme importance. So, it would be nice to be able to report right from that area, so that only the select few people could read their report live, but, later on, these notes could be made public.

And all of this may well happen over a mesh network powered by hamster lovers in that area and across the globe. With no external power supply, and with no global Internet available.

And everyone can use this Twitter. For free. The "price" of posting one Tweet is measured in the computational power you have contributed to running this mesh network, and its cost is roughly the amount of energy a few hamsters generate by spinning their wheel for several minutes.

Ah, and this Twitter would be immutable. So nobody, including Elon Musk himself, would be able to delete content from there.

The very idea is infeasible, of course. For far too many reasons, and for most intents and purposes. But it's a cute idea regardless, isn't it?

Sometimes, if I catch myself having sad thoughts about the state of IT, I remind myself that for literally every historical period of human existence I would be worse off.

I love employing critical thinking to solve problems, and I was born into an ordinary family. With these inputs, thus, looking around, nothing to complain about, really.

Then the thought goes on, to the point where I have to admit that I do believe, and almost assume, that in the future life would be even better. So that if a Dima born in 2100 would look back to 2023, he would be quite upset about what the 2023 Dima has to put up with.

This is, of course, normal; that is what is called progress, and this, too, is nothing to be unhappy about.

When I look into the past I feel terrible just thinking about how much crap would I have to put up with in order to just not be burned alive, regardless of how much evidence I can present to support my views..

The next thought is to wonder whether this trend can, and/or will, at a certain point, stop or reverse.

And here it gets really sad, because it seriously looks like it may well be true that the 90s and the 2000s were a better time for us, first-principled technologists.

Of course, we did not have powerful mobile devices / mobile internet, and the concept of working from home was largely non-existent. And, of course, we had corporate structures that are a lot more "traditional", in a sense that acquiring real-life professional skills and growing one's career was substantially more difficult back then than it is today.

But disrupting it was also much easier. Or at least so it seems.

Now, looking back at the risk of being burned alive, this facet of problems was virtually non-existent in the 90s and the 2000s, but today -- just look around. There's no shortage of topics one is supposed to either not have the "wrong" opinion on, or explicitly not touch in any conversation, unless with close friends.

And this trend, that has been brewing for some fifteen years by now, begins to touch on the topics that have direct impact on what the companies declare to be their values. To the point that major executives literally admit that they have to employ Orwellian double-think in order to stay sane while being effective at their jobs.

Just to make it clear: I'm fine, I got used to this, I can see the value of PC when it comes to merely extending the idea of politeness so that a bunch of people of very diverse sets of backgrounds and values can be effective at getting things done together.

But looking at companies that claimed to be "a big family" with their employees just a few years ago, and are now too shy to pre-announce upcoming layoffs internally, effectively keeping everyone in the guessing mode day after day ... that's just wrong. And this openly contradicts the declared values of most of these companies, and yet this contradiction is supposed to be something that all of us just turn a blind eye to.

Kind of surreal to watch, if you ask me.

A lot of ideas that float around with respect to AGI Alignment seem to converge to a rather trivial dichotomy:

Do you want to help people better achieve their goals / live happier lives as they themselves would like to? (1)

Or:

Do you believe people, en masse, are not good enough at making decisions for themselves, so that, to a certain degree, some form of "positive propaganda" would be a push for the greater good? (2)

If you would have asked me ~ten years ago, I would most definitely argue that (2) is a recipe for disaster, while (1), although quite hard, is the way to go.

Looking at the world today though, I am far less convinced.
Because even if — even if! — we believe human beings are good at deciding for themselves, and even if — even if! — we believe in some form of a well-functioning democracy, so that the will of the majority of people is what drives the society forward ... I have major questions about the sustainability of such a model in the long run.

To put it blunt: I am not positive I would definitively pick such a society, however much of a utopia it may sound, over a different society, some more similar to the 1990s / 2000s USA.

Because I am legitimately afraid such a society, however much of a utopia it might be, would not legitimately enable me to execute on all my desires. Despite the fact that, objectively, none of my desires can abuse or in any way harm the other citizens / inhabitants of such a seemingly perfect society.

Not sure what to make of the above thought. But to me it does, at the very least, serve as good reminder that we are nowhere near being able to postulate the AGI Alignment problem well enough. Because, with a perfect AI or with no AI, we, humans, are having major trouble deciding for ourselves what our lives should be in the long run.

Go figure ¯\_(ツ)_/¯

Blockchain and Web3: just released this new episode!

• https://tinyurl.com/sdm-blockchain
• https://tinyurl.com/sdm-blockchain-slides

The history and the state of the art of blockchain and Web3 from first principles.

https://medium.com/miro-engineering/how-miro-leverages-open-policy-agent-to-implement-authorization-as-a-service-763f08469e5 — Peter, our Staff Engineer, wrote an in-depth blog post about what we are working on at Miro these days.