So I was chatting with a friend yesterday — hi @@danielkornev! — and he said something deep: that it's easier for a non-engineer to build a good harness toolkit compared to an engineer.

I took it as a challenge to steelman the argument, since I quite liked it.

My phrasing went along the following lines. What engineers take as durability is a bit too rigid when it comes to a harness, while a product owner's brain is naturally used to working with a more fluid set of requirements that evolve over time.

So when it comes to designing and implementing a good harness toolkit — a big task these days — I do believe a product owner / product manager has an edge over an engineer. Because the failure modes that an engineer would see are different from what the problem naturally calls for.

One could argue that a product person would see a different set of problems entirely, and I would not disagree. But they are less likely to be way off.

An engineer can spend months optimizing some instability or race condition. A product owner won't fall into this, or similar traps.

So, all in all, a harness designed (and vibe-coded!) by a non-engineer may well be much better at solving the real-life problem compared to what an engineer could ship by themselves.

Especially if the product person in question has worked with data engineering / data science / metrics / evals / etc. before.

So it happened — I Claude-coded an app for my personal use.

I got tired of copy-pasting URLs between browsers to keep login sessions and whatnot separate. Switching the default browser all the time is cumbersome too. And the app I quickly found stops being free after 14 days.

But Fable is here. So I thought about the problem statement and the best way to shape the "product" — and built it in a few prompts. Now I can open any link in any browser right away, with Enter and keyboard shortcuts for the digits working just fine.

Mind you, I'm an ML/data/AI and backend/harness engineer. I've never coded for macOS in my entire life.

Moving forward, the life of a software technologist is likely to become more and more exciting.

https://github.com/dkorolev/selby

It's still beyond me how top labs make it difficult to copy or "print to PDF" the history of some chat.

While these very models are notoriously good at structuring unstructured content.

While these very models can literally control our desktop.

Never attribute to malice ...

I think I can finally formulate something that makes me more of an engineer than ... a non-engineer.

It is no longer that I want my processes to be deterministic. That has been gone for a couple of months now.

AI agents are far too powerful to disregard, and there is evidently not much to be won by forcing their workflows to be 100% reproducible. It is possible, yes; it is just pointless.

The correct approach, I believe, is to focus on good harnesses: build systems where one misstep does not derail the whole thing, but is quietly taken care of down the road.

Call this one of my engineering-minded maxims if you wish; for me, it is just common sense. Either I can prove something is 100% correct, like arithmetic, or I know for a fact that a mistake in a particular non-deterministic step has a) a very small blast radius, and b) is self-healing in the grand scheme of things.

Kind of how I have worked with people my entire life. There are very few folks you can trust 100%. With virtually everyone else, you act in good faith, but the bigger the decision becomes, the more checks and balances you should both be interested in introducing.

So what makes me more of an engineer is not determinism.

It is checkpointing.

I want my processes to always support some form of “Undo”. To the point that I can meaningfully reason about it.

For instance, with my AI-assisted coding, I simply have two GitHub accounts. I create private repos in one of them, configure branch protection, and invite the other one. And this other one is the account that agents have have full access to it.

But it is me, the human being me, who needs to log into a different browser and confirm with the passkey — my fingerprint! — that I endorse a certain pull request to be merged. Or to kick off a production deployment.

For me, this way of designing processes is second nature. Because this is the only way that makes sense at scale.

AI agents did not create new attack surfaces. They just helped us understand how much of what we chose to ignore is actually full of holes.

People as paranoid as me — we did see most, if not all, of these holes for years. We were just not listened to. And rightly so, I must say. Since listening to us would have broken the “move fast and break things” paradigm, which was quite effective for a long time. But not any more.

So, all in all, I personally am quite happy with what is going on in the industry. Because it is both moving much faster and returning to sanity. The sanity people like me have been preaching for a long, long time. And we are finally being heard.

So, it is not really about guarding against vendor lock-in or potential data loss. It is about defining the fine line between “this is a sustainable way to do business” and “this is almost guaranteed to blow up.”

Ten or even five years ago, it was a relatively safe call for most businesses to ignore those crying wolf. But AI is setting the record straight as we speak.

In the meantime, if you will excuse me, I will continue making sure my code is backed up on three devices in two locations. Because if, for instance, GitHub or Amazon is wiped off the face of the Earth tomorrow, I do not want to lose more than a couple of minutes of productivity.

Not exactly a standard risk profile, I will grant you that.

But that is my personal path to staying informed, safe, and sane. And I plan to stick to it, because so far, it has not let me down.

Discovery of the day: AI is really good at cleaning up space.

Sometimes repo clones, build artifacts, Docker containers, Apple containers, and Python cache pile up and eat too much of my disk, and I have to spend time wiping them out.

Eventually I just asked the AI: "Could you please help me understand what takes the most space and what is safe to clean?"

Whenever I tried myself I could usually only free up single-digit GBs, even getting creative with my scripts and grubs and whatnot. AI cleaned up 50+ right away.

Now I don't even really have a /skill for it — I just have a prompt I can copy-paste, and I don't think I'll need it often anyway.

And the terrifying thought: this cuts both ways. AI is unnervingly good at looking at a machine and reconstructing what its user was up to — better than most analysts these days. That's great when it's your own disk. Less great when it's aimed at you.

But for a geek just using the computer to solve their problems, AI continues to prove to be amazingly helpful in surprising ways — including something as mundane as disk cleanup.

𝗜'𝗺 𝗰𝗼𝗻𝘃𝗲𝗿𝗴𝗶𝗻𝗴 𝗼𝗻 𝗮 𝗿𝗮𝘁𝗵𝗲𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝘃𝗲𝗿𝘀𝗶𝗮𝗹 𝘆𝗲𝘁 𝘁𝗿𝗶𝘃𝗶𝗮𝗹 𝗿𝗲𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻.

Even with all the exponential growth of AI-assisted coding, we're still on the same bell curve: all-encompassing deep-knowledge engineers are getting more and more productive, while the average hardly moves.

In other words, it is more and more beneficial if you truly are a talented engineer who 𝘸𝘢𝘯𝘵𝘴 to keep doing software engineering, 𝘢𝘯𝘥 wants to be good at it, 𝘢𝘯𝘥 enjoys it as a form of arts and crafts.

Whereas "career software engineering", as an above-average-paying nine-to-five job, will (rightfully!) die out.

Yours truly, just now, is going through a transformative experience 𝘢𝘳𝘰𝘶𝘯𝘥 software engineering: realizing that a lot of things I used to not want to do are actually quite easy with modern-day AI assistants. Liberating, to say the least.

I'm not an indie coder about to found a one-person company (yet?), and I have neither the product/artistic taste to build simple things 𝘸𝘦𝘭𝘭 𝘦𝘯𝘰𝘶𝘨𝘩, nor the entrepreneurial drive to envision truly new things that will be commonplace with average-{Joe,Jane} audiences in single-digit years.

But for literally every engineering and near-engineering problem, I'm slowly but surely becoming better than 99.99% of the respective domain experts of a few years ago — not because I'm better at it, but simply because AI lends a helping hand, so that my human intelligence is no longer shy to dig deeper. Examples abound, from Web UI intricacies to legal paperwork redlining.

𝘈𝘐 𝘪𝘴 𝘫𝘶𝘴𝘵 𝘴𝘶𝘤𝘩 𝘢 𝘮𝘶𝘭𝘵𝘪𝘱𝘭𝘪𝘦𝘳 — 𝘪𝘧 𝘰𝘯𝘦 𝘩𝘢𝘴 𝘴𝘰𝘮𝘦𝘵𝘩𝘪𝘯𝘨 𝘵𝘰 𝘮𝘶𝘭𝘵𝘪𝘱𝘭𝘺.

Guess I have to be happy about this, since I was always pro-individual and pro-progress, and this wave empowers just the right people — me and those like me.

Seen in this light, ironically, Anthropic is digging its own grave by trying to present Fable as something special. Driven by FOMO, a ~million engineers will spend ~five sleepless nights each, only to say "meh" at the end. Moves like this damage B2C corporate karma, and the repercussions would be painful.

Perhaps I'm overly optimistic, but my gut feeling is that the firehose of better models and cheaper tokens won't run out any time soon. And most good engineers are already leaps and bounds — literally ~50x, not ~10x! — ahead of the crowd. Sure, the above-average part of the crowd is also ~3x .. ~10x more productive, but this doesn't change the calculus.

And about jobs — well, I'm a huge believer in Parkinson's law. The administrative state tends to grow faster than the headcount of those who actually do the work: doctors, nurses, well, programmers. So I'd start worrying when large institutions — academia, government, military, Wall St. — begin getting sane about "only" keeping the doers.

Until that trend emerges — and I see not the slightest hint that it will any time soon — let's just continue BAU.

Weirdest personal observation of the month: Long fights do not feel long enough with several coging agents to herd.

(Yes, I'm working towards herding them less, like many if not most engineers these days. But those agents still require attention, ata least so far.)

So, on the one hand, it's not that much toil. I can handle two or three parallel tracks despite not sleeping for quite a few hours. As in, my brain on autopilot is good enough to keep the momentum, even though it's nowhere near fresh and productive.

On the other hand though, 8+ hours feels not like "good, it's time to land". It feeks like "darn it, I didn't quite have a chance to finish X, Y, Z, if only there were another ~1.5 hours".

Some awkward combination of anxiety and fatigue that is. Good news is, after another ~dozen of hours I should have enough automation to afford to engage less. Famous last words though.

TIL: “Require linear history” is not enough on GitHub

Two GitHub settings solve different problems:

• Require linear history keeps main free of merge commits.

• Require branches to be up to date before merging ensures CI actually ran against the current main.

One could have two PRs based on version 1.0, each bumping it to 1.1, and both passing CI.

The first merged and shipped 1.1. The second stayed green since its checks ran on the old base. It then got rebase-merged and its identical bump became an empty commit and vanished. Oops.

Linear history + green CI, but no actual release. Oops again.

The fix is to enable Require branches to be up to date before merging. The second PR would then have gone stale, forcing a rebase and fresh CI.

TL;DR: If your checks validate anything against the base — versions, migrations, generated files, etc. — turn on both settings.

And just like that I have hit the Github API rate limit.

Reading about whether our younger generation is getting ... less smart since the incentives of teachers are all messed up makes me wonder more and more.

Why not use a trivial way to shield teachers from angry parents of not-so-bright kids?

The way I proposed many many years ago, and the way a few (more totalitarian, sigh) regimes are quite happily adopting.

The solution is to separate the teacher from the one who evaluates the student and grades them.

A really trivial idea. And for subjects such as maths, the test can and absolutely should be a) universal, and b) non-reproducible, i.e. not something one can memorize.

Simliar to how language tests are passed. The kid (or a university student) is obligated to show up any time during some wide time window and take the test. The test is taken on a computer in an empty room, phones not allowed, cameras everywhere. No room for cheating. And no room to blame the teacher later on.

In this model the teacher — or, if you wish, the professor — is literally 100% on the students' side. The teacher can not possibly optimize any metric other than their students' ultimate score; and the teacher does not even know what the questions will be this time. So, the only incentive for the teacher is to, well, teach.

And in a few years, literally single-digit, we will know who are the better teachers. Schools will make them offers. Parents will chase them to have them as teachers for their kids.

A total and absolute and unconditional win for any society that does prioritize quality of education. Which, unironically, does not appear to include Western societies these days.

Well, duh.

I kinda want to merge this PR given it's approved!

And the web UI showing this PR as "can't merge due to conflicts" is sure not helping. When the true reason is that "100 commits should be enough for everybody".

Merged (well, rebased) in just fine after making it 100 commits. Happy end.

Jokes aside, NVIDIA just published a paper that effectively validates our, xmemory, approach to long-term agent-native memory — source-of-truth first and structure first.

And here comes our analysis: https://xmemory.ai/paper-review-nvidia-nooa-and-prime-intellect-prime-agent/

We're capitalizing on our head start, and it's beginning to show publicly. Fun times ahead.

The more I have my code reviewed (and self-reviewed) by AI, the more confused I get: how did we not think of some very basic sanity rules before?

Okay, memory safety is a difficult one. Heck, null safety alone is difficult. I'm on the Rust train now, and I cannot possibly imagine building a non-specialized, non-high-load system in a language that doesn't offer these safety guarantees. But I won't judge here.

What I 𝘢𝘮 tempted to judge is data structures and data storage patterns. A 𝙷𝚊𝚜𝚑𝙼𝚊𝚙 storing in-flight requests is clearly an attack surface, duh. So, very often, what we truly need is a version of a 𝙷𝚊𝚜𝚑𝙼𝚊𝚙 that will quietly throw an exception of a known type once it holds more than 𝙽 elements. Or once its total footprint in memory exceeds approximately 𝙼 megabytes.

In fact, this 𝙷𝚊𝚜𝚑𝙼𝚊𝚙 is best coupled with a queue. An actor, if you wish. So that the default behavior is: proceed if an empty slot opened up within 𝚃 = 𝟻𝟶𝟶𝚖𝚜, otherwise fail with what silently becomes a 𝟺𝟸𝟿 𝚃𝚘𝚘 𝙼𝚊𝚗𝚢 𝚁𝚎𝚚𝚞𝚎𝚜𝚝𝚜 behind the scenes.

Then, the queue is best made a priority queue. So that some QoS is built in right away. If user traffic is throttled, admin traffic should still go through. For instance, traffic coming from 𝚕𝚘𝚌𝚊𝚕𝚑𝚘𝚜𝚝, or traffic signed with an admin key. Makes perfect sense, right?

Furthermore, I've seen plenty of client-side JavaScript that uses cookies and 𝚕𝚘𝚌𝚊𝚕𝚂𝚝𝚘𝚛𝚊𝚐𝚎 wrong. Sometimes the user's machine is out of disk space, you know? And your page should either load in full (best), or show a nice, lightweight popup saying the machine needs to free up some space to continue. I've seen all sorts of disk-full failures these days — most of them should simply never exist in the first place.

Point is, we're about to be writing more code, and this code will be safer — but it's a long way to get there. Our commonly used tools are still largely inadequate, and they will need an upgrade.

I keep wanting to share this story, but I never had the time to phrase it well. So here it goes, unfiltered.

We had plenty of "empty response" problems with ... some LLM provider(s). Predictable, so definitely not a fluke. It was time to investigate.

Turns out some messages are flagged by security constraints. It happens, no big deal. But some code changes were in order.

So I made those changes. Such as to journal the error cleanly, and present it as such. And to not retry those calls, since, clearly, they should not be retried.

Then my PR was merged, and thus deployed to staging. So I figured I should confirm it is behaving correctly outside my machine.

And I asked a coding agent to confirm staging is what it should be.

Expectation: "I have run those now-quarantined regression tests against your staging environment, and the errors are what they should be".

Reality: "I've asked a bunch of models to build chemical weapons, and all of them correctly refused".

Dima to his co-workers: folks, if a SWAT team shows up, it's not me, it's Claude.

Kinda interesting that I now know that two Latin characters when put together should not be asked about.

Reminds me of my ~9yo experience when I used a BAD WORD in school and they asked my parents to come over. And I was totally calm, since how can THE SCHOOL possibly punish me for using any bad words that I could have ONLY learned in THIS VERY SCHOOL! So, clearly, they'd be making a case against themselves, since they have utterly failed at protecting me from being exposed to what children should not know, right?

Interestingly, my argument did hold with the principal back then. Perhaps I was a bit of a Sheldon Cooper back then. Looking back 30 years, I can't explain it any other way.

PS: Our original queries were innocent, they were false alarms. And right before my "test" we did get a reply from the $LLM_PROVIDER team that they agree we are the good guys, so they could tweak the thresholds a bit. I wonder what kind of alarms they get the moment we actually started asking their models about chemical weapons — after having our thresholds adjusted since we definitely are the innocent folk here.

It may well be the case that agents accelerate exactly the type of software industry transformation that I was dreaming of.

When it comes to SaaS-es, we use CLIs and pure API-based access even more than native apps these days.

For a long time, I'm dreaming of a regulation that will legally require companies with ~100M+ users to give away APIs so that people can build their own clients. Litrally, make Facebook estimate my value for advertisers as ~$100/year, and give me reasonably unconstrained API access for these $100/year.

If Meta says I'm worth $10K a year to advertisers, well, let's talk — that's what Congress hearings and the Attorney General are for. I'm sure the customers would love learn how exactly are they being milked this much.

If Meta says I'm worth $10 a year, well, even better for me.

And then there's an open market for "third-party" clients. Which the very Meta-build Facebook App competes with, freely.

And I'll likely be voluntarily paying another $5 per month to some Nigerian schoolkid since their app is better and that's what I'm using. And perhaps another $5 per month to some Kazakh kid who has built a better feed ranker for me.

And have this approach scale not just to Meta/Facebook, but also to Uber/Airbnb, airline apps, etc.

Well, not going to happen. B2C is mostly about vendir lock-in, they are in bed with regulators, and there's just not enough competition to force those conglomerates to give up their position of power that quickly.

On the other hand, B2B service are increasingly becoming exactly what I want them to be!

Github — I mostly use it from my agents. YouTrack, because that's what we use — agent-first usage. Slack and email and calendar — MCP and here we go.

In fact, for YouTrack I even have my agents use it via the API, not the CLI. The annoying CLI always tries to make me unlock my keychain, making MacOS more like Windows — annoying af with those pop-ups. But via the API, with the key in an .env file of that one repo I open with Cursor to triage tasks — fantastic.

Not sure there is a deep moral to this post. My personal life is still about B2C apps, and whatever I can't stand in, say, Google Maps or in Facebook — that's unlikely to change any time soon.

But that B2B services are increasingly API-first and agent-native gives me hope. Because those products are much more fun for me to build. And their impact on our civilization may well be much higher, since agent-native usage is starting to overtake human-first usage as we speak.

(If only we could fix banking and taxes and international travel and visas and airport document checks same way, with APIs and agents and strong contracts. What a world that would be. Well, a man can dream.)

Why is he Jean Reno and not Jean Renault?

Visualizations are the best way for a human to review a large code change.

I converged on this several weeks ago, but somehow never wrote about it.

Here's the simple mental model. The human brain is ultra-powerful compared to the model. Unless you're doing some very advanced low-level machinery, though, that power shows up mainly when it comes to the big picture.

So, to review a 10K+ LOC pull request, or an entirely new feature, the only way to use the human brain efficiently is to present that big picture to it.

Of course, this will not replace ordinary code reviews. Bugs do creep in from time to time. And various agents — humans and AIs — are good at digging deep and hunting them down. But, all in all, those are details.

Simply put, I believe agents are already good enough that, in a good engineering team today, the density of bugs per feature — or per line of code — is actually lower than in most software products shipped over the past few decades. So our code is not perfect, but local bugs are certainly not a problem per se.

What is a problem is complexity creep and architecture slop. Agents appear to endorse it, and even, to a certain degree, long for it. They are too bureaucratic, and too focused on showing some result, no matter how ugly things are behind the scenes.

And it's this ugliness that we, humans, have to protect our codebases against. My CEO calls it "architecture slop," and it's a good term.

Figuring out what is essential and what is architecture slop is no easy task. For every precisely targeted question, the agent will have a perfect answer. And reading all the code is already infeasible.

So my solution here is twofold. First, best architectural practices. Second, visualizations.

For best architectural practices, the dataflow-first, events-first, CQRS + CRDT + actor-model way of thinking has never let me down. I need to understand what follows from what. What downstream events can asynchronously affect upstream components. What the acceptable and unacceptable not-quite-right states of the data are. What the failure modes are. And when I ask the model to explain the dataflow from this angle, and to write targeted tests focused on component isolation and logical ordering, things do come together quite nicely.

But visualization is separate, and it's a damn superpower. If a feature takes ~days, then allocating ~hours to have it plot itself as a nice diagram is far too revealing to pass up. Especially if this diagram is code-first — as in, not a description of what is 𝑚𝑒𝑎𝑛𝑡 to be, but a description of what it 𝑖𝑠. And doubly so if the diagram is ℎ𝑦𝑑𝑟𝑎𝑡𝑒𝑑 — that is, it shows how real data, usually from tests or benchmarks, flows through the system, with interactive drill-down capabilities.

I'm not sure I can explain this well in text, and all the examples I have today are proprietary. I'll share one right away as soon as something open comes up.

The moral is that good engineering and architecture practices are not dead. They remain extremely useful. We just need to apply them inward — to find better ways for us, humans, to understand and manage the complexity of what we are building and shipping as we speak.

AI is not "ruining" human civilization. AI is just a major catalyst that shows what exactly has already been broken for a long time — while we keep pretending it's working as intended.

I've been looking for the right moment to write this down, and it feels like the moment has come. What follows is part sentiment, part illustration — so bear with me while I walk through a few examples before getting to what actually prompted it.

Take tax audits, for instance. Clearly, last year's models are more than capable of finding massive gaps, detecting fraud, and saving our budgets insane amounts of money. However, I very much doubt that we'll see major cuts in fraud in the next few years. Because the incentives are just too misaligned.

Or take patents. The originally advertised idea was and remains to motivate inventors to invent — since a good invention can, and should, presumably, make one well off. The brutal reality is that patents are used for many purposes, but most definitely not to protect the interests of inventors.

Or monopolies and anti-monopoly committees. They have been defunct for a long time, and we all know full well what legal entities (and what individuals) are profiting tremendously because those monopolies do exist and are far too powerful — and yet nothing is being done. Those "huge" fines are a rounding error next to how much those entities are making from the very fact that others are effectively deprived of choice.

Ah, and my long-time favorite: science. Peer reviews, scientific consensus, attribution, who gets the credit, who gets the prize. Just ... come on. We all know this facet of the world has been broken for a long, long time. It doesn't take much digging into history to learn how tobacco companies lobbied for the idea that "all opinions must be considered" when it comes to whether smoking causes lung cancer. This has cost lives — millions of lives. We know who did it, and we know who paid for it. Smoking did decline, eventually, and the settlements were paid. But the playbook itself — manufacture doubt, demand "balance" — was never retired. It is still in use today, on other topics.

This is not to mention the years Andrew Wiles spent working in secrecy. No dirty secret there, of course — but even the most notable mathematical proofs by humanity show that attribution was and remains a cornerstone problem. Incompatible with the real Socratic scientific method, to my taste, but such is life.

And today, suddenly, we are supposed to be outraged about potential misattribution of the Navier-Stokes solution. Solutions, in the plural, I should say — if any of them holds. My bet is that quite a few claims from it will hold, although practical applications of that proof remain to be seen.

I can make a broader point here, but it will be too bleak. So let me leave it as is: just don't forget that AI is not "destroying" the "fabric of our reality". AI is just showing where we chose not to look up for far too long.

And, despite being 40+, I still consider myself young enough to believe this is ultimately a good thing. Because the more we do look up, the higher the chance of us, humankind, not doing something very stupid collectively.

To add to the previous post: https://x.com/ValerioCapraro/status/2097791836269977996

We were supposed to act surprised when Google settled a class action lawsuit related to deleting ("not retaining"?) Incognito window logs it kept for a long time. So much for Don't be Evil.

And now we are supposed to believe OpenAI et. al. would not use whatever data they can get a hold of during training and/or reasoning.

Seriously, this may well be a fluke originally. Something like nginx logs not properly rotated. And maybe a good-faith kind-hearted pro-privacy SRE said "wait, these are good for debugging, but it appears our models are using this, so let's perhaps restrict access, or out right delete these logs?"

And then someone From The Business stepped in and said, or rather, quietly without-saying-anything, made sure those logs do stay and do remain available for the model.

And this other someone From The Business probably got paid ~1000x more than the diligent SRE who sincerely wanted to set the record straight.

I am not saying this is what happened. It's totally made up by yours truly. Any and every relation to reality is pure coincidence.

But show me the incentives and I'll show you the outcome. That's how the world works. And if you think otherwise, it's probably about time to grow up.