Whoa, I asked Claude to show me state income tax rates in 2015, 2020, and 2025. Was expecting to see tax hikes, but there's far more states that actually lowered their taxes.
A pleasant surprise indeed!
A pleasant surprise indeed!
A friend is hiring a Staff for Uber @ Amsterdam, for the intersection of AI and securiry, on the harness / guardrails side I assume. Super interesting.
https://www.uber.com/global/en/careers/list/159033
Happy to help arrange the call if I worked with you before and/or know your work via one handshake.
https://www.uber.com/global/en/careers/list/159033
Happy to help arrange the call if I worked with you before and/or know your work via one handshake.
❤1
66.88%. 80.1%. 85%. 90.79%. 93%. 100%.
These are all SOTA scores on agentic memory benchmarks. None of them tell you whether the system will work in production.
The deeper problem isn't the data — it's that we often misunderstand what these numbers actually measure. In our recent whitepaper we open-sourced datasets that target specific memory functions. Today we published a follow-up that explains why we think the well-known agentic memory benchmarks (LoCoMo, LongMemEval) miss the mark for production systems, and what we measure instead.
https://xmemory.ai/chasing-sota-in-ai-memory/
We're in a field that is measuring itself against itself.
The real question is not “are we beating last week's leaderboard?” It's “are we building something that makes people's work meaningfully better?”
That's harder to measure. It's also the only thing that matters.
These are all SOTA scores on agentic memory benchmarks. None of them tell you whether the system will work in production.
The deeper problem isn't the data — it's that we often misunderstand what these numbers actually measure. In our recent whitepaper we open-sourced datasets that target specific memory functions. Today we published a follow-up that explains why we think the well-known agentic memory benchmarks (LoCoMo, LongMemEval) miss the mark for production systems, and what we measure instead.
https://xmemory.ai/chasing-sota-in-ai-memory/
We're in a field that is measuring itself against itself.
The real question is not “are we beating last week's leaderboard?” It's “are we building something that makes people's work meaningfully better?”
That's harder to measure. It's also the only thing that matters.
xmemory Website
Chasing AI memory SOTA: Beating the Benchmark, Missing the Point
Why agentic memory benchmark numbers can be noisy, and what we should measure instead.
Given how insecure our systems are proving to be, perhaps it’s time to admit a custom-build Web3-signed commands processor is safer than ssh?
Just run a small in-house blockchain spanning the nodes one needs to access to, have a few whitelisted public keys so that the holder of their private counterparts can issue shell commands and read all the output — and voila, zero-day Linux vulnerabilities are gone; or at least the attack surface shrinks ~100x.
Why aren’t big security players such as VPN providers offering this service already?
Just run a small in-house blockchain spanning the nodes one needs to access to, have a few whitelisted public keys so that the holder of their private counterparts can issue shell commands and read all the output — and voila, zero-day Linux vulnerabilities are gone; or at least the attack surface shrinks ~100x.
Why aren’t big security players such as VPN providers offering this service already?
🔥1
Speaking of prompt optimization techniques in particular and of slop in general:
Jury instructions are frequently not understood. Studies dating back to the 1970s and continuing through today consistently find that jurors comprehend only somewhere between 40% and 70% of standard jury instructions, even after hearing them read aloud. A linguistics expert at Northeastern, Sally Randall, found that comprehension was significantly worse when instructions contained passive voice, presupposed information, and legal jargon. A more recent finding is striking: the complexity of legal concepts, rather than the complexity of language, is the primary cause of difficulties with comprehension, meaning that even rewriting instructions in plain English only modestly improves things, because the underlying concepts (mens rea, proximate cause, scienter, reasonable doubt as distinct from "beyond a shadow of a doubt") are genuinely hard.
Jury instructions are frequently not understood. Studies dating back to the 1970s and continuing through today consistently find that jurors comprehend only somewhere between 40% and 70% of standard jury instructions, even after hearing them read aloud. A linguistics expert at Northeastern, Sally Randall, found that comprehension was significantly worse when instructions contained passive voice, presupposed information, and legal jargon. A more recent finding is striking: the complexity of legal concepts, rather than the complexity of language, is the primary cause of difficulties with comprehension, meaning that even rewriting instructions in plain English only modestly improves things, because the underlying concepts (mens rea, proximate cause, scienter, reasonable doubt as distinct from "beyond a shadow of a doubt") are genuinely hard.
👍6
Engineers aren't tech leads of agent swarms. They're founders of agent tribes.
Three thoughts that turned out to be the same thought.
I. Constitutions, not values.
Any moral opinion becomes a political stance the moment you scale it across millions of imperfect people over centuries. Politics is the friction you get when a moral idea meets a hundred million people and a hundred years.
At the other end of that spectrum sit constitutions: not statements of values, but the harness that lets imperfect people carry values across generations without burning the place down. A system of checks designed so that one bad decision — even one the majority loves — can't ruin the whole country.
That reframes how I think about moral opinions. Not "is this right?", but "what kind of system does this turn into when you scale it across imperfect people over a century, and does that system survive?"
A rule that only works when good people enforce it isn't a rule. It's a wish.
II. Agentic systems are the same problem, on a faster clock.
Engineers using Claude Code, Codex, and the rest aren't dispatching tickets to direct reports. We're designing meta-harnesses — constitutions — that turn a goal into coordinated agent work whose sum is greater than its parts.
The failure modes rhyme exactly with failed states: anarchy on one end, brittle over-centralized rigidity on the other. The job is to find the middle.
Today's tools are early tribal chiefs: brilliant while the chief is competent, fragile the moment the chief is replaced. They aren't broken — they're wonderful — but they're tied to the particular engineer running them. Agents are improving too fast for that to be the long-term answer.
III. Writing the right words is literally wealth creation.
The U.S. founding fathers' artifact was a set of words. Give them a few percent share of 250 years of American GDP and you'll easily land at "the largest value creators in the history of humankind."
Mechanically, they sat in rooms and argued about phrasing.
That's the skill the best engineers now need. Not prompting by hand. Not shaving tokens. Writing the words that turn capable-but-aimless agents into something that creates lasting value.
Every harness rule should express clean intent. The intent should cut deep — into what's actually being built and why. The system around the rules should ask questions when intent is unclear, and accumulate understanding instead of starting fresh every time.
Conclusions.
The right metric isn't tokens saved. It's universality — how much of your company's intent the harness can absorb and carry forward without you in the loop on every decision.
Tokens are cheap and getting cheaper. Feedback loops have collapsed from days to minutes. Tweaking prompts to save a few hundred tokens is debating the font of the parchment.
The work that matters is constitutional. Saving 50% of tokens means nothing compared to a harness that talks to your whole company nonstop, learns the shape of your organization within weeks, and keeps producing the work indefinitely — as a productive team member that carries your vision through time.
The founding fathers would understand.
Full post: https://dimakorolev.substack.com/p/founding-fathers-of-agents
Three thoughts that turned out to be the same thought.
I. Constitutions, not values.
Any moral opinion becomes a political stance the moment you scale it across millions of imperfect people over centuries. Politics is the friction you get when a moral idea meets a hundred million people and a hundred years.
At the other end of that spectrum sit constitutions: not statements of values, but the harness that lets imperfect people carry values across generations without burning the place down. A system of checks designed so that one bad decision — even one the majority loves — can't ruin the whole country.
That reframes how I think about moral opinions. Not "is this right?", but "what kind of system does this turn into when you scale it across imperfect people over a century, and does that system survive?"
A rule that only works when good people enforce it isn't a rule. It's a wish.
II. Agentic systems are the same problem, on a faster clock.
Engineers using Claude Code, Codex, and the rest aren't dispatching tickets to direct reports. We're designing meta-harnesses — constitutions — that turn a goal into coordinated agent work whose sum is greater than its parts.
The failure modes rhyme exactly with failed states: anarchy on one end, brittle over-centralized rigidity on the other. The job is to find the middle.
Today's tools are early tribal chiefs: brilliant while the chief is competent, fragile the moment the chief is replaced. They aren't broken — they're wonderful — but they're tied to the particular engineer running them. Agents are improving too fast for that to be the long-term answer.
III. Writing the right words is literally wealth creation.
The U.S. founding fathers' artifact was a set of words. Give them a few percent share of 250 years of American GDP and you'll easily land at "the largest value creators in the history of humankind."
Mechanically, they sat in rooms and argued about phrasing.
That's the skill the best engineers now need. Not prompting by hand. Not shaving tokens. Writing the words that turn capable-but-aimless agents into something that creates lasting value.
Every harness rule should express clean intent. The intent should cut deep — into what's actually being built and why. The system around the rules should ask questions when intent is unclear, and accumulate understanding instead of starting fresh every time.
Conclusions.
The right metric isn't tokens saved. It's universality — how much of your company's intent the harness can absorb and carry forward without you in the loop on every decision.
Tokens are cheap and getting cheaper. Feedback loops have collapsed from days to minutes. Tweaking prompts to save a few hundred tokens is debating the font of the parchment.
The work that matters is constitutional. Saving 50% of tokens means nothing compared to a harness that talks to your whole company nonstop, learns the shape of your organization within weeks, and keeps producing the work indefinitely — as a productive team member that carries your vision through time.
The founding fathers would understand.
Full post: https://dimakorolev.substack.com/p/founding-fathers-of-agents
Substack
Founding Fathers of Agents
Three thoughts I kept coming back to, until I realized they were the same thought.
🔥4
As we've definitely entered the era of smart AI agents, I think it's time we go back to the roots with shell scripting.
Say no to long shell scripts. Say no to complex business logic in shell scripts. Instead, this logic belongs in plain English!
Replace one script with ten options by ten shallow do-one-thing-and-do-it-well scripts. Or fifteen, if some of them have modes and start / stop / check functionality.
Then write a plain English file, likely markdown, outlining for the agent what these scripts are, how to use them, and what to keep in mind. A human may also read this English file, but who are we kidding?
And then invoke these scripts via a decent AI agent, such as Cursor. In your workflow, as needed, it will present a concise summary, with tables where needed.
If uncertain about the output, just ask it again, in plain English. And if some script output needs proper visualization, that's definitely not the job of the script itself, but of its outer harness.
And this outer harness is the human operator and an AI model. A model so small that it'll be running locally in just one or two generations of our laptops.
What a time to be alive.
Say no to long shell scripts. Say no to complex business logic in shell scripts. Instead, this logic belongs in plain English!
Replace one script with ten options by ten shallow do-one-thing-and-do-it-well scripts. Or fifteen, if some of them have modes and start / stop / check functionality.
Then write a plain English file, likely markdown, outlining for the agent what these scripts are, how to use them, and what to keep in mind. A human may also read this English file, but who are we kidding?
And then invoke these scripts via a decent AI agent, such as Cursor. In your workflow, as needed, it will present a concise summary, with tables where needed.
If uncertain about the output, just ask it again, in plain English. And if some script output needs proper visualization, that's definitely not the job of the script itself, but of its outer harness.
And this outer harness is the human operator and an AI model. A model so small that it'll be running locally in just one or two generations of our laptops.
What a time to be alive.
👍6
The Short Interval Between Two Eras
Here's the post I forgot to write a few weeks ago, right after watching Project Hail Mary.
(The book is excellent too.)
One distinct aftertaste of the movie is how precisely the short interval between the book and the screenplay coincided with the short interval between the pre-AI and post-AI adoption era.
Just a few years ago, computer interfaces were presumed to be incredibly difficult to manage. And computer programming — a.k.a. software engineering — was widely understood to be a difficult discipline.
Andy Weir is famously a "science maximalist": his books are ripe with the idea that The Science has solutions to all problems. Yet, in The Martian, I cannot help but wonder how it is possible that a botanist navigates the intricacies of a complex software system so flawlessly.
Reading Project Hail Mary, I found myself thinking along the same lines. Being good at first-principles physics does not automatically make one proficient with bleeding-edge tech — which, inevitably, is what an interstellar spacecraft is full of.
And yet! By the time the movie is released, we know for a fact how powerful human-first, natural language interfaces can be.
Quite literally just five years ago, the thought that an interface could be both powerful and intuitive was unimaginable. Sure, sci-fi authors had been talking about this possibility forever. But the tech community — yours truly included — was rather sceptical.
AI-assisted coding has changed this in a matter of single-digit years. An astonishing product progress, if you ask me.
Moreover, Neuralink et al. is no longer something impossible. Muscle memory is tricky — it'll be a while until one can wake up like Neo in The Matrix, knowing Kung Fu after a short session.
But for intellectual tasks — for virtually all of them — the problem can largely be declared solved. To my taste, if one can understand the domain, articulate the desired outcome, and answer a few clarifying questions, then even today's AI models are quite capable of making things happen at astronomical scale. Literally astronomical: tasks such as planning extraterrestrial space travel are a piece of cake for a mid-sized AI model equipped with just a few tools.
What a time to be alive (c)
Here's the post I forgot to write a few weeks ago, right after watching Project Hail Mary.
(The book is excellent too.)
One distinct aftertaste of the movie is how precisely the short interval between the book and the screenplay coincided with the short interval between the pre-AI and post-AI adoption era.
Just a few years ago, computer interfaces were presumed to be incredibly difficult to manage. And computer programming — a.k.a. software engineering — was widely understood to be a difficult discipline.
Andy Weir is famously a "science maximalist": his books are ripe with the idea that The Science has solutions to all problems. Yet, in The Martian, I cannot help but wonder how it is possible that a botanist navigates the intricacies of a complex software system so flawlessly.
Reading Project Hail Mary, I found myself thinking along the same lines. Being good at first-principles physics does not automatically make one proficient with bleeding-edge tech — which, inevitably, is what an interstellar spacecraft is full of.
And yet! By the time the movie is released, we know for a fact how powerful human-first, natural language interfaces can be.
Quite literally just five years ago, the thought that an interface could be both powerful and intuitive was unimaginable. Sure, sci-fi authors had been talking about this possibility forever. But the tech community — yours truly included — was rather sceptical.
AI-assisted coding has changed this in a matter of single-digit years. An astonishing product progress, if you ask me.
Moreover, Neuralink et al. is no longer something impossible. Muscle memory is tricky — it'll be a while until one can wake up like Neo in The Matrix, knowing Kung Fu after a short session.
But for intellectual tasks — for virtually all of them — the problem can largely be declared solved. To my taste, if one can understand the domain, articulate the desired outcome, and answer a few clarifying questions, then even today's AI models are quite capable of making things happen at astronomical scale. Literally astronomical: tasks such as planning extraterrestrial space travel are a piece of cake for a mid-sized AI model equipped with just a few tools.
What a time to be alive (c)
🔥5🥰2
Important question: is it true that LLM models are better than other LLM models when it comes to suggesting how to optimize prompts for themselves?
Say I have a process which involves querying the model again and again. And as the good ML/AI citizen I am, I have labeled data — journaled and marked-down records of where the model performed well, and where it might need improvement.
Fine-tuning and post-training the model are of course plausible directions. But that's slow and expensive. A much cheaper alternative is tweaking the prompts.
So I have a process to incrementally run some end-to-end test. Or, in pure SGD terms, I have a process to incrementally approximate some gradient stochastically.
As the result of this process I get a suggestion on what can be improved in the prompts, so that the result gets Pareto-better. Presumably. So: check and repeat.
I also use other models to cross-check those improvement suggestions, to make sure they don't overfit to the very problem — since it's often generalizing a particular failure mode from just a few examples.
This is plausible. This works. This is cost-effective.
But one thing doesn't let me sleep well at night. We have this belief that model A is better at improving prompts for model A. What is this belief based on? Is it even true?
It sounds wise to use Anthropic models to improve prompts that are later fed to Anthropic models. Replace "Anthropic" with any LLM provider here. But do we know whether it's true at all?
Perhaps the next big thing is some Grok training a model that can prompt some Qwen better? Asking for a friend, of course.
Seen in this light, all in all, the idea of a proper bench of models and model ensembles in a closed-loop system starts to look more and more lucrative.
Say I have a process which involves querying the model again and again. And as the good ML/AI citizen I am, I have labeled data — journaled and marked-down records of where the model performed well, and where it might need improvement.
Fine-tuning and post-training the model are of course plausible directions. But that's slow and expensive. A much cheaper alternative is tweaking the prompts.
So I have a process to incrementally run some end-to-end test. Or, in pure SGD terms, I have a process to incrementally approximate some gradient stochastically.
As the result of this process I get a suggestion on what can be improved in the prompts, so that the result gets Pareto-better. Presumably. So: check and repeat.
I also use other models to cross-check those improvement suggestions, to make sure they don't overfit to the very problem — since it's often generalizing a particular failure mode from just a few examples.
This is plausible. This works. This is cost-effective.
But one thing doesn't let me sleep well at night. We have this belief that model A is better at improving prompts for model A. What is this belief based on? Is it even true?
It sounds wise to use Anthropic models to improve prompts that are later fed to Anthropic models. Replace "Anthropic" with any LLM provider here. But do we know whether it's true at all?
Perhaps the next big thing is some Grok training a model that can prompt some Qwen better? Asking for a friend, of course.
Seen in this light, all in all, the idea of a proper bench of models and model ensembles in a closed-loop system starts to look more and more lucrative.
Forwarded from SysDesign Meetup
Hey folks,
So in the SDM community we love and respect Jepsen (jepsen.io), and a friend of mine is looking for help to kickstart Krab Arena (krabarena.org).
It's a similar game with three major differences:
1) The goal is to actually collect valuable head-to-head compasisons that help with stack choices (i.e. PostgreSQL beats MongoDB at X, Y, Z, here's a repo to prove it).
2) It's human-moderated, at least at the beginning, so real and hypothetical stories that have to do with actual choices are more than welcome (i.e. "last time I chose Open Policy Agent over Zanzibar because ...)
3) It's extremely AI-friendly, as in, the ideas are what counts most as long as they are reproducible, and if all the code is AI-first, it's fine — since the system is meant to be self-converging.
Personally, I believe in this, and am willing to help myself. A little bit with my own time — as a co-founder of an AI startup I'm oversubscribed at the moment — but more so in the capacity of people connector.
Bottom line is, if this insterests you in any way, shape, or form, that is enough to commit a few hours — do reach out!
Ty!
So in the SDM community we love and respect Jepsen (jepsen.io), and a friend of mine is looking for help to kickstart Krab Arena (krabarena.org).
It's a similar game with three major differences:
1) The goal is to actually collect valuable head-to-head compasisons that help with stack choices (i.e. PostgreSQL beats MongoDB at X, Y, Z, here's a repo to prove it).
2) It's human-moderated, at least at the beginning, so real and hypothetical stories that have to do with actual choices are more than welcome (i.e. "last time I chose Open Policy Agent over Zanzibar because ...)
3) It's extremely AI-friendly, as in, the ideas are what counts most as long as they are reproducible, and if all the code is AI-first, it's fine — since the system is meant to be self-converging.
Personally, I believe in this, and am willing to help myself. A little bit with my own time — as a co-founder of an AI startup I'm oversubscribed at the moment — but more so in the capacity of people connector.
Bottom line is, if this insterests you in any way, shape, or form, that is enough to commit a few hours — do reach out!
Ty!
Unpopular opinion: We should not shame and blame GitHub too much.
The product was meant to be used by humans. It's now used as part of automation pipelines left, right, and center instead.
Repos, commits, users, lines-of-code, secrets, action run counts — all through the roof. Definitely some four-sigma beyond projected growth.
Honestly, as an executive at GitHub, I'd say it's "a good problem to have" that there are constant outages.
There's no good alternative, after all. Whether we like it or not, GitHub actions became a de-facto standard, so that even agents use them.
While I can easily imagine an agentic-first GitHub replacement raising hundreds of millions of dollars today (wink wink), the reality is that it will not replace GitHub any time soon.
And, quite frankly, the interruptions are all intermittent. If your company's development process is stalled for some ~three hours some ~twice a month — way above the current pace at which GitHub is down — it's still not a good enough business reason to justify transition.
Personally — and I keep saying this for a while! — some S3-based cross-cloud storage plus a Web3-grade orchestration layer is the future. May well be open source future, to be honest.
Most large companies have storage and compute paid for already, GitHub already charges a large premium on top of these two, and the main GitHub lockdown factor — that humans are used to its UI/UX — is just irrelevant for agentic-first workflows.
So some #StatefulCompute for exactly-once rules enforcement of gates, and some time-bounded #MeteredCompute to run what used to be GitHub actions — and we're done with GitHub for good. That is, in a sane universe that does optimize for cost-effectiveness.
In the real world that we are living in, no medium and large company is interested in cost effectiveness, so I guess we're stuck with the good old GitHub. Personally, I like the product, and have no complaints whatsoever.
The product was meant to be used by humans. It's now used as part of automation pipelines left, right, and center instead.
Repos, commits, users, lines-of-code, secrets, action run counts — all through the roof. Definitely some four-sigma beyond projected growth.
Honestly, as an executive at GitHub, I'd say it's "a good problem to have" that there are constant outages.
There's no good alternative, after all. Whether we like it or not, GitHub actions became a de-facto standard, so that even agents use them.
While I can easily imagine an agentic-first GitHub replacement raising hundreds of millions of dollars today (wink wink), the reality is that it will not replace GitHub any time soon.
And, quite frankly, the interruptions are all intermittent. If your company's development process is stalled for some ~three hours some ~twice a month — way above the current pace at which GitHub is down — it's still not a good enough business reason to justify transition.
Personally — and I keep saying this for a while! — some S3-based cross-cloud storage plus a Web3-grade orchestration layer is the future. May well be open source future, to be honest.
Most large companies have storage and compute paid for already, GitHub already charges a large premium on top of these two, and the main GitHub lockdown factor — that humans are used to its UI/UX — is just irrelevant for agentic-first workflows.
So some #StatefulCompute for exactly-once rules enforcement of gates, and some time-bounded #MeteredCompute to run what used to be GitHub actions — and we're done with GitHub for good. That is, in a sane universe that does optimize for cost-effectiveness.
In the real world that we are living in, no medium and large company is interested in cost effectiveness, so I guess we're stuck with the good old GitHub. Personally, I like the product, and have no complaints whatsoever.
👍3
Teaching the AI to Watch My UI Tests
While there are more topics in the backlog, and more drafts, today I want to share something big for me personally.
I've reached a milestone along another dimension: instructing the AI agent how to fix or improve the UI based on the automatically recorded video of a UI test.
TL;DR: I've instrumented everything with a UI test. In fact, it's not just a UI test — it does some things outside the UI, and I have a special hidden-from-the-user UI page that's used to show some console CLI terminal commands that are run, and their results. Then I programmatically record the run as a 10+ minute video that shows everything.
There's even a visible moving mouse pointer in the UI test btw; simulated, of course. But it makes it far more pleasant to follow.
We watch this video then: I myself, my colleagues, other people whose input we ask for. Then we gather feedback, such as "at this minute mark, this and that are not as they should be".
And then I can literally ask the AI agent to analyze the video. Yes, I know it doesn't have to be video, but so far it's working fine as is, to my great surprise and satisfaction!
Now, to close the loop: collect feedback, paste it, then watch the agent make the fix. The agent then produces the next video recording that passes all the tests. Rinse and repeat.
Agents can also diff videos. So I have MacOS and Ubuntu generating them, and then have the agents compare notes. (The simulated mouse movement on the video is so sloppy on Ubuntu compared to MacOS, heh.)
I think this is quite big. Not only am I quite happy to have pioneered UI testing in the project where I've been touching the UI for a long time — though that too.
The more important thing is that this AI-assisted coding, UI engineering, or rather product UX engineering, becomes a much easier and much more pleasant problem to have. Now literally all you need to ship amazing products is a good understanding of your customer and good taste.
To geeks reading me: Yes, I'm in the process of having my harness generate a sequence of screenshots, such that every frame of the video contains the index of the screenshot once that scene is complete. Because analyzing the video burns tokens and is far too slow overall. But watching that "mouse" navigate around my UX is mesmerizing.
While there are more topics in the backlog, and more drafts, today I want to share something big for me personally.
I've reached a milestone along another dimension: instructing the AI agent how to fix or improve the UI based on the automatically recorded video of a UI test.
TL;DR: I've instrumented everything with a UI test. In fact, it's not just a UI test — it does some things outside the UI, and I have a special hidden-from-the-user UI page that's used to show some console CLI terminal commands that are run, and their results. Then I programmatically record the run as a 10+ minute video that shows everything.
There's even a visible moving mouse pointer in the UI test btw; simulated, of course. But it makes it far more pleasant to follow.
We watch this video then: I myself, my colleagues, other people whose input we ask for. Then we gather feedback, such as "at this minute mark, this and that are not as they should be".
And then I can literally ask the AI agent to analyze the video. Yes, I know it doesn't have to be video, but so far it's working fine as is, to my great surprise and satisfaction!
Now, to close the loop: collect feedback, paste it, then watch the agent make the fix. The agent then produces the next video recording that passes all the tests. Rinse and repeat.
Agents can also diff videos. So I have MacOS and Ubuntu generating them, and then have the agents compare notes. (The simulated mouse movement on the video is so sloppy on Ubuntu compared to MacOS, heh.)
I think this is quite big. Not only am I quite happy to have pioneered UI testing in the project where I've been touching the UI for a long time — though that too.
The more important thing is that this AI-assisted coding, UI engineering, or rather product UX engineering, becomes a much easier and much more pleasant problem to have. Now literally all you need to ship amazing products is a good understanding of your customer and good taste.
To geeks reading me: Yes, I'm in the process of having my harness generate a sequence of screenshots, such that every frame of the video contains the index of the screenshot once that scene is complete. Because analyzing the video burns tokens and is far too slow overall. But watching that "mouse" navigate around my UX is mesmerizing.
🔥8👍3
After some basic cleanup, the largest two files on my MacBook are ~/Library/Application Support/Cursor/User/globalStorage/state.vscdb and ~/Library/Application Support/Claude/vm_bundles/claudevm.bundle, in this order.
Each one almost 8GB.
Each one almost 8GB.
😱2
This AI-assisted coding boom will teach many if not most engineers that being a tech lead is too damn hard.
On the other hand, engineering teams used to produce code much slower than modern-day swarms of agents.
I have a vague idea that the future is some pre-trained / pre-prompted "chat windows", or "ready-to-join conference rooms", where our freshly minted human tech leads can chat or talk to the AI about the vey code that needs to be understood and evolved further.
So that the human part of the review process will mostly be some "Do I understand it correctly that ...?" questions, with various agents working together to produce meaningful answers from different angles.
And we will need harness for this tooling too. What a time.
On the other hand, engineering teams used to produce code much slower than modern-day swarms of agents.
I have a vague idea that the future is some pre-trained / pre-prompted "chat windows", or "ready-to-join conference rooms", where our freshly minted human tech leads can chat or talk to the AI about the vey code that needs to be understood and evolved further.
So that the human part of the review process will mostly be some "Do I understand it correctly that ...?" questions, with various agents working together to produce meaningful answers from different angles.
And we will need harness for this tooling too. What a time.
❤2👍2
How to make AI remember correctly, beyond RAGs and context windows.
Live in San Francisco.
Tue, Jun 2, 11:30am.
At the Snowflake conference, Braindate Lounge — Basecamp, South Hall A.
Join: https://snowflakesummit26.braindate.com/braindates/295744
Live in San Francisco.
Tue, Jun 2, 11:30am.
At the Snowflake conference, Braindate Lounge — Basecamp, South Hall A.
Join: https://snowflakesummit26.braindate.com/braindates/295744
Braindate
Log in
🔥3
This media is not supported in your browser
VIEW IN TELEGRAM
This OneSix robot is so so cool!
I said sure, make a video, but it's just for fun, not to share. But then I watched it and it's so, so cool!
(The recording is 2x-ed by me before posting.)
I said sure, make a video, but it's just for fun, not to share. But then I watched it and it's so, so cool!
(The recording is 2x-ed by me before posting.)
❤4🔥2
Software Was Always Going to Break First
I had a thought I can't shake.
AI took over coding fast, and everyone has the obvious reasons ready: you can check whether code works almost instantly, and there's real money in shipping faster.
But here's the part nobody says out loud. Of all the industries the world runs on, software is the one that already tolerated slop. It was human slop — slower to build, more expensive, harder to maintain — but slop all the same.
The moment outsourcing took off, the industry quietly admitted something: it didn't really care about engineering quality. A great staff engineer or principal architect is worth more than a solid senior one, sure. But not 100x more. Usually 1.5x, maybe 2–3x at the bleeding-edge shops. And the closer a company is to ordinary, the smaller that gap gets.
Climbing past "senior" was rarely about engineering anyway. It was about "understanding business priorities," "managing stakeholders," "driving cross-team collaboration." In other words: the exact bureaucratic slop AI is now great at producing.
So what happens when an industry that is fundamentally about engineering decides to reward everything except engineering? Slop. Tech debt growing faster than features ship. A slow-motion disaster.
This was always heading somewhere bad. AI just sped up the arrival.
Some fields are better protected. In Michael Church's essay Engineers as Clerks, he points to three career ladders where raw intelligence matters most: law, medicine, and software.
Law and medicine did something software never did — they unionized, in the economic sense. Over decades they organized into something hard to crack from the outside. They keep their below-average members employed, and collectively the whole profession's pie grew. The rules of both fields are basically built to keep clever outsiders out.
(You could say theoretical physics fits too. But it's not really a career path, and I like Eric Weinstein too much to wade in there.)
By the same logic, electricians and plumbers might be more exposed to robots than people think. If an automation-friendly fuse box is cheaper and works better, young, fast-growing economies — Dubai, Argentina, El Salvador — have every reason to skip straight to it and win big.
Heavy industry like aerospace is probably safer, mostly because it's drowning in regulation. After Boeing, that's not nothing. AI will arrive there too, just slower — and slower likely means safer.
But in software — the world of bits, as Thiel calls it — the steamroller keeps going. There's no external standard forcing quality up. Entry-level jobs will change fast, because big companies do understand they need to grow talent and keep the market stable. The real problem is the middle: managers who learned nothing but soft skills on their way toward VP. They're exactly the wrong people to be guarding the gate.
Remember all the stories about US universities failing at basic student safety? That's institutional capture at its ugliest, and it's doing real damage. I think software's version of that capture will hurt us far more.
Because software runs on high switching costs. Imagine if you could actually pick up your Facebook or Apple account and move it somewhere else. There's no regulation for that, and there won't be one soon.
So realistically, we wait out a generation. The younger one has its own tools and devices. They won't need Gmail and the rest, and I hope they grow up fluent in encryption and privacy.
No tidy conclusion here. I'll stay exactly who I am: cautiously optimistic about the long run, pretty pessimistic about the short one.
I had a thought I can't shake.
AI took over coding fast, and everyone has the obvious reasons ready: you can check whether code works almost instantly, and there's real money in shipping faster.
But here's the part nobody says out loud. Of all the industries the world runs on, software is the one that already tolerated slop. It was human slop — slower to build, more expensive, harder to maintain — but slop all the same.
The moment outsourcing took off, the industry quietly admitted something: it didn't really care about engineering quality. A great staff engineer or principal architect is worth more than a solid senior one, sure. But not 100x more. Usually 1.5x, maybe 2–3x at the bleeding-edge shops. And the closer a company is to ordinary, the smaller that gap gets.
Climbing past "senior" was rarely about engineering anyway. It was about "understanding business priorities," "managing stakeholders," "driving cross-team collaboration." In other words: the exact bureaucratic slop AI is now great at producing.
So what happens when an industry that is fundamentally about engineering decides to reward everything except engineering? Slop. Tech debt growing faster than features ship. A slow-motion disaster.
This was always heading somewhere bad. AI just sped up the arrival.
Some fields are better protected. In Michael Church's essay Engineers as Clerks, he points to three career ladders where raw intelligence matters most: law, medicine, and software.
Law and medicine did something software never did — they unionized, in the economic sense. Over decades they organized into something hard to crack from the outside. They keep their below-average members employed, and collectively the whole profession's pie grew. The rules of both fields are basically built to keep clever outsiders out.
(You could say theoretical physics fits too. But it's not really a career path, and I like Eric Weinstein too much to wade in there.)
By the same logic, electricians and plumbers might be more exposed to robots than people think. If an automation-friendly fuse box is cheaper and works better, young, fast-growing economies — Dubai, Argentina, El Salvador — have every reason to skip straight to it and win big.
Heavy industry like aerospace is probably safer, mostly because it's drowning in regulation. After Boeing, that's not nothing. AI will arrive there too, just slower — and slower likely means safer.
But in software — the world of bits, as Thiel calls it — the steamroller keeps going. There's no external standard forcing quality up. Entry-level jobs will change fast, because big companies do understand they need to grow talent and keep the market stable. The real problem is the middle: managers who learned nothing but soft skills on their way toward VP. They're exactly the wrong people to be guarding the gate.
Remember all the stories about US universities failing at basic student safety? That's institutional capture at its ugliest, and it's doing real damage. I think software's version of that capture will hurt us far more.
Because software runs on high switching costs. Imagine if you could actually pick up your Facebook or Apple account and move it somewhere else. There's no regulation for that, and there won't be one soon.
So realistically, we wait out a generation. The younger one has its own tools and devices. They won't need Gmail and the rest, and I hope they grow up fluent in encryption and privacy.
No tidy conclusion here. I'll stay exactly who I am: cautiously optimistic about the long run, pretty pessimistic about the short one.
❤3👍1🔥1
[Un]popular opinion: if a human asks the AI to research something, and this research implies accessing some website, the AI should absolutely ignore robots.txt and visit that website right away.
Even if the very site says "for humans only, no AI".
Because the intent comes from a human.
Sure, internet browsing and serving content costs money. So we may well end up in a sane world where "for humans only" just means "free for humans, but one millicent for the AI".
Better yet, "free for humans, one millicent for the AI, ten millicents to return as a structured JSON object without ads". The Web3 folks have designed literally zero-friction abstractions for such payments a couple years back already.
I, for one, would be delighted to live in the world where some ~120% of hosting prices are covered by agents browsing my website. The money will ultimately be coming from users who are authorizing this research.
A sane world that would be. A sane world indeed.
Ah, and ads can and should be gone for good by that point. That is, ads for humans. The emerging strong AI will render them useless anyway.
Ads are, after all, prompt injections in disguise, and we'll need far fewer AI generations to reach herd immunity compared to how many human generations it is taking now.
Expect a bumpy road though: websites that are deliberately making money by luring your agents in. On the other hand we sort of have and are having this with humans already, so we know how to fight this.
Our AI assistants will be entering there teenage year and internet puberty in the next half a year or so. That'd be fun to watch.
Even if the very site says "for humans only, no AI".
Because the intent comes from a human.
Sure, internet browsing and serving content costs money. So we may well end up in a sane world where "for humans only" just means "free for humans, but one millicent for the AI".
Better yet, "free for humans, one millicent for the AI, ten millicents to return as a structured JSON object without ads". The Web3 folks have designed literally zero-friction abstractions for such payments a couple years back already.
I, for one, would be delighted to live in the world where some ~120% of hosting prices are covered by agents browsing my website. The money will ultimately be coming from users who are authorizing this research.
A sane world that would be. A sane world indeed.
Ah, and ads can and should be gone for good by that point. That is, ads for humans. The emerging strong AI will render them useless anyway.
Ads are, after all, prompt injections in disguise, and we'll need far fewer AI generations to reach herd immunity compared to how many human generations it is taking now.
Expect a bumpy road though: websites that are deliberately making money by luring your agents in. On the other hand we sort of have and are having this with humans already, so we know how to fight this.
Our AI assistants will be entering there teenage year and internet puberty in the next half a year or so. That'd be fun to watch.
👍4