Thought of the day.
In the age of AI, it's the LLM tokens that are expensive.
Compute, as in EC2 or ECS or Hetzner, is merely collateral damage. Nobody cares about those costs as long as the LLM tokens are burned with high utility.
AI did to compute what compute did to storage.
Which also means there's tons of money to be made in compute in the years to come — much like there's tons of money to be made in storage as of the past 5+ years.
My bet is the consistency and durability is what will sell well. Both with storage, as of a few years already. And with compute, which is starting about now, since compute becomes the fungible auxiliary unit next to LLM tokens utilized at scale.
In the age of AI, it's the LLM tokens that are expensive.
Compute, as in EC2 or ECS or Hetzner, is merely collateral damage. Nobody cares about those costs as long as the LLM tokens are burned with high utility.
AI did to compute what compute did to storage.
Which also means there's tons of money to be made in compute in the years to come — much like there's tons of money to be made in storage as of the past 5+ years.
My bet is the consistency and durability is what will sell well. Both with storage, as of a few years already. And with compute, which is starting about now, since compute becomes the fungible auxiliary unit next to LLM tokens utilized at scale.
👍4
Weird how Elon says he [co-]founded OpenAI because he wanted a counter to Google’s approach to AI: closed, private, for-profit.
And as of now the largest benefactor of the ongoing trial is indeed Google — which did soften up since, but all of the original arguments against it by Musk clearly hold true.
And as of now the largest benefactor of the ongoing trial is indeed Google — which did soften up since, but all of the original arguments against it by Musk clearly hold true.
👍3
Whoa, I asked Claude to show me state income tax rates in 2015, 2020, and 2025. Was expecting to see tax hikes, but there's far more states that actually lowered their taxes.
A pleasant surprise indeed!
A pleasant surprise indeed!
A friend is hiring a Staff for Uber @ Amsterdam, for the intersection of AI and securiry, on the harness / guardrails side I assume. Super interesting.
https://www.uber.com/global/en/careers/list/159033
Happy to help arrange the call if I worked with you before and/or know your work via one handshake.
https://www.uber.com/global/en/careers/list/159033
Happy to help arrange the call if I worked with you before and/or know your work via one handshake.
❤1
66.88%. 80.1%. 85%. 90.79%. 93%. 100%.
These are all SOTA scores on agentic memory benchmarks. None of them tell you whether the system will work in production.
The deeper problem isn't the data — it's that we often misunderstand what these numbers actually measure. In our recent whitepaper we open-sourced datasets that target specific memory functions. Today we published a follow-up that explains why we think the well-known agentic memory benchmarks (LoCoMo, LongMemEval) miss the mark for production systems, and what we measure instead.
https://xmemory.ai/chasing-sota-in-ai-memory/
We're in a field that is measuring itself against itself.
The real question is not “are we beating last week's leaderboard?” It's “are we building something that makes people's work meaningfully better?”
That's harder to measure. It's also the only thing that matters.
These are all SOTA scores on agentic memory benchmarks. None of them tell you whether the system will work in production.
The deeper problem isn't the data — it's that we often misunderstand what these numbers actually measure. In our recent whitepaper we open-sourced datasets that target specific memory functions. Today we published a follow-up that explains why we think the well-known agentic memory benchmarks (LoCoMo, LongMemEval) miss the mark for production systems, and what we measure instead.
https://xmemory.ai/chasing-sota-in-ai-memory/
We're in a field that is measuring itself against itself.
The real question is not “are we beating last week's leaderboard?” It's “are we building something that makes people's work meaningfully better?”
That's harder to measure. It's also the only thing that matters.
xmemory Website
Chasing AI memory SOTA: Beating the Benchmark, Missing the Point
Why agentic memory benchmark numbers can be noisy, and what we should measure instead.
Given how insecure our systems are proving to be, perhaps it’s time to admit a custom-build Web3-signed commands processor is safer than ssh?
Just run a small in-house blockchain spanning the nodes one needs to access to, have a few whitelisted public keys so that the holder of their private counterparts can issue shell commands and read all the output — and voila, zero-day Linux vulnerabilities are gone; or at least the attack surface shrinks ~100x.
Why aren’t big security players such as VPN providers offering this service already?
Just run a small in-house blockchain spanning the nodes one needs to access to, have a few whitelisted public keys so that the holder of their private counterparts can issue shell commands and read all the output — and voila, zero-day Linux vulnerabilities are gone; or at least the attack surface shrinks ~100x.
Why aren’t big security players such as VPN providers offering this service already?
🔥1
Speaking of prompt optimization techniques in particular and of slop in general:
Jury instructions are frequently not understood. Studies dating back to the 1970s and continuing through today consistently find that jurors comprehend only somewhere between 40% and 70% of standard jury instructions, even after hearing them read aloud. A linguistics expert at Northeastern, Sally Randall, found that comprehension was significantly worse when instructions contained passive voice, presupposed information, and legal jargon. A more recent finding is striking: the complexity of legal concepts, rather than the complexity of language, is the primary cause of difficulties with comprehension, meaning that even rewriting instructions in plain English only modestly improves things, because the underlying concepts (mens rea, proximate cause, scienter, reasonable doubt as distinct from "beyond a shadow of a doubt") are genuinely hard.
Jury instructions are frequently not understood. Studies dating back to the 1970s and continuing through today consistently find that jurors comprehend only somewhere between 40% and 70% of standard jury instructions, even after hearing them read aloud. A linguistics expert at Northeastern, Sally Randall, found that comprehension was significantly worse when instructions contained passive voice, presupposed information, and legal jargon. A more recent finding is striking: the complexity of legal concepts, rather than the complexity of language, is the primary cause of difficulties with comprehension, meaning that even rewriting instructions in plain English only modestly improves things, because the underlying concepts (mens rea, proximate cause, scienter, reasonable doubt as distinct from "beyond a shadow of a doubt") are genuinely hard.
👍6
Engineers aren't tech leads of agent swarms. They're founders of agent tribes.
Three thoughts that turned out to be the same thought.
I. Constitutions, not values.
Any moral opinion becomes a political stance the moment you scale it across millions of imperfect people over centuries. Politics is the friction you get when a moral idea meets a hundred million people and a hundred years.
At the other end of that spectrum sit constitutions: not statements of values, but the harness that lets imperfect people carry values across generations without burning the place down. A system of checks designed so that one bad decision — even one the majority loves — can't ruin the whole country.
That reframes how I think about moral opinions. Not "is this right?", but "what kind of system does this turn into when you scale it across imperfect people over a century, and does that system survive?"
A rule that only works when good people enforce it isn't a rule. It's a wish.
II. Agentic systems are the same problem, on a faster clock.
Engineers using Claude Code, Codex, and the rest aren't dispatching tickets to direct reports. We're designing meta-harnesses — constitutions — that turn a goal into coordinated agent work whose sum is greater than its parts.
The failure modes rhyme exactly with failed states: anarchy on one end, brittle over-centralized rigidity on the other. The job is to find the middle.
Today's tools are early tribal chiefs: brilliant while the chief is competent, fragile the moment the chief is replaced. They aren't broken — they're wonderful — but they're tied to the particular engineer running them. Agents are improving too fast for that to be the long-term answer.
III. Writing the right words is literally wealth creation.
The U.S. founding fathers' artifact was a set of words. Give them a few percent share of 250 years of American GDP and you'll easily land at "the largest value creators in the history of humankind."
Mechanically, they sat in rooms and argued about phrasing.
That's the skill the best engineers now need. Not prompting by hand. Not shaving tokens. Writing the words that turn capable-but-aimless agents into something that creates lasting value.
Every harness rule should express clean intent. The intent should cut deep — into what's actually being built and why. The system around the rules should ask questions when intent is unclear, and accumulate understanding instead of starting fresh every time.
Conclusions.
The right metric isn't tokens saved. It's universality — how much of your company's intent the harness can absorb and carry forward without you in the loop on every decision.
Tokens are cheap and getting cheaper. Feedback loops have collapsed from days to minutes. Tweaking prompts to save a few hundred tokens is debating the font of the parchment.
The work that matters is constitutional. Saving 50% of tokens means nothing compared to a harness that talks to your whole company nonstop, learns the shape of your organization within weeks, and keeps producing the work indefinitely — as a productive team member that carries your vision through time.
The founding fathers would understand.
Full post: https://dimakorolev.substack.com/p/founding-fathers-of-agents
Three thoughts that turned out to be the same thought.
I. Constitutions, not values.
Any moral opinion becomes a political stance the moment you scale it across millions of imperfect people over centuries. Politics is the friction you get when a moral idea meets a hundred million people and a hundred years.
At the other end of that spectrum sit constitutions: not statements of values, but the harness that lets imperfect people carry values across generations without burning the place down. A system of checks designed so that one bad decision — even one the majority loves — can't ruin the whole country.
That reframes how I think about moral opinions. Not "is this right?", but "what kind of system does this turn into when you scale it across imperfect people over a century, and does that system survive?"
A rule that only works when good people enforce it isn't a rule. It's a wish.
II. Agentic systems are the same problem, on a faster clock.
Engineers using Claude Code, Codex, and the rest aren't dispatching tickets to direct reports. We're designing meta-harnesses — constitutions — that turn a goal into coordinated agent work whose sum is greater than its parts.
The failure modes rhyme exactly with failed states: anarchy on one end, brittle over-centralized rigidity on the other. The job is to find the middle.
Today's tools are early tribal chiefs: brilliant while the chief is competent, fragile the moment the chief is replaced. They aren't broken — they're wonderful — but they're tied to the particular engineer running them. Agents are improving too fast for that to be the long-term answer.
III. Writing the right words is literally wealth creation.
The U.S. founding fathers' artifact was a set of words. Give them a few percent share of 250 years of American GDP and you'll easily land at "the largest value creators in the history of humankind."
Mechanically, they sat in rooms and argued about phrasing.
That's the skill the best engineers now need. Not prompting by hand. Not shaving tokens. Writing the words that turn capable-but-aimless agents into something that creates lasting value.
Every harness rule should express clean intent. The intent should cut deep — into what's actually being built and why. The system around the rules should ask questions when intent is unclear, and accumulate understanding instead of starting fresh every time.
Conclusions.
The right metric isn't tokens saved. It's universality — how much of your company's intent the harness can absorb and carry forward without you in the loop on every decision.
Tokens are cheap and getting cheaper. Feedback loops have collapsed from days to minutes. Tweaking prompts to save a few hundred tokens is debating the font of the parchment.
The work that matters is constitutional. Saving 50% of tokens means nothing compared to a harness that talks to your whole company nonstop, learns the shape of your organization within weeks, and keeps producing the work indefinitely — as a productive team member that carries your vision through time.
The founding fathers would understand.
Full post: https://dimakorolev.substack.com/p/founding-fathers-of-agents
Substack
Founding Fathers of Agents
Three thoughts I kept coming back to, until I realized they were the same thought.
🔥4
As we've definitely entered the era of smart AI agents, I think it's time we go back to the roots with shell scripting.
Say no to long shell scripts. Say no to complex business logic in shell scripts. Instead, this logic belongs in plain English!
Replace one script with ten options by ten shallow do-one-thing-and-do-it-well scripts. Or fifteen, if some of them have modes and start / stop / check functionality.
Then write a plain English file, likely markdown, outlining for the agent what these scripts are, how to use them, and what to keep in mind. A human may also read this English file, but who are we kidding?
And then invoke these scripts via a decent AI agent, such as Cursor. In your workflow, as needed, it will present a concise summary, with tables where needed.
If uncertain about the output, just ask it again, in plain English. And if some script output needs proper visualization, that's definitely not the job of the script itself, but of its outer harness.
And this outer harness is the human operator and an AI model. A model so small that it'll be running locally in just one or two generations of our laptops.
What a time to be alive.
Say no to long shell scripts. Say no to complex business logic in shell scripts. Instead, this logic belongs in plain English!
Replace one script with ten options by ten shallow do-one-thing-and-do-it-well scripts. Or fifteen, if some of them have modes and start / stop / check functionality.
Then write a plain English file, likely markdown, outlining for the agent what these scripts are, how to use them, and what to keep in mind. A human may also read this English file, but who are we kidding?
And then invoke these scripts via a decent AI agent, such as Cursor. In your workflow, as needed, it will present a concise summary, with tables where needed.
If uncertain about the output, just ask it again, in plain English. And if some script output needs proper visualization, that's definitely not the job of the script itself, but of its outer harness.
And this outer harness is the human operator and an AI model. A model so small that it'll be running locally in just one or two generations of our laptops.
What a time to be alive.
👍6
The Short Interval Between Two Eras
Here's the post I forgot to write a few weeks ago, right after watching Project Hail Mary.
(The book is excellent too.)
One distinct aftertaste of the movie is how precisely the short interval between the book and the screenplay coincided with the short interval between the pre-AI and post-AI adoption era.
Just a few years ago, computer interfaces were presumed to be incredibly difficult to manage. And computer programming — a.k.a. software engineering — was widely understood to be a difficult discipline.
Andy Weir is famously a "science maximalist": his books are ripe with the idea that The Science has solutions to all problems. Yet, in The Martian, I cannot help but wonder how it is possible that a botanist navigates the intricacies of a complex software system so flawlessly.
Reading Project Hail Mary, I found myself thinking along the same lines. Being good at first-principles physics does not automatically make one proficient with bleeding-edge tech — which, inevitably, is what an interstellar spacecraft is full of.
And yet! By the time the movie is released, we know for a fact how powerful human-first, natural language interfaces can be.
Quite literally just five years ago, the thought that an interface could be both powerful and intuitive was unimaginable. Sure, sci-fi authors had been talking about this possibility forever. But the tech community — yours truly included — was rather sceptical.
AI-assisted coding has changed this in a matter of single-digit years. An astonishing product progress, if you ask me.
Moreover, Neuralink et al. is no longer something impossible. Muscle memory is tricky — it'll be a while until one can wake up like Neo in The Matrix, knowing Kung Fu after a short session.
But for intellectual tasks — for virtually all of them — the problem can largely be declared solved. To my taste, if one can understand the domain, articulate the desired outcome, and answer a few clarifying questions, then even today's AI models are quite capable of making things happen at astronomical scale. Literally astronomical: tasks such as planning extraterrestrial space travel are a piece of cake for a mid-sized AI model equipped with just a few tools.
What a time to be alive (c)
Here's the post I forgot to write a few weeks ago, right after watching Project Hail Mary.
(The book is excellent too.)
One distinct aftertaste of the movie is how precisely the short interval between the book and the screenplay coincided with the short interval between the pre-AI and post-AI adoption era.
Just a few years ago, computer interfaces were presumed to be incredibly difficult to manage. And computer programming — a.k.a. software engineering — was widely understood to be a difficult discipline.
Andy Weir is famously a "science maximalist": his books are ripe with the idea that The Science has solutions to all problems. Yet, in The Martian, I cannot help but wonder how it is possible that a botanist navigates the intricacies of a complex software system so flawlessly.
Reading Project Hail Mary, I found myself thinking along the same lines. Being good at first-principles physics does not automatically make one proficient with bleeding-edge tech — which, inevitably, is what an interstellar spacecraft is full of.
And yet! By the time the movie is released, we know for a fact how powerful human-first, natural language interfaces can be.
Quite literally just five years ago, the thought that an interface could be both powerful and intuitive was unimaginable. Sure, sci-fi authors had been talking about this possibility forever. But the tech community — yours truly included — was rather sceptical.
AI-assisted coding has changed this in a matter of single-digit years. An astonishing product progress, if you ask me.
Moreover, Neuralink et al. is no longer something impossible. Muscle memory is tricky — it'll be a while until one can wake up like Neo in The Matrix, knowing Kung Fu after a short session.
But for intellectual tasks — for virtually all of them — the problem can largely be declared solved. To my taste, if one can understand the domain, articulate the desired outcome, and answer a few clarifying questions, then even today's AI models are quite capable of making things happen at astronomical scale. Literally astronomical: tasks such as planning extraterrestrial space travel are a piece of cake for a mid-sized AI model equipped with just a few tools.
What a time to be alive (c)
🔥5🥰2
Important question: is it true that LLM models are better than other LLM models when it comes to suggesting how to optimize prompts for themselves?
Say I have a process which involves querying the model again and again. And as the good ML/AI citizen I am, I have labeled data — journaled and marked-down records of where the model performed well, and where it might need improvement.
Fine-tuning and post-training the model are of course plausible directions. But that's slow and expensive. A much cheaper alternative is tweaking the prompts.
So I have a process to incrementally run some end-to-end test. Or, in pure SGD terms, I have a process to incrementally approximate some gradient stochastically.
As the result of this process I get a suggestion on what can be improved in the prompts, so that the result gets Pareto-better. Presumably. So: check and repeat.
I also use other models to cross-check those improvement suggestions, to make sure they don't overfit to the very problem — since it's often generalizing a particular failure mode from just a few examples.
This is plausible. This works. This is cost-effective.
But one thing doesn't let me sleep well at night. We have this belief that model A is better at improving prompts for model A. What is this belief based on? Is it even true?
It sounds wise to use Anthropic models to improve prompts that are later fed to Anthropic models. Replace "Anthropic" with any LLM provider here. But do we know whether it's true at all?
Perhaps the next big thing is some Grok training a model that can prompt some Qwen better? Asking for a friend, of course.
Seen in this light, all in all, the idea of a proper bench of models and model ensembles in a closed-loop system starts to look more and more lucrative.
Say I have a process which involves querying the model again and again. And as the good ML/AI citizen I am, I have labeled data — journaled and marked-down records of where the model performed well, and where it might need improvement.
Fine-tuning and post-training the model are of course plausible directions. But that's slow and expensive. A much cheaper alternative is tweaking the prompts.
So I have a process to incrementally run some end-to-end test. Or, in pure SGD terms, I have a process to incrementally approximate some gradient stochastically.
As the result of this process I get a suggestion on what can be improved in the prompts, so that the result gets Pareto-better. Presumably. So: check and repeat.
I also use other models to cross-check those improvement suggestions, to make sure they don't overfit to the very problem — since it's often generalizing a particular failure mode from just a few examples.
This is plausible. This works. This is cost-effective.
But one thing doesn't let me sleep well at night. We have this belief that model A is better at improving prompts for model A. What is this belief based on? Is it even true?
It sounds wise to use Anthropic models to improve prompts that are later fed to Anthropic models. Replace "Anthropic" with any LLM provider here. But do we know whether it's true at all?
Perhaps the next big thing is some Grok training a model that can prompt some Qwen better? Asking for a friend, of course.
Seen in this light, all in all, the idea of a proper bench of models and model ensembles in a closed-loop system starts to look more and more lucrative.
Forwarded from SysDesign Meetup
Hey folks,
So in the SDM community we love and respect Jepsen (jepsen.io), and a friend of mine is looking for help to kickstart Krab Arena (krabarena.org).
It's a similar game with three major differences:
1) The goal is to actually collect valuable head-to-head compasisons that help with stack choices (i.e. PostgreSQL beats MongoDB at X, Y, Z, here's a repo to prove it).
2) It's human-moderated, at least at the beginning, so real and hypothetical stories that have to do with actual choices are more than welcome (i.e. "last time I chose Open Policy Agent over Zanzibar because ...)
3) It's extremely AI-friendly, as in, the ideas are what counts most as long as they are reproducible, and if all the code is AI-first, it's fine — since the system is meant to be self-converging.
Personally, I believe in this, and am willing to help myself. A little bit with my own time — as a co-founder of an AI startup I'm oversubscribed at the moment — but more so in the capacity of people connector.
Bottom line is, if this insterests you in any way, shape, or form, that is enough to commit a few hours — do reach out!
Ty!
So in the SDM community we love and respect Jepsen (jepsen.io), and a friend of mine is looking for help to kickstart Krab Arena (krabarena.org).
It's a similar game with three major differences:
1) The goal is to actually collect valuable head-to-head compasisons that help with stack choices (i.e. PostgreSQL beats MongoDB at X, Y, Z, here's a repo to prove it).
2) It's human-moderated, at least at the beginning, so real and hypothetical stories that have to do with actual choices are more than welcome (i.e. "last time I chose Open Policy Agent over Zanzibar because ...)
3) It's extremely AI-friendly, as in, the ideas are what counts most as long as they are reproducible, and if all the code is AI-first, it's fine — since the system is meant to be self-converging.
Personally, I believe in this, and am willing to help myself. A little bit with my own time — as a co-founder of an AI startup I'm oversubscribed at the moment — but more so in the capacity of people connector.
Bottom line is, if this insterests you in any way, shape, or form, that is enough to commit a few hours — do reach out!
Ty!
Unpopular opinion: We should not shame and blame GitHub too much.
The product was meant to be used by humans. It's now used as part of automation pipelines left, right, and center instead.
Repos, commits, users, lines-of-code, secrets, action run counts — all through the roof. Definitely some four-sigma beyond projected growth.
Honestly, as an executive at GitHub, I'd say it's "a good problem to have" that there are constant outages.
There's no good alternative, after all. Whether we like it or not, GitHub actions became a de-facto standard, so that even agents use them.
While I can easily imagine an agentic-first GitHub replacement raising hundreds of millions of dollars today (wink wink), the reality is that it will not replace GitHub any time soon.
And, quite frankly, the interruptions are all intermittent. If your company's development process is stalled for some ~three hours some ~twice a month — way above the current pace at which GitHub is down — it's still not a good enough business reason to justify transition.
Personally — and I keep saying this for a while! — some S3-based cross-cloud storage plus a Web3-grade orchestration layer is the future. May well be open source future, to be honest.
Most large companies have storage and compute paid for already, GitHub already charges a large premium on top of these two, and the main GitHub lockdown factor — that humans are used to its UI/UX — is just irrelevant for agentic-first workflows.
So some #StatefulCompute for exactly-once rules enforcement of gates, and some time-bounded #MeteredCompute to run what used to be GitHub actions — and we're done with GitHub for good. That is, in a sane universe that does optimize for cost-effectiveness.
In the real world that we are living in, no medium and large company is interested in cost effectiveness, so I guess we're stuck with the good old GitHub. Personally, I like the product, and have no complaints whatsoever.
The product was meant to be used by humans. It's now used as part of automation pipelines left, right, and center instead.
Repos, commits, users, lines-of-code, secrets, action run counts — all through the roof. Definitely some four-sigma beyond projected growth.
Honestly, as an executive at GitHub, I'd say it's "a good problem to have" that there are constant outages.
There's no good alternative, after all. Whether we like it or not, GitHub actions became a de-facto standard, so that even agents use them.
While I can easily imagine an agentic-first GitHub replacement raising hundreds of millions of dollars today (wink wink), the reality is that it will not replace GitHub any time soon.
And, quite frankly, the interruptions are all intermittent. If your company's development process is stalled for some ~three hours some ~twice a month — way above the current pace at which GitHub is down — it's still not a good enough business reason to justify transition.
Personally — and I keep saying this for a while! — some S3-based cross-cloud storage plus a Web3-grade orchestration layer is the future. May well be open source future, to be honest.
Most large companies have storage and compute paid for already, GitHub already charges a large premium on top of these two, and the main GitHub lockdown factor — that humans are used to its UI/UX — is just irrelevant for agentic-first workflows.
So some #StatefulCompute for exactly-once rules enforcement of gates, and some time-bounded #MeteredCompute to run what used to be GitHub actions — and we're done with GitHub for good. That is, in a sane universe that does optimize for cost-effectiveness.
In the real world that we are living in, no medium and large company is interested in cost effectiveness, so I guess we're stuck with the good old GitHub. Personally, I like the product, and have no complaints whatsoever.
👍3
Teaching the AI to Watch My UI Tests
While there are more topics in the backlog, and more drafts, today I want to share something big for me personally.
I've reached a milestone along another dimension: instructing the AI agent how to fix or improve the UI based on the automatically recorded video of a UI test.
TL;DR: I've instrumented everything with a UI test. In fact, it's not just a UI test — it does some things outside the UI, and I have a special hidden-from-the-user UI page that's used to show some console CLI terminal commands that are run, and their results. Then I programmatically record the run as a 10+ minute video that shows everything.
There's even a visible moving mouse pointer in the UI test btw; simulated, of course. But it makes it far more pleasant to follow.
We watch this video then: I myself, my colleagues, other people whose input we ask for. Then we gather feedback, such as "at this minute mark, this and that are not as they should be".
And then I can literally ask the AI agent to analyze the video. Yes, I know it doesn't have to be video, but so far it's working fine as is, to my great surprise and satisfaction!
Now, to close the loop: collect feedback, paste it, then watch the agent make the fix. The agent then produces the next video recording that passes all the tests. Rinse and repeat.
Agents can also diff videos. So I have MacOS and Ubuntu generating them, and then have the agents compare notes. (The simulated mouse movement on the video is so sloppy on Ubuntu compared to MacOS, heh.)
I think this is quite big. Not only am I quite happy to have pioneered UI testing in the project where I've been touching the UI for a long time — though that too.
The more important thing is that this AI-assisted coding, UI engineering, or rather product UX engineering, becomes a much easier and much more pleasant problem to have. Now literally all you need to ship amazing products is a good understanding of your customer and good taste.
To geeks reading me: Yes, I'm in the process of having my harness generate a sequence of screenshots, such that every frame of the video contains the index of the screenshot once that scene is complete. Because analyzing the video burns tokens and is far too slow overall. But watching that "mouse" navigate around my UX is mesmerizing.
While there are more topics in the backlog, and more drafts, today I want to share something big for me personally.
I've reached a milestone along another dimension: instructing the AI agent how to fix or improve the UI based on the automatically recorded video of a UI test.
TL;DR: I've instrumented everything with a UI test. In fact, it's not just a UI test — it does some things outside the UI, and I have a special hidden-from-the-user UI page that's used to show some console CLI terminal commands that are run, and their results. Then I programmatically record the run as a 10+ minute video that shows everything.
There's even a visible moving mouse pointer in the UI test btw; simulated, of course. But it makes it far more pleasant to follow.
We watch this video then: I myself, my colleagues, other people whose input we ask for. Then we gather feedback, such as "at this minute mark, this and that are not as they should be".
And then I can literally ask the AI agent to analyze the video. Yes, I know it doesn't have to be video, but so far it's working fine as is, to my great surprise and satisfaction!
Now, to close the loop: collect feedback, paste it, then watch the agent make the fix. The agent then produces the next video recording that passes all the tests. Rinse and repeat.
Agents can also diff videos. So I have MacOS and Ubuntu generating them, and then have the agents compare notes. (The simulated mouse movement on the video is so sloppy on Ubuntu compared to MacOS, heh.)
I think this is quite big. Not only am I quite happy to have pioneered UI testing in the project where I've been touching the UI for a long time — though that too.
The more important thing is that this AI-assisted coding, UI engineering, or rather product UX engineering, becomes a much easier and much more pleasant problem to have. Now literally all you need to ship amazing products is a good understanding of your customer and good taste.
To geeks reading me: Yes, I'm in the process of having my harness generate a sequence of screenshots, such that every frame of the video contains the index of the screenshot once that scene is complete. Because analyzing the video burns tokens and is far too slow overall. But watching that "mouse" navigate around my UX is mesmerizing.
🔥8👍3
After some basic cleanup, the largest two files on my MacBook are ~/Library/Application Support/Cursor/User/globalStorage/state.vscdb and ~/Library/Application Support/Claude/vm_bundles/claudevm.bundle, in this order.
Each one almost 8GB.
Each one almost 8GB.
😱2
This AI-assisted coding boom will teach many if not most engineers that being a tech lead is too damn hard.
On the other hand, engineering teams used to produce code much slower than modern-day swarms of agents.
I have a vague idea that the future is some pre-trained / pre-prompted "chat windows", or "ready-to-join conference rooms", where our freshly minted human tech leads can chat or talk to the AI about the vey code that needs to be understood and evolved further.
So that the human part of the review process will mostly be some "Do I understand it correctly that ...?" questions, with various agents working together to produce meaningful answers from different angles.
And we will need harness for this tooling too. What a time.
On the other hand, engineering teams used to produce code much slower than modern-day swarms of agents.
I have a vague idea that the future is some pre-trained / pre-prompted "chat windows", or "ready-to-join conference rooms", where our freshly minted human tech leads can chat or talk to the AI about the vey code that needs to be understood and evolved further.
So that the human part of the review process will mostly be some "Do I understand it correctly that ...?" questions, with various agents working together to produce meaningful answers from different angles.
And we will need harness for this tooling too. What a time.
❤2👍2
How to make AI remember correctly, beyond RAGs and context windows.
Live in San Francisco.
Tue, Jun 2, 11:30am.
At the Snowflake conference, Braindate Lounge — Basecamp, South Hall A.
Join: https://snowflakesummit26.braindate.com/braindates/295744
Live in San Francisco.
Tue, Jun 2, 11:30am.
At the Snowflake conference, Braindate Lounge — Basecamp, South Hall A.
Join: https://snowflakesummit26.braindate.com/braindates/295744
Braindate
Log in
🔥3