I’m getting jaded by LLM code generating ability.

It’s become so normalized for me in my daily coding workflow and flowstate that it’s going to take a lot to impress me.

It’s pretty a obvious line to me in my head what I can ask it to do and what I can’t ask it to do.

gpt4-o supposedly being 100 ELO points above any other model in coding ability means that it is better than gpt-4 on coding tasks 64% of the time, and worse than gpt-4 36% of the time.

I get excellent code generated for specific types of tasks, and it still struggles in the same types of requests for more complex tasks.

I wonder when there will be an LLM, (maybe gpt-5?) where I will feel like a whole new slew of things I can comfortably ask it to do has opened up a new vista of coding, like when I switched from coding raw to coding with GPT-3.5 and cursor.

That probably looks like an ELO difference compared to gpt-4 of at least 500 (ELO>1800), which corresponds to a model beating gpt-4 96% of the time, until I’ll feel like an order of magnitude difference in code generation has occurred.

Or maybe instead of such a discrete jump, the improvement will be gradual as the competition catches up for gpt4-o in code generation, the whole ecosystem of LLM’s inching each other out by a few ELO points every month?

Time will tell, but I can’t wait until the next jump in code-generation LLM occurs, whether gradual or all at once, for my own craft.

Keeping the most advanced AI centralized for the greater good is an exercise in tyranny & totalitarianism.

I fear the centralization of tyranny, the tyranny of centralization, way more than I fear potential hate speech or knowledge on advanced weaponry being put in front of plebs.

While quantum entanglement can synchronize the spin of two particles light years apart, the reason you can’t send information faster than the speed of light is while you don’t know their spins, you know they’re synced.

My grad school quantum physics professor explained it like such:

Imagine you shuffled two decks of cards separately, and keep them hidden, face down.

Then, you rub them against each other, and magically (“quantum entanglement” according to physicists), the randomly shuffled orders are the same.

Now, you put each synchronized deck of cards in a separate spaceship and send each one to the opposite side of the galaxy.

When you turn over the Queen Of Hearts followed by the Seven Of Spades on one side of the galaxy, the that also shows up in the deck of cards (the spin of the entangled particle to escape the analogy) that’s because they were entangled.

But that’s also why you can’t send information faster than the speed of light via quantum entanglement as we understand it today - because while it’s miraculous that the entangled particles across the galaxy will collapse their quantum wave functions to the same specific quantum states, and those states show that knowledge stretches across time, no signal of information can pass across since the collapsed value is still randomly decided.

As in, you know the order of the cards is the same across the galaxy, but you can’t predict ahead of time what those cards will be, so you can’t send information faster than light via quantum entanglement.

When will we actually empirically look at government policies and determine if they were effective?

Did the Dept of Ed quantitatively improve education?

Did the Dept of Agriculture quantitatively improve our food supply?

Did the income tax improve our economy?

Did the environmental protection agency protect the environment?

I’m betting the answer isn’t all yes nor all no, but without sufficient quantitative metrics that decide if a policy continues, all policies seem to just settle into the sediment of bureaucracy, only to be spoken of as “this is just the way it’s done” with too much bureaucratic momentum to ever consider reversing, downscaling, or otherwise acting on ineffectual results.

Sad state of affairs tbh.

We’re spending hundreds of billions of dollars to feed in more sentences to LLM’s so they better predict the next language token, but comparatively spending mere pennies on analyzing the thought processes of typical humans during a typical day’s decision making. Retarded AI dev.

Do you realize how much easier space travel would be if we had FTL communication (not necessarily FTL travel)?

Quantum entanglement can’t help since you can’t control the resulting spin of the entangled particles once they’re observed so while you know they will end up with the same spin, even across the galaxy, you can’t control the spin and thus cannot send information between them.

Temporary micro-wormholes might work, but also would likely require a machine to travel to the other side before establishing a tunnel.

Hopefully AI helps us devise new physics (that’s falsifiable by experiment, the bedrock of true science) that can enable it.

Otherwise it’s gonna be interesting as different cultures, different markets, different currency rates, are all out of sync across the galaxy.

Can you image the git merge conflicts!?

This is the future we fight against: powerful AI under lock and key at a military base.

The best AI should be open for all.

“Open” as in open source, open weights, and maybe even open data if we’re lucky.

Not closed source but “free” (which is what OpenAI thinks the word “open” means).

And definitely not under lock and key to only be used by a few “trusted” persons.

The military controlling it because it’s too powerful for the regular pleb is just too dystopian and totalitarian.

They’ll use it against us all, such is the nature of power.

But the drooling powerhungry bureaucrats will couch it under the guise of “we can’t let those evil Chinese and Russian bad guys get it! They’ll steal our best ideas, build on top of it, and then never open up their super DUPER powerful AI!

I believe there’s a greater risk of corruption from centralized powerful AI from within the gates than from losing intellectual property to villains but that’s just me.

I say give it to everyone, all at once, and let the community build upon it. The old hacker ethos of “information belongs to the world, information should be free to all!” will prevail in the long run.

You just need one good magnet torrent link and it’s over.

Open-source, open-weights, open-data, decentralized AGI can help return the web to its roots.

So I use https://perplexity.ai before Google for any knowledge based question. It works better if you ask it questions in complete sentences.

Perplexity is an entirely better way to gain knowledge from the web over search engines.

It’s the killer app because it’s an abstraction atop the LLM’s - as in you can choose to use GPT-3.5 or GPT-4o or Claude Opus or Llama 3 as the backend summarizer.

But all it’s doing is searching the web, summarizing the results, and spitting the LLM-generated summary back out to you in paragraphs and bullet points and the like.

All this the free version of Perplexity provides.

Use it every day.

Pay the $20/month for the Pro version of Perplexity?

Regarding the underlying LLM being used to summarize your search results, how much would you want GPT-4 vs GPT-3.5 doing the summarization? Meh?

The second benefit to Pro is it does a step where it uses an LLM to “understand your question” first and then generate a few web search terms based on its preprocessing. Is this beneficial? Maybe?

The third benefit is early access to features like their new Pages, where you can do a bunch of your own queries and then turn the results into a Wikipedia-looking page on a topic that you can immediately share with other people to educate them.

So I pay for it, because I like advanced features, but imho you’re getting most of the real benefits (an order of magnitude better knowledge gathering experience) from the free version.

Use it.

We’re smarter than cows 🐮 so we domesticate and control them - not because we’re malicious, but merely because our intelligence is clearly beyond their dumb bovine capabilities.

We control them, breed them, eat them 🥩, and it’s not particularly difficult given such the difference in intelligence between us 🧠.

They’re incapable of understanding what we’re doing and it’s not even close.

Many people believe this is how the relationship between humans and Artificial Superintelligence will be 🤖.

I don’t; I see tech as a tool but that’s just me 🤷🏻‍♂️

So when I was in tenth grade (15 years old) my English teacher flipped out at the whole class because we couldn’t figure out what “anthropomorphizing” meant from context clues and Latin roots.

Her point was that if we were smarter, or trying harder, or thinking better, we’d realizing that “anthro” meant human, and “morph” meant change, and “izing” meant performing an act.

That even if we had never seen that word before, just by mastery of the English language, we should have been able to figure out what a new word meant.

I felt dumb.

I always remembered that day.

She was right, all the information was there for us, we should have been smarter.

It’s a funny coincidence that that particular word (“anthropomorphism”) from my teen years, is one of the core issues with modern AI. I’ve been thinking about this word for decades lol

Does raising money for a startup dilute your drive to actually find a profitable business model?

Is the incentive to just focus on whatever metrics help you with your next round, whether or not that’s actually profitable?

Glad I never took big money. It’s harder this way, and I’m constantly hustling, but I’m free in a unique way, and I have to actually think about cashflow and profit in a way that backed startups don’t.

To each their own.

I’ve had trouble finding other founders who’ve taken my approach and have to figure out how to reinvest profit into growing the business instead of making a better slideshow talking about “traction” and other vanity metrics.

Again to each their own, but I’d encourage young ambitious founders to try to grow a profitable business before or instead of just looking for big funding.

Maybe I’m being foolish, we shall see.

This is my first Father’s Day as a father myself. And what a blessing it is! My baby son just lighting up when he sees me, laughing and smiling and curious about the world, is an indescribable feeling.

And yet today I’m also thinking of my father, who, 12 years ago, died from a heart attack at 52 years old when I was 25.

Take care of your health and don’t wait for tomorrow to be happy, to take that risk, start that business, begin exercising, or say what you have to say to someone. The end may come too soon and you don’t want to live with regrets.

I find myself daydreaming of conversations I might have had with my dad have regarding the modern world. He and I would probably stay up late trying to have conversations about physics with ChatGPT. He’d see society changing and yet only care about how it was affecting me and my reaction. He’d be so happy that I was able to start a family with my loving wife. He’d be a wonderful grandfather to my son.

But alas, death means that the end of 2012 was the last time we’d ever have a conversation. His coffee cup from the morning was still in the sink; he clearly thought he’d have time to wash it tomorrow and didn’t expect that day to be his last day on Earth.

His demeanor and mannerisms and even his voice tone are so vivid in my mind, but have faded from this world as his soul is now in the great beyond.

I undoubtedly didn’t become my full self until he died and I dealt with the death of my father; and in some macabre way, we should all be lucky enough to lose our parents instead of the other way around. But ideally not until much later, if fate permits.

I feel his personality, his quirks, his intelligence, him teaching me chess, our deep convos, him caring about my every need, wanting me to have everything he never had growing up, all present with me in my heart.

Life’s gears turn from the notches that grief creates, and yet we move on and appreciate those still with us.

If your dad is still with us, have a bit of an extra long conversation with him today, for one day your conversation will be the last. And in the meantime, enjoy your time together even if it’s lighthearted. Make the effort to heal any rifts and don’t wait for him to reach out, for a heavy heart does nobody any good.

Spending all day with my son today made me both happy and sad. But I’m grateful for the wonderful relationship I had with my father and the fact that I get to experience it myself. Bittersweet in the most heartwarming way.

Happy Father’s Day!

The more you use these LLM’s for code gen, whether through cursor or just in a chat window, the more you get a fingertip feel for what it can and can’t do.

Yes it can refactor a function into smaller modular functions, no it can’t center a div.

Yes it can generate boilerplate flask, no it can’t turn my webapp async calls into a job queue with celery and redis.

Each new model that dropped after GPT4 (15 months ago), whether Opus or Llama or Gemini, has been unable to do something GPT4 cannot do.

Incremental improvements will indeed move the needle over time, but large orders of magnitude jumps in code gen don’t seem like they’ll come from better LLM’s as it stands but rather with agents, abstractions atop LLM’s, extremely high quality curated datasets, or some other new paradigm.

Regardless, I’m looking forward to have that (very clear) line of what LLM’s can’t do, moved visibly. This is the process of normalizing new capabilities by regular use and then moving the goalposts. Because the code gen ability of pre-GPT3.5 and post-GPT3.5 was indeed an order of magnitude jump.

In fact, on their blog post last year, OpenAI said that when they realized that their transformers could actually generate functional code instead of just prose, that’s when they realized they needed to productize it.

It’s funny how companies know they’re supposed to integrate AI, but either don’t know how, or are worried about sending proprietary data to other companies’ servers.

Regarding the former, they’re giving lip service to something that they are actually really timid to execute on.

Regarding the latter, they’re too cowardly to invest in local open source LLM and are waiting for a mature enterprise solution in two years. Maybe cowardly is too harsh, they’re just late adopters (to an inevitability…).

How much of an effect does the neural architecture have versus the training data?

Put another way: with billions of sentences available for training, was the transformer self-attention architecture necessary to make a convincing chatbot?

Read more books, anon.

Three Body Problem, Dark Forest, Death’s End is a great trilogy that will expand your mind.

Children of Time is another great sci-fi trilogy.

Bobiverse is four books and is a great series. Do your own research on the premise.

Bobiverse is four books and is a great series. Do your own research on the premise.

I hate the AI people that always seem to have some drama that reminds me of high school. I’m 37 years old, I find juvenile drama juvenile.

I hate the smug coders who declare their chosen language or path is superior despite being mediocre coders at best.

I hate the venture capital posters who came in fresh off crypto, thinking they’re the gods and market movers for giant trends in AI, and if you’re not up on their vibe you’re clearly an outsider.

I hate the ingroup, outgroup dynamic of the young AI enthusiasts who decided their chosen flavor of cult is obviously the future of the species and you’re either with them or against them.

I hate the esoteric tech enthusiasts who decided they figured out sentience aka consciousness and the secret summoning of machine demons and obviously you don’t understand the nature of the universe if you don’t understand “What’s Coming” ™️.

I hate the subtle implications that if you’re not part of the right cool kids group chat or ingroup or event, you’re a nobody unworthy of consideration.

I hate the people on this platform that got into AI a year or two ago and clearly are the cutting edge “thought leaders” you must accept as having the correct opinions.

I hate the people who seemingly know every minutiae of modern neural networks yet have never built an actual production AI system.

I hate the entrepreneurs who add “AI” to their offering and yet don’t have the self-awareness to realize they’re just adding buzzwords they don’t actually understand.

I hate threadbois.

I hate fake coders.

I hate AI “safety experts”.

I hate the snarky “I’m smarter than you” assholes.

I hate the usurpers and the frauds and the grifters.

I hate the engagement baiters and the over-confident alphas.

I truly hate the AI social media scene, and am just going to build what I know how to build.

We’ll see who ends up at the top once the dust settles.

I tend to write code in layers. I get an end-to-end system working as a base layer and then I go back over the whole system and fill in the gaps with subsequent layers.

I find that this is cumulatively faster than building each component piece by piece and assembling them at the end.

It allows for rapid feedback in bugs, it gets most of the tedious boring code stuff out of the way early, and allows for modularizing as you go given the system as a whole.

What do I mean by “layers”?

Let’s use a fictional example. Say a client wants me to build them an object detection system that multiple users can interact with simultaneously.

I would first stand up a very simple react app that allows an image to be uploaded and then an output image to be displayed. A few lines of code. I’d have a backend flask app have a single api that inputs an image and outputs random noise.

Once I have random noise being output, I’ll then go over the whole system and make the output generator a separate modular function that inputs an image and outputs noise.

Then I’ll maybe spin up a celery docker with redis for a simple job queue for generating the noise.

Now it can handle scale and simultaneous requests.

I’d make a new api that checks the status of the job.

At each step, so far, the entire end to end system is working to display something, and is being tested and hardened end to end as I go.

Then perhaps I’ll swap out the noise generator function with a yolo PyTorch model from hugging face to do the object detection.

Now, instead of the output react panel showing noise, it shows the result.

Then I’ll maybe add an active learning loop where the output is corrected and the model is fine-tuned.

The point is not about this hypothetical example, but rather how I think about it: layer by layer getting the whole system functional from the start, and layering on the requisite feature without ever breaking the end to end.

The next thing I might do is containerize the whole system. Making sure it still works.

Then I’ll separate the containers for the frontend and backend. Making sure it still works.

Then perhaps I’ll horizontally scale the backend. Still making sure the entire end to end system works.

Etc etc.

I’ve seen too many coders just dive in without a clear understanding of the process of engineering a system, spending hours and days on one piece, then moving on to the next piece. Finally when they integrate it all, they deal with bugs that arise from that integration.

I find that a subpar approach to engineering than ensuring an entire end to end system is working at each stage and layering on complexity and modularization.

I fully support mass deportations of all illegal immigrants by force (they shouldn’t have snuck into another country).

I fully support border patrol agents using rubber bullets, barbed wire, or similar against anyone who isn’t actually running for their lives (very small percentage of them actually need asylum).

We have a process for citizenship, which can definitely be improved, but which illegals completely ignore. It requires a citizenship test, pledge of allegiance, and actually following the law.

Right now, border patrol agents just let anyone in.

Don’t care how it affects the economy (it’ll help).

Don’t care how uncompassionate it is (go through the legal citizenship process if you want to be an American so badly).

Don’t care how logistically difficult it’ll be (we can do it if we wanted to).

Don’t care how many low skilled jobs will be unfilled (robots and high schoolers).

Don’t care if I get labeled xenophobic (I don’t have a phobia of xenos).

Fake asylum seekers are just criminals.

Open borders are a disaster.

Anyway, just sharing my opinion; you can disagree with me and that’s okay too.

Any climate change “computer model” being used to dictate global policy should be open source, with open data, and have way more public eyes on it than merely a “trust the experts” approach that we have today in order to trigger an emotional reaction.

I’m not denying anything here (we’ve had some alarmingly warm winters here compared to my memories from last millennium), but we need a more data-driven and measured approach instead of fearmongering.

(Also it’s not cow farts, relax…)