Alright, attempting to move from posgresql with pgvector to self-hosted weaviate on EKS for an AI RAG system.
pgvector's embedding search worked great with HSNW index, but the keyword search is quite annoying.
If I only wanted semantic search, I'd stick with pgvector. But sometimes, I need keyword search. For instance, querying my AI with "what do you know about my projects with the client SomeCompanyName" will fail on semantic embedding search, and required a keyword search.
On postgres with tsvector, AND isn't correct for long queries, and OR returned hundreds of thousands of rows, which made the retrieval take several seconds.
I was able to build my own RRF with SQL but I didn't like doing it. I'll still use my own cohere reranker afterwards.
Postgres doesn't have bm25 built-in, and I can't install extensions on AWS Aurora. I looked into setting up my own bm25 implementation with SQL queries (creating my own IDF table for word frequencies with sql triggers), but even then it doesn't have BlockMax-WAND the way the new Weaviate does.
I want to self-host, and I want to practice using kubernetes with a private VPC without paying for Weaviate's enterprise support, so I'm going to use the open source weaviate option with helm. I want to deploy everything with IAC (serverless framework) instead of AWS marketplace so I won't be doing that either. And while EC2 is an option, I'd rather not manage servers.
If any of what I said didn't make sense, sorry this post isn't for you. But I wanted to share my thought process of setting up my AI infrastructure.
Time to have fun with the tech! Wish me luck.
pgvector's embedding search worked great with HSNW index, but the keyword search is quite annoying.
If I only wanted semantic search, I'd stick with pgvector. But sometimes, I need keyword search. For instance, querying my AI with "what do you know about my projects with the client SomeCompanyName" will fail on semantic embedding search, and required a keyword search.
On postgres with tsvector, AND isn't correct for long queries, and OR returned hundreds of thousands of rows, which made the retrieval take several seconds.
I was able to build my own RRF with SQL but I didn't like doing it. I'll still use my own cohere reranker afterwards.
Postgres doesn't have bm25 built-in, and I can't install extensions on AWS Aurora. I looked into setting up my own bm25 implementation with SQL queries (creating my own IDF table for word frequencies with sql triggers), but even then it doesn't have BlockMax-WAND the way the new Weaviate does.
I want to self-host, and I want to practice using kubernetes with a private VPC without paying for Weaviate's enterprise support, so I'm going to use the open source weaviate option with helm. I want to deploy everything with IAC (serverless framework) instead of AWS marketplace so I won't be doing that either. And while EC2 is an option, I'd rather not manage servers.
If any of what I said didn't make sense, sorry this post isn't for you. But I wanted to share my thought process of setting up my AI infrastructure.
Time to have fun with the tech! Wish me luck.
When you have kids, life is so much better when you have a big family and live near them.
We have all grandparents within a half hour.
There are approximately 20 children in our family within an hour of us between 1 and 9 years old.
Large birthday parties every month, activities for kids, family friendly events, support, bonding.
It just creates such a better experience for the little ones, and makes life more fulfilling.
Don’t deprive kids of family!
Grow large families!
We have all grandparents within a half hour.
There are approximately 20 children in our family within an hour of us between 1 and 9 years old.
Large birthday parties every month, activities for kids, family friendly events, support, bonding.
It just creates such a better experience for the little ones, and makes life more fulfilling.
Don’t deprive kids of family!
Grow large families!
Currently, the code gen LLM’s are still really bad at knowing:
- when to break functions into sub-functions
- where to include try-catch logic and when to let errors propagate
- when to create a service class versus call a sequence of functions
- when to return tuples versus dictionaries versus classes versus lists versus sets
- how to not repeat commonly used code, and inversely when to not create a “reusable” function that is only used once
- when it makes sense to use an environment variable versus a global scope versus a local scope
- where and how to properly parallelize
In short, if you know what to ask for, it does a great job. But if you’re using AI to design code architecture, it will not do better than a skilled coder.
- when to break functions into sub-functions
- where to include try-catch logic and when to let errors propagate
- when to create a service class versus call a sequence of functions
- when to return tuples versus dictionaries versus classes versus lists versus sets
- how to not repeat commonly used code, and inversely when to not create a “reusable” function that is only used once
- when it makes sense to use an environment variable versus a global scope versus a local scope
- where and how to properly parallelize
In short, if you know what to ask for, it does a great job. But if you’re using AI to design code architecture, it will not do better than a skilled coder.
Today marks 13 years since my father died from a heart attack at 52 years old.
His coffee mug from his morning coffee was still in the sink unwashed; he clearly thought he’d have time to wash it tomorrow.
But that tomorrow never came.
Don’t wait for tomorrow to be happy. Don’t think, “I’ll be happy when…” something happens. Tomorrow is not guaranteed.
He smoked about two packs of cigarettes a day. Don’t smoke cigs, don’t fall for immature psy-ops about how they make you look cool. They increase the risk of nearly every disease. And nicotine addiction is real.
He didn’t exercise enough, and didn’t eat very well. But plenty of people who don’t exercise or eat well live far past 52. It was the cigarettes.
He was loving, kind, and taught me good character. He and I would nerd out over Einstein’s theories of relativity, space travel, chess, and sci-fi books from when I was a young kid. He always had my back and would gently guide me through any situation.
He was born in Hungary, as was all his ancestors. His family fled to America (legally) to escape the Hungarian communists in the 1960s.
He never cared about politics, only his immediate family. He was extremely smart (many people said he was the smartest person they’d ever met) and had deep, complex thoughts.
I wish he could have met his grandchildren (both I and my sister have two children), but alas that was not his fate.
I keep his memory alive, and I try to learn whatever lessons I can from his death.
If your dad is still alive, keep the relationship flowing; one conversation may be your last and you’ll never know it until it’s too late.
I will meet him in the Great Beyond when my time is up.
🤍
His coffee mug from his morning coffee was still in the sink unwashed; he clearly thought he’d have time to wash it tomorrow.
But that tomorrow never came.
Don’t wait for tomorrow to be happy. Don’t think, “I’ll be happy when…” something happens. Tomorrow is not guaranteed.
He smoked about two packs of cigarettes a day. Don’t smoke cigs, don’t fall for immature psy-ops about how they make you look cool. They increase the risk of nearly every disease. And nicotine addiction is real.
He didn’t exercise enough, and didn’t eat very well. But plenty of people who don’t exercise or eat well live far past 52. It was the cigarettes.
He was loving, kind, and taught me good character. He and I would nerd out over Einstein’s theories of relativity, space travel, chess, and sci-fi books from when I was a young kid. He always had my back and would gently guide me through any situation.
He was born in Hungary, as was all his ancestors. His family fled to America (legally) to escape the Hungarian communists in the 1960s.
He never cared about politics, only his immediate family. He was extremely smart (many people said he was the smartest person they’d ever met) and had deep, complex thoughts.
I wish he could have met his grandchildren (both I and my sister have two children), but alas that was not his fate.
I keep his memory alive, and I try to learn whatever lessons I can from his death.
If your dad is still alive, keep the relationship flowing; one conversation may be your last and you’ll never know it until it’s too late.
I will meet him in the Great Beyond when my time is up.
🤍
I feel like one of the big missing pieces with AI systems is RAG that throws useful knowledge into the context window that doesn’t directly answer the query but is valuable.
If the query is about marketing, retrieval brand voice information is useful.
If asking a strategic question, throwing some analysis of Napoleon may be useful.
If generating code, pulling in SOLID principles book may be valuable context.
But those may not give a low cosine distance with embeddings, nor would they necessarily show up on a node or a knowledge graph from a random user query.
And relying on the LLM to consider outside information, even with RLHF has never worked.
This is, I feel, a missing piece: to determine outside context that needs to be fed in to a RAG system that the cosine embedding similarity, the keyword search, and knowledge graphs might miss.
This might potentially be solved with a good mcp server and a langgraph or ADK references that can pull extra knowledge with some context clues, but that requires pre-programming specific doc-strings for the outside nearby context and thus is not fully generalizability.
Any ideas on how to do this?
If the query is about marketing, retrieval brand voice information is useful.
If asking a strategic question, throwing some analysis of Napoleon may be useful.
If generating code, pulling in SOLID principles book may be valuable context.
But those may not give a low cosine distance with embeddings, nor would they necessarily show up on a node or a knowledge graph from a random user query.
And relying on the LLM to consider outside information, even with RLHF has never worked.
This is, I feel, a missing piece: to determine outside context that needs to be fed in to a RAG system that the cosine embedding similarity, the keyword search, and knowledge graphs might miss.
This might potentially be solved with a good mcp server and a langgraph or ADK references that can pull extra knowledge with some context clues, but that requires pre-programming specific doc-strings for the outside nearby context and thus is not fully generalizability.
Any ideas on how to do this?
I just spend several hours going back and forth with ChatGPT 5.2 on the proper way to do a distributed microservice system with sqs, lambdas, weaviate.
It kept adding extraneous hash checks, different status fields in the database, and tried to plug every hole for every situation which will not exist.
It took a lot of time for us to converge on the proper architecture and workflow and to handle the edge cases we needed to handle.
It made up fake race conditions and came up with situations like a database failing halfway through a write with a weird rollback operation that simply wouldn’t occur.
It kept saying things like “well you could have concurrent processing and wasted compute” and only capitulated when I walked it through how that wouldn’t be wasted compute.
I don’t understand how vibe coders expect to one-shot AI code gen for advanced complex systems in a way that makes maintainable code that itself won’t need to be re-engineered later.
I still use AI to write the code for me (Opus 4.5 is my favorite right now in opencode cli) but not without back and forths.
People who don’t engineer things properly and think through a solid design are going to end up with slop.
Use the tools, don’t let the tools use you.
It kept adding extraneous hash checks, different status fields in the database, and tried to plug every hole for every situation which will not exist.
It took a lot of time for us to converge on the proper architecture and workflow and to handle the edge cases we needed to handle.
It made up fake race conditions and came up with situations like a database failing halfway through a write with a weird rollback operation that simply wouldn’t occur.
It kept saying things like “well you could have concurrent processing and wasted compute” and only capitulated when I walked it through how that wouldn’t be wasted compute.
I don’t understand how vibe coders expect to one-shot AI code gen for advanced complex systems in a way that makes maintainable code that itself won’t need to be re-engineered later.
I still use AI to write the code for me (Opus 4.5 is my favorite right now in opencode cli) but not without back and forths.
People who don’t engineer things properly and think through a solid design are going to end up with slop.
Use the tools, don’t let the tools use you.
You can write high quality code using AI agents.
Not all AI generated code is vibe coded slop.
But it requires you to know what to ask for.
To tell it how to design your distributed database system;
Which hooks and mutations to implement in your web app;
How to do proper authentication with refresh tokens and oauth;
To know when to ask it to break up monolithic functions into small self-contained service classes;
When to ask it to use a dictionary versus a list;
Precisely how to handle Websocket dropped connections;
The way the agents should be designed with either mcp or simple tool lists.
If you know how to code, AI accelerates what you do yourself.
If you review every line the AI writes, you become more skilled at both coding and prompting.
Or if you just let the AI take the wheel, lord help you with a complicated mess of patch upon patch that can never be maintained.
Be smarter than your average vibe coder lest you stay a normie in a different hat.
Not all AI generated code is vibe coded slop.
But it requires you to know what to ask for.
To tell it how to design your distributed database system;
Which hooks and mutations to implement in your web app;
How to do proper authentication with refresh tokens and oauth;
To know when to ask it to break up monolithic functions into small self-contained service classes;
When to ask it to use a dictionary versus a list;
Precisely how to handle Websocket dropped connections;
The way the agents should be designed with either mcp or simple tool lists.
If you know how to code, AI accelerates what you do yourself.
If you review every line the AI writes, you become more skilled at both coding and prompting.
Or if you just let the AI take the wheel, lord help you with a complicated mess of patch upon patch that can never be maintained.
Be smarter than your average vibe coder lest you stay a normie in a different hat.
There is a deep sadness in seeing someone in their 60s or 70s who wishes they were a grandparent, but whose adult kids decided to be child-free.
For them to survive into old age and then not be able to experience the joy of grandchildren, to see their bloodline end despite having kids themselves, makes me extremely sad for them.
For them to survive into old age and then not be able to experience the joy of grandchildren, to see their bloodline end despite having kids themselves, makes me extremely sad for them.
If you want to educate yourself, you should study the math of how the KVQ matrixes work in modern AI.
Transformers in AI are at the heart of learning. In essence, the way it works is AI sees a new sentence from its training data and tweaks what matrix multiplication to apply to all the tokens such that it would have paid attention to the correct words in order to complete the sentence as close to the training sentence as possible.
The AI transformer basically learns how to change the way the input text is embedded in three different ways (K, V, Q) and keeps learning how to better embed them until it pays attention to the correct words and completes the sentence accurately.
Transformers in AI are at the heart of learning. In essence, the way it works is AI sees a new sentence from its training data and tweaks what matrix multiplication to apply to all the tokens such that it would have paid attention to the correct words in order to complete the sentence as close to the training sentence as possible.
The AI transformer basically learns how to change the way the input text is embedded in three different ways (K, V, Q) and keeps learning how to better embed them until it pays attention to the correct words and completes the sentence accurately.
The greatest sadness in my life is that my father never got to meet his grandchildren.
Take care of your health; heart attacks can come in your early 50s and then that’s all she wrote.
He would have been a great grandpa.
Take care of your health; heart attacks can come in your early 50s and then that’s all she wrote.
He would have been a great grandpa.
If you haven’t read all 6 Dune books, the big lessons was the risk of centralization.
The God Emperor (Paul’s son, half-man half-sandworm) saw The Golden Path like Paul did, and acted on it.
The Golden Path is a narrow path in the multiverse that was the only way to prevent the extinction of humanity.
It required him to become “The Tyrant” for 3500 years, controlling government and religion, being a fully centralized planner, so that people would experience the extreme of centralization.
Then, when he died, when finally free from a truly centralized government humanity would scatter to the stars.
The God Emperor wanted to “teach human a lesson they would feel in their bones” to avoid centralization.
It was the only way to ensure humanity wouldn’t go extinct. Central planning and control always leads to stagnation.
The Tyrant acted as a collar on humanity, showing them just how much they should not desire centralization, and when that collar was released, humanity finally underwent “The Scattering” and guaranteed they wouldn’t go extinct.
As the instinct of centralized government and control and religion comes up in our real life, it’s important to remember the lessons from Dune.
Decentralization is the way, The Golden Path.
The God Emperor (Paul’s son, half-man half-sandworm) saw The Golden Path like Paul did, and acted on it.
The Golden Path is a narrow path in the multiverse that was the only way to prevent the extinction of humanity.
It required him to become “The Tyrant” for 3500 years, controlling government and religion, being a fully centralized planner, so that people would experience the extreme of centralization.
Then, when he died, when finally free from a truly centralized government humanity would scatter to the stars.
The God Emperor wanted to “teach human a lesson they would feel in their bones” to avoid centralization.
It was the only way to ensure humanity wouldn’t go extinct. Central planning and control always leads to stagnation.
The Tyrant acted as a collar on humanity, showing them just how much they should not desire centralization, and when that collar was released, humanity finally underwent “The Scattering” and guaranteed they wouldn’t go extinct.
As the instinct of centralized government and control and religion comes up in our real life, it’s important to remember the lessons from Dune.
Decentralization is the way, The Golden Path.