Show HN: KVSplit – Run 2-3× longer contexts on Apple Silicon
26 by dipampaul17 | 0 comments on Hacker News.
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss- K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss- The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer context on the same Mac. Memory usage scales with sequence length, so savings compound as context grows. Implementation was straightforward: 1. Added --kvq-key and --kvq-val flags to llama.cpp2. Applied existing quantization logic separately to K and V tensors3. Validated with perplexity metrics across context lengths4. Used Metal for acceleration (with -mlong-calls flag to avoid vectorization issues) Benchmarked on an M4 MacBook Pro running TinyLlama with 8K context windows. Compatible with Metal/MPS and optimized for Apple Silicon. GitHub: https://ift.tt/A0kwa2I
26 by dipampaul17 | 0 comments on Hacker News.
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss- K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss- The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer context on the same Mac. Memory usage scales with sequence length, so savings compound as context grows. Implementation was straightforward: 1. Added --kvq-key and --kvq-val flags to llama.cpp2. Applied existing quantization logic separately to K and V tensors3. Validated with perplexity metrics across context lengths4. Used Metal for acceleration (with -mlong-calls flag to avoid vectorization issues) Benchmarked on an M4 MacBook Pro running TinyLlama with 8K context windows. Compatible with Metal/MPS and optimized for Apple Silicon. GitHub: https://ift.tt/A0kwa2I
ClojureScript forks Google Closure to guarantee backward compatibility
19 by Borkdude | 2 comments on Hacker News.
19 by Borkdude | 2 comments on Hacker News.
Show HN: Solidis – Tiny TS Redis client, no deps, for serverless
4 by jayl-e-e | 0 comments on Hacker News.
Hey everyone! Over the past two years I threw myself back into full-time engineering with a simple goal: write code that gives back to the community. After a lot of late-night FOMO (“AI will do it all for us, right?”) and some painful production incidents, I finally turned my weekend project into an open-source library. [ What is Solidis ? ] - Super-light (< 30 KB) RESP2/RESP3 client with zero runtime deps and first-class ESM/CJS support. - Fully tree-shakable – import only the commands you need. - Written with SOLID principles & full TypeScript typings for every command. - Designed for cold-start sensitive serverless platforms (small bundle + tiny memory footprint). [ Why I built it ] 1. node-redis & ioredis pain - ESM is still an after-thought. - Hidden deadlocks on RST, vague error surfaces. - Everything gets bundled, even commands you’ll never call. 2. I refuse to add a dependency I don’t fully understand – I literally read candidates 10× before `npm i`. 3. Serverless bills love to remind me that every KB and millisecond matters. [ Key features ] - Protocols: RESP2 and RESP3 (auto-negotiation) - Bundle size: `<30 KB` (core) / `<105 KB` (full) - Dependencies: 0 - Extensibility: Drop-in command plugins, custom transactions - Reliability: Auto-reconnect, per-command timeouts, type-checked replies [ Roadmap / Help wanted ] - Benchmarks against `node-redis` & `ioredis` (PRs welcome!) - More first-class Valkey love - Fuzz-testing the parser - Docs site – the README came first; I’d love help polishing full docs This might be my last big OSS push for a while, so stars, issues, and PRs mean the world . If Solidis saves you some cold-start time or just scratches a TypeScript itch, let me know! Repo: https://github.com/vcms-io/solidis License: MIT Thanks for reading, and happy hacking! (Feel free to AMA in the comments – I’m around.)
4 by jayl-e-e | 0 comments on Hacker News.
Hey everyone! Over the past two years I threw myself back into full-time engineering with a simple goal: write code that gives back to the community. After a lot of late-night FOMO (“AI will do it all for us, right?”) and some painful production incidents, I finally turned my weekend project into an open-source library. [ What is Solidis ? ] - Super-light (< 30 KB) RESP2/RESP3 client with zero runtime deps and first-class ESM/CJS support. - Fully tree-shakable – import only the commands you need. - Written with SOLID principles & full TypeScript typings for every command. - Designed for cold-start sensitive serverless platforms (small bundle + tiny memory footprint). [ Why I built it ] 1. node-redis & ioredis pain - ESM is still an after-thought. - Hidden deadlocks on RST, vague error surfaces. - Everything gets bundled, even commands you’ll never call. 2. I refuse to add a dependency I don’t fully understand – I literally read candidates 10× before `npm i`. 3. Serverless bills love to remind me that every KB and millisecond matters. [ Key features ] - Protocols: RESP2 and RESP3 (auto-negotiation) - Bundle size: `<30 KB` (core) / `<105 KB` (full) - Dependencies: 0 - Extensibility: Drop-in command plugins, custom transactions - Reliability: Auto-reconnect, per-command timeouts, type-checked replies [ Roadmap / Help wanted ] - Benchmarks against `node-redis` & `ioredis` (PRs welcome!) - More first-class Valkey love - Fuzz-testing the parser - Docs site – the README came first; I’d love help polishing full docs This might be my last big OSS push for a while, so stars, issues, and PRs mean the world . If Solidis saves you some cold-start time or just scratches a TypeScript itch, let me know! Repo: https://github.com/vcms-io/solidis License: MIT Thanks for reading, and happy hacking! (Feel free to AMA in the comments – I’m around.)
Getting AI to write good SQL: Text-to-SQL techniques explained
45 by richards | 8 comments on Hacker News.
45 by richards | 8 comments on Hacker News.
They Were Identical 'Twinnies' Who Charmed Orwell, Camus and More
5 by lermontov | 1 comments on Hacker News.
5 by lermontov | 1 comments on Hacker News.
Show HN: Merliot – plugging physical devices into LLMs
3 by sfeldma | 0 comments on Hacker News.
Merliot Hub is an AI-integrated device hub. What does that mean? It means you can control and interact with your physical devices, your security cameras, your thermometer, seamlessly using natural language from an LLM host such as Claude Desktop or Cursor. The hub is a gateway between AI and the physical world. What could go wrong?
3 by sfeldma | 0 comments on Hacker News.
Merliot Hub is an AI-integrated device hub. What does that mean? It means you can control and interact with your physical devices, your security cameras, your thermometer, seamlessly using natural language from an LLM host such as Claude Desktop or Cursor. The hub is a gateway between AI and the physical world. What could go wrong?
Show HN: Roast My Dish – AI roasts your food photos with brutal honesty
3 by romeumaleiane | 0 comments on Hacker News.
Hey HN!I created a web app called Roast My Dish, it's a fun side project where you upload a photo of your dish, and an AI chef roasts it brutally. After the roast, it gives you a personalized "redemption" recipe to help you improve (or laugh). It's built using Next.js, Tailwind.css, MongoDB, Azure Open Ai API, and I tried to mix entertainment with a bit of usefulness. Great for foodies, meme lovers, and anyone curious about AI + humor use cases. Would love your feedback or ideas to improve it!
3 by romeumaleiane | 0 comments on Hacker News.
Hey HN!I created a web app called Roast My Dish, it's a fun side project where you upload a photo of your dish, and an AI chef roasts it brutally. After the roast, it gives you a personalized "redemption" recipe to help you improve (or laugh). It's built using Next.js, Tailwind.css, MongoDB, Azure Open Ai API, and I tried to mix entertainment with a bit of usefulness. Great for foodies, meme lovers, and anyone curious about AI + humor use cases. Would love your feedback or ideas to improve it!
A Linux kernel developer plays with Home Assistant: case studies
8 by pabs3 | 1 comments on Hacker News.
8 by pabs3 | 1 comments on Hacker News.
Show HN: Kokoro TTS – Free Text to Speech Converter with Multilingual AI Voices
3 by sudofoo | 0 comments on Hacker News.
3 by sudofoo | 0 comments on Hacker News.