AWS MSK on AWS Graviton
https://aws.amazon.com/blogs/big-data/amazon-msk-now-provides-up-to-29-more-throughput-and-up-to-24-lower-costs-with-aws-graviton3-support/

#MSK

yet another message broker

Memphis

https://docs.memphis.dev/memphis/memphis-broker/architecture

ZomboDB brings powerful text-search and analytics features to Postgres by using Elasticsearch as an index type. Its comprehensive query language and SQL functions enable new and creative ways to query your relational data.

ZomboDB is a 100% native Postgres extension written in Rust with PGRX. ZomboDB uses Postgres's Index Access Method API to directly manage and optimize ZomboDB's specialized indices. As a native Postgres index type, ZomboDB allows you to CREATE INDEX ... USING zombodb on your existing Postgres tables. At that point, ZomboDB takes over and fully manages the remote Elasticsearch index, guaranteeing transactionally-correct text-search query results.

https://github.com/zombodb/zombodb/

This project contains a series of tiny broken programs (and one nasty surprise). By fixing them, you'll learn how to read and write Zig code.

#zig

https://codeberg.org/ziglings/exercises

KahaDB is a file based persistence database that is local to the message broker that is using it. It has been optimized for fast persistence. It is the the default storage mechanism since ActiveMQ Classic 5.4. KahaDB uses less file descriptors and provides faster recovery than its predecessor, the AMQ Message Store.

In order to facilitate rapid retrieval of messages from the data logs, a B-tree index is created, which contains pointers to the locations of all the messages embedded in the data log files. The complete B-tree index is stored on disk and part or all of the B-tree index is held in a cache in memory. Evidently, the B-tree index can work more efficiently, if the complete index fits into the cache.

https://github.com/apache/activemq/tree/main/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb

Interesting article about Kafka latency issue in scale due to ext4 filesystem, the sciencific approach they followed is really beuatiful.

https://blog.allegro.tech/2024/03/kafka-performance-analysis.html

Slim allows developers to inspect, optimize and debug their containers .

#docker #optimization
https://github.com/slimtoolkit/slim

Kafka Tierd storage feature, a novel approach for archiving kafka messages. IMO NOT READY FOR PRODUCTION BUT KEEP EYE ON IT

https://developers.redhat.com/articles/2024/03/13/kafka-tiered-storage-deep-dive

https://aws.amazon.com/blogs/big-data/deep-dive-on-amazon-msk-tiered-storage/

How neural network works under the hood? explaining about vector embeddings and it's use cases.
https://www.datastax.com/guides/what-is-a-vector-embedding

We've all heard about RocksDB, but there's also Speedb, which may not be as famous as RocksDB, but it appears to be the fastest key-value storage engine in the world. Redis recently announced that they own Speedb and have included it in Redis's core. You can find more information about it here:
https://www.speedb.io/

"I wrote this Format dialog back on a rainy Thursday morning at Microsoft in late 1994, I think it was.

We were porting the bajillion lines of code from the Windows95 user interface over to NT, and Format was just one of those areas where WindowsNT was different enough from Windows95 that we had to come up with some custom UI.

I got out a piece of paper and wrote down all the options and choices you could make with respect to formatting a disk, like filesystem, label, cluster size, compression, encryption, and so on.

Then I busted out VC++2.0 and used the Resource Editor to lay out a simple vertical stack of all the choices you had to make, in the approximate order you had to make. It wasn't elegant, but it would do until the elegant UI arrived.

That was some 30 years ago, and the dialog is still my temporary one from that Thursday morning, so be careful about checking in "temporary" solutions!

I also had to decide how much "cluster slack" would be too much, and that wound up constraining the format size of a FAT volume to 32GB. That limit was also an arbitrary choice that morning, and one that has stuck with us as a permanent side effect.

So remember... there are no "temporary" checkins :)

Follow me for more random code musings!"

From Dave W Plummer, Developer of many famous Windows components such as Task Manager, Windows Pinball, Calc, ZIPFolders, Product Activation, etc.

https://twitter.com/davepl1968/status/1772042158046146792

https://people.freebsd.org/~phk/
Poul-Henning Kamp (a.k.a phk) is one of the most well-known software engineers in the world, known for developing many parts of the FreeBSD kernel and Varnish Cache.

This is one of his notes titled 'Notes from the Architect' on the Varnish Cache website.
https://varnish-cache.org/docs/trunk/phk/notes.html

—
"Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage.

So what happens with squids elaborate memory management is that it gets into fights with the kernels elaborate memory management, and like any civil war, that never gets anything done."

—-
Multi-CPU systems is nothing new, but writing programs that use more than one CPU at a time has always been tricky and it still is.
—-

https://google.github.io/pytype/
https://github.com/facebook/pyre-check/
https://github.com/google/importlab

python static analysis code, especially type checking , and modules dependencies on each other

Do not forget these two rules:

Everything in software architecture is a trade off,
and always why is more important than how.

Anderson's law:
I have yet to see any problem, however complicated, which, when you looked at it in the right way, did not become still more complicated.

so when initially examining a problem, it's common to find that it becomes more complex as you go deeper into it. This complexity often requires a deeper level of investigation to determine the best solution.

#architecture

human interaction could be a bottleneck in most software projects, and should be managed very carefully.
there is a law around it that says "Adding manpower to a late software project makes it later."
it called Brooks's law

#architecture