Green Broadcasting

On Pavel Durov's Decentralized LLM Compute Network

Pavel Durov announced an open network that would give out TON cryptocurrency for GPU compute.

Let's set aside the use of cryptocurrency that is volatile as heck and additional overhead of distributed computing for already quite power hungry hallucinating text prediction machines. I want to scrutinize the security of the concept.

The more likely approach:
They can employ Trusted Execution Environment present in NVIDIA GPUs. You encrypt your data with an Nvidia key, and only a genuine Nvidia graphics card can decrypt it. It won't give you, the card owner, the decrypted data, but will encrypt the result with the user's key.

Edit: This is a gross simplification of the process. Actually, you deploy an image onto a remote machine that runs on CPUs TEE, and from it, using NVIDIA's SDK you can also verify that the GPU is real. Prompt encryption/decryption can be implemented inside the image. All communications between your image in the CPU TEE and GPU TEE are encrypted. This way you can also get proof that the GPU runs the model it claims to be for free. Of course, you don't store the key in the image, you do DH key exchange as it runs (maybe, you use user's public key hash as nonce for the TEE Attestation Report to avoid MITM attacks, since there is no CA here but the vendors).

The only problem is that if a single one of these cards gets hacked, it will unleash absolute hell across the entire network 🙂 (in other words: it is not good if even a single node node in a zero-trust network can read your messages)

There is a fool proof solution to this problem that I was very sceptical about until this day, when I decided to finally look into it. It's called Homomorphic Encryption.

It allows you to do neural network inference on encrypted data without actually being able to read the data. Yeah, it might sound insane, but after reading into it and understanding key concepts, I think it's probably fine. But it's also probably not what's gonna be done in this project for reasons described below (TL;DR — Impractical on large scale).

This section requires you to have basic understanding about how neural networks work.

So this announcement made me look into how neural networks work with homomorphic encryption, as it's the only reasonable way to implement such a feature without leaking all user communications to random people on the internet or hoping on a vendor.

In short:
1. There's a way to encrypt two numbers, do addition or multiplication on them, and then get the correct result back on decryption. Do your own research, if curious.
2. Generative transformers with attention algorithms (esp. important — activation functions) can be approximated by using polynomials, which is just adding and multiplying numbers.
Combine these two and you can run an LLM on encrypted data, and get encrypted output back, that only the creator of the data can read!

Yes, instead of 0..1 range you get huge numbers, you can't tell if the neuron is a 1 or a 0, but because we approximate all the neuron logic with the allowed operations, the math guarantees the result to be correct on decryption (to a certain precision)

This method is well-suited for relatively small models, or models that can be given a long time to think in the background. Stuffing DeepSeek into FHE will either result in a very bad and slow model, or a very expensive but still slow model that requires a ton of resources and simply cannot be optimized with modern MoE tricks; it has to be a dense model, where every neuron is evaluated. It also means you can't have dynamic output length using the end of output token, you'd have to generate a fixed number of tokens for every request.

It's almost certainly not the way they are gonna go with, though, they'll probably just use TEE, will be sending your conversations with the LLMs to random people on the internet, and you'll just have to pray that the random node isn't an NSO Group honeypot with a jailbroken CPU/GPU.

There's a chance we'll get distributed LLMs in the future, but not today

❤3

177 viewsedited 15:45