Posted by ltononro 4 days ago
The problems of inference proof for a closed model provider are more difficult than an open model provider -- you can get a probabilistic proof much more cheaply out of four independent inference providers running the same non-batched prompt than this 90-100x compute hit.
Anthropic and others also have a significant problem with these proofs in that they have a lot of incentive internally to have some portions of the pre-prompt secret; I haven't read this newest scheme closely enough to determine if they have a way to hide parts of the prompt with the proof. But if they don't, they are almost certainly not going to want to send these proofs out in the general case.
Last, I'll take this moment to soapbox and say for the researchers out there -- provable inference is nice. Provable training corpus information would be spectacular. It would be very, very, very nice to provide a set of model weights and a proof that they had only been trained on a dataset. To be clear, this is a really hard problem, but it would change fundamentally what levels of trust we could have about a given set of weights.
Is it hard? Seems like it isn't that hard if you use NVIDIA's remote attestation/enclave support instead of trying to do everything with ZKPs which have always been something of a dead end. CC overhead 2%, ZKP 100x+ and gigabyte sized proofs - just for inference. It's never been competitive.
I'm very skeptical anyone would care about this for training though. I mean, if there's someone out there who genuinely really really wants proofs of the dataset used to train model weights, let's talk? But it seems the value is mostly for open weights inference providers. They can prove what weights they're serving and they can prove they can't see people's prompts. It would take a lot of the risk out of using a no-name LLM cloud service.
Unfortunately, after many years of this stuff I have become old and cynical. The stated/revealed preference mismatch is huge. The majority of people say they want privacy in theory but in practice won't trade anything off to get it, and especially not convenience.
I’d love to read a plan that lets someone trust a pile of weights that comes with some sort of provenance proof. I’ll happily call it and you brilliant even if it was really easy to write and come up with. All my ideas for it look really difficult.
Here's the architecture I'd use. I'm going off of NVIDIA's current recommended stack / what seems most mature, if I understand correctly after a quick chat with GPT.
Basically I'd start with a bunch of Xeon 6s (Granite Rapids) connected to Blackwells. You could do it with AMD too but I trust their stack less. That gives you TDX as a starting point, which is pretty mature by now. I'd prepare a very minimal Linux image in which the OS boots straight into running a training server. You need the usual stuff like dm-verity or IMA to ensure the disk image is covered by the kernel image. The image also contains NVIDIA's driver and attestation SDK. Boot it all inside a TDX VM with GPU passthrough. I believe that currently uses encrypted bounce buffers so there'd be some perf loss, but supposedly there's some PCIe extension that can be used to eliminate that (I haven't investigated).
The trick is to combine NVIDIA's RA with Intel's. The best way to do that is just hash the evidence from the GPU attestation process into an intermediate data structure which also contains the usual stuff (e.g. ECC pubkey, nonce), that itself is hashed into the user slot of the TDX RA and then bundled all up together. After doing this you have a single über-data structure which can be used to prove that the given public key was generated inside a genuine Xeon TDX VM connected to a genuine NVIDIA GPU running in CC mode. I'd probably just use a zip file to collect all these bits of data together, for convenience when implementing the verifier. It might be worth including an RFC 3161 timestamp over all that as well. The RA protocols for something like this are always very tricky: I'd need to do some careful design work before committing to anything so don't over-read this comment.
Now the TDX RA covers the training server (indirectly, via the dm-verity root via the kernel args+hash). What is this server? It's training code (e.g. PyTorch) combined with a server that accepts document batches and - for more advanced training - maybe stuff like AllReduce traffic. Each time a document batch is accepted each doc is hashed into a sorted Merkle tree. The weight updates are then computed on the GPU and the results are hashed into a second accumulator.
Each node creates a blockchain-like log in which each record is signed by the static key embedded into the TDX RA. The records point at the hash of the previous records, and contain the current value of the Merkle root and weight accumulator. A verifier can rebuild the roots by iterating over the training set themselves in the right order and just hashing the text. I won't go into the details as you know how all this stuff works already.
I'm handwaving a lot of details around parallelism here. For a large scale training run with many nodes you'd need to solve a bunch of details like making the training servers verify each other's remote attestations before they join the cluster, etc. During the AllReduce step nodes would need to verify the data comes from a authenticated TDX VMs before accepting the gradients.
Anyway from time to time the weight updates have to be applied and the log also records the hash of the full sets of weights at that checkpoint. The final model consumer can be given the weights, along with the signed logs generated during the training run. They verify the chained RAs, verify the signatures on the logs, and if they wish, verify [a subset of] the hashes of the documents in the open training set.
Because the document Merkle tree of each batch is sorted, you can build both inclusion and exclusion proofs. Verifying an exclusion proof is more expensive as you'd have to check each batch versus a quick Merkle branch check for inclusion, but exclusion checks can be done in parallel. Normally in ML documents are shuffled rather than sorted, but the training server can sort the set for building the log entry Merkle tree, and then submit to the GPU in a shuffled order.
The bigger issue for exclusion proofs is that SHA256 is too rigid. You might need some sort of simhashing scheme to run along side it. Dunno, needs more tightly specified requirements.
Most do not even realize they have a choice and are not educated enough to make the one in their own best interest. Fast food wins as a staple for many people over a healthy home cooked meal.
Creating healthy and safe choices and making them the default is some of the most important work there is.
> the resulting commitment is good for about 16 million requests, after which the provider has to commit again.
> The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's check
I know they mentioned $ETH in the technical report, but this is exactly the mechanic for a crypto currency Proof-of-Work which is expensive by design.
When they announced it, I was hoping for a 5% overhead validation that the model that served you is really Fable and not Opus.
The quantization bit caught me out, because we saw the boundary crossing from good-context-size to bad is so sharp it feels like longer context requests are sent to quantized models to make more room for more KV-Cache. This would expose those sorts of tricks people can play to have the same GPU do better on short context benchmarks and gracefully degrade on long context.
At work, we did a bit of digging[1] into exactly why BaseTen, Fireworks and Together serve Inkling weights differently, we suspected the issue was a combination of how you do disaggregated prefill (+streaming) and quantization for longer context tasks.
[1] - https://isotopes.ai/blog/everyone-is-selling-the-same-model-...
ZkPs are expensive to check and even more so to generate because the problem is intrinsically very compute heavy. The ecosystem would immediately jump into a faster way of doing thing, compute time are seen as a flaw, not a feature.
Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon. That means we have to choose between a) fancy ZKPs, b) signed responses, or c) comparing results from multiple inference providers.
Whatever method ends up being used, it's clear to me that some form of provable inference is inevitable. People haven't caught on to its more significant applications, but they will. (This post touches upon one of them: model-driven audits.) Think about it: we can prove that a particular mind authored a particular artifact. That's never been possible before! You can't prove that Melville wrote Moby Dick, for example; but you can prove that Claude wrote a poem, or that Gemini approved a pull request, or that GPT got a particular score on some eval. This is a more powerful primitive than a cryptographic signature.
> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.
As a client that sounds like it might be much cheaper to verify than reprocessing.
With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.
Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!
This could be worked around by signing the output now and releasing the model after an embargo period, paired with contract penalties if the output of the released model does not match or the cryptographic scheme is compromised in the meantime. However, such guarantees would be worth little, since they can be just moved to a bad bank that defaults on them.
Who will find this useful? I can't really think of a use for it, because if you don't trust the provider of your models, it's game over really - you're already giving them all sorts of private data, you're relying on their service, you have to trust them to some extent and this doesn't really help because unless you are running the service yourself. Seems like more of an internal problem for Anthropic than something that would interest others.
1. it proves your models are running correctly. FP ops tend to deviate in small amounts between hardware providers and tensor ops even more so due to the relatively new and unstable standardisation environment atm. So it proves your HW configuration is delivering outputs correctly. i.e. it's a good test for what is currently a very non-trivial validation environment.
2. it proves your external provider is running the inference they said they are. i.e. it eliminates the concerns of model providers running quantized versions silently behind the scenes since quantization breaks the proof and you need to use a new, updated proof signature for the quantized version.
3. likewise for fine tuned or tampered models. Or outright substitution for another model (ex the accusation by OAI and Anthropic that Kimi and DS and co are periodically forwarding requests to their platforms to harvest responses for knowledge distillation).
All in all it's a nice chain of trust utility if it can evolve out of the toy stage and reach production viability (since rn it's limited to int-only weights and 1k context which is practically useless atm).
I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.
2. Models only change weekly for closed frontier models and even then they should be reporting model "sub-versions" if only for observability purposes.
3. This doesn't really hold true. Kimi and DS release their models on a monthly or bi-weekly cycle and while it's still DS-v4.1-Flash or whatever, under the hood it's DS-v4.1-Flash-1017 or some other date code identifier of which specific model it is and that is both exposed to the user and that model version can be requested explicitly by the user.
You're talking about this as if a bug or defect in their inference stack has no impact on you as the customer.
It might also provide an additional guarantee for users of Open Router and Fireworks.
It's worth reading past the opening paragraph before asking these kinds of questions:
>When a user sends a request to an AI model, today they have no way of knowing for certain that the model they selected is the one that generated the response. For everyday uses, in which users mostly care about the output and not necessarily which specific model produced it, this is a less pressing concern. But there are several cases where such assurance would be useful. For example, independent evaluators who test AI models before and after release need to know that the model they are evaluating is the model being served. Clients who use a model through a third-party routing service want assurance that the tokens came from the model they paid for. And if a cyberattacker modifies the token stream while pretending to be a given model, a user has no way to detect it from the output alone.
This is from the first paragraph in the Introduction.
For me if you use a routing service you have to trust it, it sees all your inputs and outputs.
If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
In all seriousness, this is mostly just one of those defense in depth kind of things. Data exfiltration via a compromised provider is also a very different class of vulnerabilities than prompt injection via malicious tool calls.
So I don't really see this as useful, except in the very limited use case of using a third party proxy who is trying to cheat you (in which case why are you sending all your data to someone you can't trust?).
I think that's where your assumption differs from stated claims.
According to Anthropics own docs[0], they don't alter the models at all after release. So for each model release you should be able to prove that a response was created with that exact model without dumbing it down.
Zero knowledge proofs also still allow you to have disclosed inputs. You can make one of the inputs be a publicly commited hash of the model weights, and have the ZKP validate that the inference was done with that committed set of model weights, in addition to all the inferences in the batch all coming from the same model.
[0]: https://platform.claude.com/docs/en/about-claude/models/mode...
None of these are directly valuable to me. But you asked "Who will find this useful?" and I gave you the answer to that question (provided by Anthropic in this post). Not everything being worked on will be useful for you. It's just bizarre that you'd be trying to make these arguments.
>If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
Sure, so it'd be great if you could know if someone is modifying your token stream. If only there was some advancement in provable inference that could allow you to verify that what you're getting from a provider is what you expect to get from them.
This is a solved problem if you trust your counterparty and if you don’t it’s the least of your problems.
All this provides is a tag saying 'x' version (which your counterparty defines anyway) is being used.
This solves for the latter with a bunch of handwaving for the former. The end goal is gonna be FUD for cheaper model routing because that obviously is at odds with their biz model.
You don't want all you eval work to go to waste just because the discount inference provider you're using decided AcmeModel4.23 is as good as the Anthropics GilgameshEpic7.2 that you used to validate your solution.
We already have trouble enough with model retirement.
I find this a bit odd and backwards. Why do we choose not to make the deployment code to be robust, but instead to observe the model inference as proven? This moves unobservability layer to the system that collects the proofs. Given the assumption that code is fluid and LLMs are rewriting it often, how can we know it isn't being abused against the observer?
Goes without saying, it's also disappointing that this has nothing to do with making the model inference observable for consumers.
It makes the inference more expensive and helps controlling the users.
> This moves unobservability layer to the system that collects the proofs.
I'd say it's adding security through a secondary system, rather than displacing security to a secondary system. Security by layers.
But I've also heard, as complete rumour, sometimes some named models perform significantly worse on some hosts and the theory is they're running quantized, without that being published. Would this solve that? The post mentions proving a set of weights, but not how those are represented? What would stop a host using true weights for generating the signature, while running on quantized versions?
> We prove an exact integer version of the published weights, and a proof asserts that each checked token is that model’s greedy choice. A reply from a floating-point implementation, or sampled decoding, could depart at some position...
So, no...?
[1] https://github.com/anthropics/FlashProve/blob/main/docs/what...
> The integer version is generally expected and empirically tested to be within some small error rate of true floating-point inference generically (even taking into account non-determinism of floating-point computations), though this does not cover adversarially designed prompts.
That's literally the most common gripe I see from Anthropic users and, so far from being "table stakes" there no existing mechanism by which those users could verify it is or isn't happening. If Anthropic can figure this out then most of this "is it just me or is Opus hurr durr today" noise can evaporate.
We've published initial performance metrics in August of 87 t/s on Muse Glimmer 30B (a dense model) - orders of magnitude faster than existing OS projects. Antropic's FlashProve supports only a fully integer model with a small context length (significantly hurting evals).
There is significant progress since then, and we'll publish more results soon.