Trust, but Verify: Proving Which AI Model Answered

Quick Take

Researchers at Anthropic and several universities just open-sourced FlashProve, a system that cryptographically proves which AI model generated a response, without ever exposing the model’s weights.

On a 400B-parameter model, it generates proofs (~19 MB) at ~110 tokens/sec on a single H100 GPU. Verification takes ~17 seconds on a CPU. Think of it as a tamper-evident digital receipt you can request for AI outputs, generated after the fact rather than attached to every response.

The Problem: “How do we know which model actually answered?”

Right now, AI responses are just text. There’s no cryptographic proof tying the output to a specific set of weights. That leaves room for three annoying (and sometimes dangerous) scenarios:

Silent substitution: A router or vendor quietly serves a cheaper/different model than what you paid for.
Server tampering: An attacker compromises an inference endpoint and slips in modified or malicious outputs while blending in with genuine responses.
Evaluation drift: You test a model pre-release but can’t guarantee end users are actually hitting that same version later.

Quality checks won’t catch this, since a substitute model can still write good answers. And re-running the model yourself is usually impractical (weights are locked down, and inference is computationally heavy).

How It Works (Without the Crypto Jargon)

The system uses a zkSNARK (zero-knowledge succinct argument of knowledge). Skip the math; just remember two things: it’s zero-knowledge (leaks nothing about weights) and succinct (verification is cheap).

The flow is straightforward:

Commit: The provider publishes a cryptographic fingerprint of the model weights (one-time setup per model).
Prove: For each prompt, the provider generates a proof that this specific fingerprint produced this exact response.
Verify: Anyone with the fingerprint, prompt, and response can verify the proof in ~17 seconds on a CPU. No GPU or weights required.

Honest providers always pass. Fake proofs are computationally infeasible to forge (though model architecture remains visible to verifiers).

The Benchmarks & Costs

The team tested FlashProve on Llama 4 Maverick, a very large AI model from Meta, using one server with a single high-end AI chip. In short: making a proof takes a lot of computing power, but checking one is quick and cheap.

Speed: Proofs are produced at about 110 tokens per second (a token is roughly a word or part of a word).
Proof size: About 19 MB per proof, covering up to 1,024 tokens of text.
Checking a proof: About 17 to 21 seconds on an ordinary computer, with no AI chip needed and no access to the model’s secrets.
The cost: Making a proof takes roughly 70–95 times the computing time of just answering the question.
Setup: About 39 minutes to register a model’s fingerprint, which then covers about 16 million requests before it needs renewing.

For comparison: The only earlier attempt on this model that the researchers know of took more than 14 hours, produced a 93.6 GB proof and took almost a day to check. The code and documents are free to use and build on (open source, Apache 2.0 license).

What Practitioners Can Actually Do With This

Catch a swapped or hacked model. If a system only accepts answers with a valid proof, a hacked server or stand-in model can’t fake one. Proofs reveal nothing secret, so they can be shared with users too.
Check you’re getting what you paid for. If you use an AI model through a third-party service, a proof confirms the answer came from the model you’re paying for, not a cheaper substitute.
Show the work was done, not that it was done well. A provider could prove a model spent at least a million tokens’ worth of effort reviewing a software update and found no bugs. Useful for audits, but not a safety guarantee.
What it takes to run. Making proofs needs one server with a single high-end AI chip. Checking them needs only an ordinary computer and a small 6.5 MB download of the model’s fingerprint.

The Catch: Current Limits

This is still a research prototype, not a production-ready solution. Key limitations include:

Short responses: Currently supports only 1,024 tokens, far below the context lengths offered by modern AI models.
Limited compatibility: Works only with specific AI architectures and settings. It doesn’t yet support all model types or response-generation methods.
Architecture visibility: Model weights remain private, but the underlying model architecture is visible to anyone verifying the proof.
Potential information leaks: The time taken to generate proofs could reveal information about the model’s internal workings.
Incomplete verification guarantees: The core protocol has been formally verified, but the faster verification software has not yet received the same level of mathematical validation.

Where the Gaps Really Are (The Real Talk)

Provable inference is a major step forward, but it proves only that an answer came from a specific model, not that the entire AI system is trustworthy.

Doesn’t guarantee the right model: A provider could register an inferior model and still produce valid proofs. Independent checks are needed to confirm the model matches what was promised.
Doesn’t cover everything: It verifies the model serving a request, not what else the data center might be doing.
Model identity needs clear rules: Even small technical changes can affect outputs. Standards must define what counts as the same model.
Cost remains a barrier: Verification takes seconds, but generating proofs costs 70–95× more than standard inference. Widespread use may initially rely on random checks.
Not a standalone safeguard: It works best alongside secure hardware, independent testing, and other verification methods.
Effort doesn’t equal quality: Proving an AI tried to find bugs doesn’t prove the code is safe or bug-free.
Verification itself needs oversight: Standards must define exactly what is being proven and ensure the verification software is trustworthy.

Bottom Line

FlashProve turns “trust me” into “prove it.” It’s a working prototype that can verify which AI model weights were used to generate a response, though it still has practical limitations and measurable costs. It’s not a complete solution to security or compliance, but it’s a promising step toward making AI systems more transparent and accountable. The next big challenges are independent testing and support for longer inputs.