The Unverifiable Model: Trust, Optimisation and the Economics of LLM Inference
From bootlegged API's to service degradation, a lack of verifiability highlights that we do not get always get what we pay for.
As competition among AI providers intensifies and profit margins tighten, pressure increasingly falls on the infrastructure layer. Hyperscalers, neoclouds and specialist inference providers have never been more strongly incentivised to extract the maximum economic value from their GPU fleets. Every additional token generated from the same hardware improves revenue per GPU. Every reduction in memory use creates room for larger batches, more concurrent requests or additional model replicas.
This pressure is not inherently problematic. Providers should optimise their systems. Without improvements in serving efficiency, inference would remain slower, more expensive and less widely available. The difficulty is that the economic interests of the provider and the inference consumer are not always perfectly aligned. A provider benefits whenever it can reduce the cost of serving a request. The customer benefits only when those savings do not materially reduce the quality, reliability or capabilities of the model they believe they are using.
Today, there is no bulletproof black-box method for validating either the model behind an API or the complete configuration used to serve it. An endpoint may advertise a particular model, but an external consumer generally cannot prove which weights were loaded, which numerical precision was used, whether the full context was processed, how much reasoning compute was allocated or whether some requests were routed to a cheaper substitute.
In other areas of computing, similar trust problems have been addressed through cryptographic mechanisms. Hashes can prove that a file or software binary matches a known version, while digital signatures can establish who produced an artefact and whether it has been altered. Secure hardware can attest that approved code is running inside a protected environment, and zero-knowledge proofs can demonstrate that a computation satisfied a set of rules without revealing the underlying data.
Hosted LLM inference has no widely deployed equivalent. A model provider might publish a hash of its weights, but that does not prove those weights were used for a particular request. A signed response proves only that the provider signed it, unless the signature is securely bound to the complete execution environment. Zero-knowledge proofs and verifiable-computation systems could eventually provide stronger guarantees, but proving the execution of a model containing billions of parameters remains computationally expensive and operationally complex.
As a result, inference consumers are largely forced to rely on provider reputation, contractual promises and statistical observation rather than cryptographic proof. They can test whether an endpoint behaves consistently with a claimed model, but they cannot yet independently verify the complete chain of computation that produced an individual response.
At best, consumers can observe behaviour and infer what is probably happening behind the API. That creates a fundamental trust problem at the heart of hosted AI.
Efficiency drives inference economics
The economics of inference are shaped by a simple objective: produce as many commercially valuable tokens as possible from a fixed amount of compute, memory and power. The more effectively a provider uses its GPU fleet, the lower its cost per token and the higher its potential gross margin. Marketplaces such as OpenRouter may expose characteristics such as maximum context length, advertised model variant, weight quantisation, latency and token pricing, but these visible specifications reveal only a small part of the serving configuration.
Behind the API, a provider may be adjusting batching, request scheduling, prefix caching, speculative decoding, multi-token prediction, memory allocation, tensor parallelism, activation precision, KV-cache precision, reasoning budgets and context handling. It may be separating prefill and decode across different hardware, using custom kernels, changing sampling parameters or dynamically routing requests between several models.
Some of these changes are almost entirely invisible to the consumer. Others can alter the quality of the service in ways that are difficult to measure and even harder to attribute. From the consumer’s perspective, inference optimisations can be understood as either behaviour-preserving or behaviour-altering.
Behaviour-preserving optimisations aim to reduce cost or latency without materially changing what the model does. Better batching, prefix caching, exact speculative decoding, improved request scheduling and faster attention kernels can produce substantial economic gains while leaving the user experience effectively unchanged. An inference consumer generally does not care whether a provider uses a custom CUDA kernel, a disaggregated serving architecture or a specialised accelerator, provided that the resulting service behaves as promised.
Behaviour-altering optimisations are more complicated. Aggressive quantisation, reduced reasoning budgets, context truncation, shorter output limits, model substitution and dynamic routing may all reduce serving costs, but they can also change the reliability or capability of the service.
The effects are not always obvious. A model can appear completely normal for most requests while becoming less reliable on difficult reasoning, long-context retrieval, code generation, multilingual tasks or structured tool use. Unlike lossy media compression, where discarded information may be largely imperceptible to the human eye or ear, degradation in an LLM can quickly compound over many steps and manifest as a reduction in intelligence.
The model still speaks fluently, it still produces polished paragraphs, it may continue to perform well on ordinary prompts.. Yet it becomes more likely to lose the thread, invent details, mishandle tools or confidently follow an incorrect chain of reasoning. The service appears intact while its failure rate quietly increases.
Model substitution
The clearest way to reduce inference costs is to serve a cheaper model. An API might advertise access to a frontier model while routing requests to a smaller open-weight model, an older version, a non-reasoning variant from the same family or a distilled derivative. A provider could also serve the advertised model for some requests while sending others to a less expensive substitute.
Research into so-called shadow APIs or less reputable inference providers suggests that this is not merely a theoretical concern. Some endpoints have produced behaviour that appears inconsistent with the models they claim to serve. In controlled experiments, behavioural fingerprinting systems have identified substantial substitutions with high accuracy.
These systems do not ask the model which model it is. That would be unreliable, because a model can simply repeat whatever identity has been written into its system prompt. Instead, they submit a set of carefully selected prompts and construct a behavioural fingerprint from the responses.
That fingerprint is then compared with fingerprints collected from known reference models. A related technique, Model Equality Testing, asks whether responses from the suspect API appear to come from the same statistical distribution as responses from a trusted reference endpoint.
These approaches can provide strong evidence that an endpoint behaves more like one model than another. They are particularly effective when the substitution is large, such as replacing a frontier reasoning model with a much smaller general-purpose model.
They do not, however, provide cryptographic proof.
A behavioural fingerprint identifies resemblance. It does not prove that a particular set of weights executed a particular request.
A provider could also attempt to evade such testing. It might recognise public audit prompts and route them to the genuine model. It might serve the genuine model only for a small percentage of requests. It could change routing by customer, prompt complexity or time of day. It might cache expected answers to known probes or use a distilled model trained to imitate the reference.
For this reason, black-box testing is much better at falsifying a model claim than proving one.
A failed test may provide strong evidence that an endpoint is not behaving like the claimed model. Passing the test only shows that the service behaved consistently with the reference during the prompts and period tested.
Quantisation is harder to verify
Quantisation presents a more subtle problem. A provider may genuinely be serving the advertised base model while using a lower-precision representation than the customer expects. Instead of replacing a 70-billion-parameter model with a 7-billion-parameter model, the provider might serve the same 70-billion-parameter model using FP8, INT8 or a 4-bit format such as AWQ, GPTQ or NF4.
The model name may remain technically correct, while the economics and behavioural characteristics of the deployment change substantially. Quantisation can affect the model weights, the intermediate activations created during inference and the KV cache used by the attention mechanism.
Lower-precision weights use less memory and consume less memory bandwidth. This can allow a provider to fit larger models onto cheaper hardware, increase batch sizes or operate more replicas across the same fleet.
Lower-precision activations can increase computational throughput, although preserving accuracy becomes more difficult when the model encounters outlier values or numerically sensitive operations.
KV-cache quantisation can reduce the memory required for long contexts and concurrent sessions. Its effects may be nearly invisible during short prompts but become more apparent during long conversations, distant-context retrieval or attention-heavy tasks.
Providers can also reduce costs without changing the nominal weight format. They may limit internal reasoning tokens, alter sampling parameters, truncate context, shorten outputs or route some requests through a cheaper path.
From outside the system, these effects can look similar to quantisation even when their cause is entirely different.
Why numerical precision matters
An LLM repeatedly performs matrix multiplications, additions, normalisation and probability calculations. Every value is stored using a finite amount of precision.
Smaller numerical formats use less memory and can often be processed much more quickly. Modern accelerators may perform many more low-precision operations per second than high-precision ones. The trade-off is approximation.
Imagine that the model assigns a next-token score of 12.031 to the word “yes” and 12.028 to the word “no”. At higher precision, the model slightly prefers “yes”.
After a long sequence of lower-precision operations, those values might instead become 12.02 for “yes” and 12.03 for “no”. The preference flips.
One changed token will not necessarily ruin an answer. But autoregressive models feed every generated token back into the next generation step. An early divergence can move the model onto a different path, and that difference can compound across the rest of the response.
The result may be an incorrect factual claim, broken code, malformed JSON, an invalid tool call or a reasoning chain that gradually departs from the original task.
The most difficult aspect of this degradation is that it may be intermittent. The model can perform correctly across dozens of requests and then unexpectedly fail on one prompt whose competing token probabilities happen to be particularly close.
This makes diagnosis difficult. A consumer cannot easily determine whether a failure was caused by quantisation, model substitution, a changed system prompt, context truncation, sampling randomness or the normal fallibility of the original model.
Quantisation is not inherently bad
Quantisation should not be treated as synonymous with poor service. A carefully calibrated lower-precision deployment may deliver almost identical application-level quality at a fraction of the cost. For many workloads, this is an excellent engineering outcome. As you can see in the image below, the official Huggingface page for Deepseek V4 Flash and Pro presents both FP4 and FP8 weights for download.
A well-configured FP8 deployment may be preferable to a poorly configured BF16 deployment. A quantised model may outperform a supposedly full-precision endpoint that truncates context, restricts reasoning or applies an unsuitable chat template.
The problem is not optimisation itself. The problem is undisclosed optimisation that materially changes what the inference consumer receives.
A model name alone is not a complete description of an inference service. It says nothing about the exact checkpoint revision, weight representation, activation precision, KV-cache format, usable context length, reasoning allocation, tokenizer, system prompt or routing policy.
Two providers can advertise the same model while delivering meaningfully different products. The real product is not simply the model. It is the deployment.
Why consumers cannot easily validate the service
Traditional software benefits from decades of verification techniques. Formal methods can prove that some programs satisfy a specification. Cryptographic hashes can show that a file has not changed. Digital signatures can establish who produced a message. Zero-knowledge proofs can validate certain computations without revealing their inputs.
Hosted LLM inference has no widely deployed equivalent. Consumers generally cannot inspect the model weights loaded into GPU memory. They cannot observe the data types used during execution. They cannot prove that the complete prompt reached the model or that a particular response was generated by a particular checkpoint. They cannot know whether middleware rewrote the request, truncated the context or modified the output.
Even carefully crafted identical requests may not produce identical responses and behave deterministically. Serverless inference providers typically describe their service typically as only being “best-effort” in determinism.
Differences in batching, kernels, hardware, floating-point reduction order, model revisions and serving software can all influence the output. Stochastic sampling introduces further variation.
A serious statistical comparison may require a trusted reference endpoint, many carefully selected prompts, repeated generations, controlled parameters, a model-fingerprint database and specialist statistical analysis.
Most consumers do not possess this infrastructure. As a result, model selection often involves an informal assessment. A team tries a handful of prompts, observes whether the answers look plausible and chooses a sufficiently capable model as a hedge against hidden degradation.
This is commonly thought of as a best practice but ultimately weak. Fluent output is not evidence that the advertised service was delivered.
What consumers can do without fingerprinting
Consumers who lack specialist fingerprinting tools cannot conclusively validate model identity or quantisation, but they can still detect meaningful service degradation.
The most practical approach is to evaluate behavioural stability rather than attempting to reverse-engineer the exact backend.
A consumer should maintain a private set of prompts drawn from its real workloads. These prompts should measure outcomes that can be objectively checked. Generated code can be executed against tests. JSON can be checked against a schema. Tool calls can be validated. Long-context questions can be compared with known answers. Calculations can be verified automatically.
The tests should include the kinds of tasks on which weaker or more aggressively optimised models are most likely to fail. That may include difficult reasoning, long-document retrieval, multilingual instructions, exact formatting, structured tool use and prompts containing several competing constraints.
These tests should remain private and should change over time. Public benchmarks are useful for broad comparisons, but they are easier for providers to recognise and optimise around.
A small number of canary prompts can then be run periodically. The goal should not be to demand identical wording, because stochastic systems will naturally vary. Instead, the consumer should measure whether the endpoint continues to complete the underlying task successfully.
Where cost permits, some requests should also be sent to a trusted reference endpoint. The comparison should focus on measurable outcomes such as task success, code correctness, structured-output validity, context recall, factual error rates, latency and token usage.
A single disagreement proves little. Persistent differences across a representative workload are more meaningful.
Consumers should also retain a history of these evaluations. Without a baseline, gradual degradation can easily be dismissed as anecdotal. Historical data makes it possible to identify changes in tool-call reliability, malformed outputs, reasoning quality, context retention and refusal behaviour.
Operational changes may provide supporting evidence. A dramatic reduction in latency could indicate a genuine infrastructure improvement, but it could also reflect reduced reasoning, a smaller model or more aggressive quantisation. Unexpected changes in output length, token accounting or context handling may similarly warrant investigation.
These signals are not proof. They are indicators that the service has changed and should be examined more closely.
Transparency is more practical than reverse engineering
For most consumers, contractual disclosure will be more useful than attempting to infer every serving detail from outputs. This is part of why accessing models via a Router like OpenRouter or Requesty provides better contractual guarantees, as these routers can hold providers contractually honest. In practice, market forces also ensure this.
A provider should be able to state the exact model revision, the weight precision, the quantisation method, the activation and KV-cache data types, the usable context length and the reasoning configuration. It should also disclose whether requests may be routed dynamically, whether model substitution is permitted and whether prompts or outputs are modified by middleware.
Change notification is particularly important. A provider may reasonably need to update its serving stack, but customers should know when a material change has occurred.
The level of assurance required will depend on the workload. For brainstorming, summarisation or low-stakes content generation, broad behavioural consistency may be sufficient.
For legal, medical, financial, scientific, security or autonomous-agent workloads, an unannounced change in model identity or serving configuration can materially affect safety and reproducibility. These consumers should seek version-pinned endpoints, explicit configuration commitments, audit logs, independent evaluation rights and clear remedies for undeclared substitutions.
The limits of behavioural validation
Behavioural testing can reveal whether an endpoint has become less capable, whether it differs from a trusted reference or whether it is failing more frequently on a particular workload.
It cannot conclusively reveal which exact weight file was loaded, which layers were quantised, which KV-cache format was used or whether a specific request ran on the advertised model. Even sophisticated fingerprinting remains statistical.
Research suggests that substantial model substitutions can often be detected under controlled conditions. More subtle changes, including high-quality quantisation, hidden reasoning limits and serving-stack modifications, are significantly harder to identify.
Passing an audit should therefore be described carefully. The strongest defensible statement is that the service was behaviourally consistent with the declared deployment during the tested period. It is not that the declared model and configuration were proven to have executed every request.
From statistical trust to verifiable inference
AI systems require a different model of trust from traditional deterministic software. Today, that trust is built primarily through provider reputation, contractual commitments, statistical testing, empirical observation and historical consistency.
Stronger technical guarantees will require additional infrastructure.
A provider could issue signed inference receipts that bind together the model identifier, configuration, request hash, response hash, timestamp and token usage. This would prove what the provider attested, although it would still require trust in the provider’s signing system.
Hardware-backed attestation could offer stronger evidence by proving that a particular runtime and model artifact were loaded in a trusted environment. For this to be meaningful, the attestation would need to cover the full execution path, including the model weights, GPU runtime, tokenizer, configuration, request and response.
Cryptographic systems may eventually make it possible to prove that some or all of an inference computation was performed correctly. Full verification of modern neural networks remains expensive, but selective verification of a sample of requests may become commercially practical. The team over at Equilibrium Research examine the progress of some of these approaches and more in their “The State of Verifiable Inference and Future Directions”
Inference marketplaces could also provide independent auditing. Rather than presenting a binary “verified” badge, they could distinguish between provider-declared identity, behavioural consistency, continuous monitoring, hardware attestation and per-request verification. That would communicate the strength of the available evidence more honestly.
The real product is the deployment
The AI industry often talks about models as though a model name uniquely defines the product. It does not.
The service delivered to the customer is the combination of model weights, quantisation, numerical precision, context handling, reasoning allocation, sampling configuration, system prompts, safety policies, serving software, hardware, routing, caching and post-processing.
The same model can be deployed in ways that produce very different performance, cost and reliability.
As infrastructure competition increases, providers will continue to find more sophisticated ways to reduce their cost per token. Most of those innovations will be beneficial. Some will quietly transfer cost savings from the provider into quality risk for the customer.
The challenge is not to prevent optimisation. It is to make the consequences of optimisation visible.
Until inference can be reliably attested, customers will remain dependent on statistical evidence, private evaluations, contractual disclosure and provider reputation. That is a weaker foundation than the cryptographic and formal guarantees available elsewhere in computing, but it is currently the most practical one.
An API model name is a claim, not proof. Consumers should evaluate the behaviour they actually receive. Providers should disclose the deployment they actually operate. Marketplaces should distinguish clearly between what has been declared, what has been statistically observed and what has been technically verified.
About Anera
Our goal is to continue to stay on top of the economics of inference. Every new model moves markets. We are lucky that we get to work with teams at the bleeding edge of frontier open source inference arming them with capital and compute to keep pace with the surging demand for tokens. If you are in the business of inference and need capital or frontier compute, feel free to reach out!
contact@aneralabs.xyz