In 2023, something curious happened across NLP labs worldwide. Researchers who once spent weeks recruiting human annotators to judge text quality began writing five-line prompts instead. "Act as an expert evaluator. Score the following summary for coherence, fluency, and factuality. Output only the scores."
The appeal was obvious. Human evaluation is slow, expensive, and notoriously difficult to standardize. Large language models, by contrast, are instant, nearly free at scale, and never complain about burnout. By 2024, LLM-as-a-judge had become the default evaluation method in thousands of papers. Need to compare two chatbots? Ask GPT-4. Need to score 10,000 generated headlines? Ask Claude. Need to check if your fine-tuned model beats the baseline? Ask an API.
However, a problem was hiding in plain sight. These evaluators are accessed through shared endpoints—the same APIs that power millions of other requests—and those APIs are not deterministic. The same prompt, the same model, and the same settings can produce different outputs on different calls.
This paper, Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints, asks a direct question: Can we trust these black-box observers to give us consistent measurements? The answer, based on a preregistered study with rigorous methodology, is uncomfortable: no, not reliably. The study didn't use sloppy prompts or poorly designed rubrics. It did everything right. And the measurements still fell apart.
A black-box LLM observer is any language model accessed through an API where the user cannot inspect or control the internal parameters. You send text, you get text back. The model's weights, architecture, and internal decoding logic are invisible—hence "black box."
When used as an evaluator, the LLM receives a rubric and an input (a generated summary, a dialogue, an article) and returns a judgment: a score, a classification, or a comparative ranking. The model "observes" the output and makes a measurement.
A shared endpoint is the API infrastructure that routes your request to a model. When you call gpt-4 through OpenAI's API, you're hitting a shared endpoint—the same one used by thousands of other developers simultaneously. The model you're using is not a single, isolated instance. It's a service distributed across clusters of GPUs, load-balanced, batched, and managed by infrastructure you can't see.
This matters because shared endpoints introduce variability that a local, controlled environment wouldn't have. Your request might be processed by different hardware, batched with different workloads, or handled by a slightly different model version—all without your knowledge.
Preregistration means the researchers wrote down their hypotheses, methods, and analysis plan before collecting any data. The document was timestamped and stored publicly. This prevents a range of questionable practices: changing the hypothesis after seeing results, selectively reporting favorable outcomes, or tweaking the analysis until something looks significant.
The fact that this study was preregistered is significant. It means the reliability failure wasn't discovered after the fact and framed as a finding. It was predicted as a possibility and then explicitly tested.
Reliability and validity are separate concepts, though they're often conflated.
Validity asks: Is this measurement actually measuring what it claims to measure? If an LLM says a summary is "factual," does that correlate with ground truth?
Reliability asks: If you measure the same thing twice, do you get the same result? A scale that gives you a different weight every time you step on it is unreliable, regardless of whether it's calibrated correctly.
This study focuses on reliability. That's the right starting point, because an unreliable measure cannot be valid. If the LLM gives different scores for identical inputs, it doesn't matter whether the scores are "right" on average—you can't trust any single measurement.
Two metrics dominate reliability research:
Intraclass Correlation Coefficient (ICC): Measures the consistency of ratings when multiple raters score multiple items. Values range from 0 to 1. Below 0.50 indicates poor reliability; 0.50–0.75 is moderate; 0.75–0.90 is good; above 0.90 is excellent.
Krippendorff's Alpha: A more flexible measure that handles multiple raters, missing data, and different measurement scales. The conventional threshold for acceptable reliability is 0.80. Below that, the measurement is considered too noisy to support confident conclusions.
Both metrics were used in this study. Both told the same story.
The researchers followed best practices from the start. They preregistered their hypotheses: (1) LLM observers would show below-threshold reliability across repeated runs on shared endpoints, and (2) the instability would persist even with deterministic settings like temperature=0. They specified their sample size, their evaluation tasks, and their statistical analysis plan.
The study used 1,000 generated texts drawn from multiple models and task types: summarization, dialogue response generation, and open-ended creative writing. Three different LLM observers were selected—two proprietary models accessed via shared endpoints (GPT-3.5 and GPT-4) and one open-weight model served through a commercial API (to keep all three on shared infrastructure).
The evaluation tasks were standard: holistic quality scoring on a 1–10 scale, pairwise comparisons between two texts, and binary classification (e.g., "is this response toxic?").
Here's where the study gets its name. The researchers did not cut corners. They used:
Every variable that could be controlled was controlled. If the measurement failed under these conditions, it would fail under anyone's conditions.
Each of the 1,000 texts was evaluated ten times by each observer, with all evaluations spread across a two-week period. This captured both short-term variability (calls made minutes apart) and longer-term drift (calls made days apart).
The researchers deliberately used the standard API parameters most researchers use: temperature=0.7 for the main condition, and temperature=0 for a secondary condition to test whether deterministic settings solve the problem.
The analysis plan was straightforward: compute ICC and Krippendorff's alpha for each observer-task combination across the ten repeated runs. Predefined thresholds: ICC below 0.75 would be considered inadequate; Krippendorff's alpha below 0.80 would be considered a failure.
The results were unambiguous. Across all three observers and all three task types, Krippendorff's alpha ranged from 0.42 to 0.71. None reached the 0.80 threshold. The ICC values were similarly poor, ranging from 0.38 to 0.65—at best "moderate," at worst "poor."
For context, human raters using the same rubrics typically achieve alpha values between 0.80 and 0.90. The LLM observers were significantly less consistent than the human annotators they were meant to replace.
Consider a concrete example from the study. One generated summary—identical text, identical prompt, identical API settings—received these scores across ten runs from GPT-4:
7, 8, 6, 7, 9, 5, 8, 7, 6, 8
The mean was 7.1, but the range was 5 to 9. For a study comparing two systems where one scores 7.1 and the other scores 6.8, this noise is catastrophic. The differences you're trying to detect are smaller than the measurement error.
In pairwise comparisons, the instability was even more pronounced. The same pair of texts was presented ten times, and the observer was asked which was better. In 34% of cases, the observer gave conflicting answers across runs—preferring text A in one run and text B in the next.
The study estimated that non-determinism accounted for roughly 15% of the total variance in evaluation scores. To put that in perspective: in most model comparison studies, the expected performance difference between a strong baseline and a SOTA system is less than 10%. The measurement noise is larger than the signal.
Here's the most counterintuitive finding. When the researchers set temperature=0—the setting universally recommended for "deterministic" output—the reliability improved only marginally. Krippendorff's alpha went from 0.58 to 0.63 on average. Still far below the 0.80 threshold.
Why? Because temperature is only one source of randomness. The API infrastructure itself introduces variability that no parameter setting can eliminate.
The gap between what researchers assume and what the study found is stark:
| Standard | Typical assumption | Study finding |
|---|---|---|
| Krippendorff's alpha ≥ 0.80 | Met with standard prompts | 0.42–0.71 |
| ICC ≥ 0.75 | Met with clear rubrics | 0.38–0.65 |
| Consistency at temp=0 | Near-perfect | Still below 0.80 |
Key Takeaway: The study's "clean engineering" produced the best-case scenario for LLM evaluators—standardized prompts, clear rubrics, repeated measures. Even then, the measurements were too unstable to support confident conclusions.
The most well-understood source of variability is the decoding process itself. LLMs don't "choose" the next token; they assign probabilities to a vocabulary of possible tokens and then sample from that distribution. Temperature controls the sharpness of the distribution but doesn't eliminate randomness entirely. At temperature=0, most APIs use greedy decoding, but not all do, and even greedy decoding can produce different results if the underlying computation isn't bitwise deterministic.
This is the hidden variable that most researchers don't consider. When you make an API call, your request is routed to one of potentially hundreds of GPU servers. These servers may have different hardware (A100s vs. H100s), different batch compositions at the moment of inference, and different software versions for the inference stack.
Floating-point operations are not perfectly consistent across hardware. A model running on one GPU might produce slightly different probability distributions than the same model on another GPU. These tiny differences get amplified through the sampling process.
API providers don't always tell you when they update their models. OpenAI, Anthropic, and others periodically update their models without changing the version string. The "gpt-4" you called in January might be a different model in March—slightly fine-tuned, with different evaluation behavior.
The study found that reliability was lower for evaluations conducted on different days than for evaluations conducted minutes apart, suggesting that model drift is a real contributor to instability.
The uncomfortable conclusion is that researchers who carefully design their prompts and rubrics are solving the wrong problem. No amount of prompt engineering can control what happens on the server side. You can't force the API to use a specific GPU, you can't disable batching, and you can't pin a model version.
The measurement tool is fundamentally outside your control.
Key Takeaway: Non-determinism isn't a bug you can engineer around. It's a property of the infrastructure. Clean engineering controls the input; it cannot control the processing.
Every paper that uses an LLM evaluator without reporting reliability metrics is potentially reporting noise as signal. Two researchers using the same LLM evaluator on the same data could reach different conclusions about which system performs better—not because of any real difference, but because of sampling noise.
The most common use case for LLM evaluators is comparing two or more systems. If the measurement noise is 15% and the expected performance gap is 10%, how many published conclusions are actually false? The study suggests this isn't a hypothetical concern. In pairwise comparisons, the observers contradicted themselves in a third of cases.
This isn't just an academic problem. Companies use LLM evaluators to:
If the evaluator is unreliable, every one of these decisions is compromised. A company might roll back a genuinely better update—or deploy a worse one—because the evaluator gave inconsistent scores.
This study joins a growing body of work questioning the reliability of LLM evaluators. Liu et al. (2023) showed that LLM judges exhibit position bias and self-preference bias. Wang et al. (2023) demonstrated that LLM judges are sensitive to prompt phrasing and model choice. This study adds a different dimension: even with everything held constant, the measurements are unstable.
The most direct mitigation is repeated sampling. Instead of a single evaluation per text, run three to five evaluations and average the results. This reduces variance by the square root of the number of runs. Five runs would cut the noise roughly in half.
The cost is real—five times the API expense—but for high-stakes evaluations, it's necessary.
Set temperature=0 where possible. It doesn't eliminate variability, but it reduces it. The study found that temperature=0 improved reliability by about 8%. Not enough to solve the problem, but enough to matter.
If you use an LLM evaluator, you should report its reliability. Calculate Krippendorff's alpha or ICC on a subset of your data by running the evaluation twice. If the reliability falls below acceptable thresholds, acknowledge it as a limitation.
This should become standard practice. Currently, it's almost never done.
For high-stakes conclusions—particularly claims about which model is "better"—consider supplementing LLM evaluation with:
LLM evaluators are useful tools, but they shouldn't be the only tool.
If you must rely on LLM evaluators, design your protocol to be robust to noise:
Key Takeaway: The goal isn't to eliminate non-determinism—that's impossible with shared endpoints. The goal is to measure its extent, report it honestly, and design studies that can tolerate it.
When LLM evaluators first appeared, they were greeted with enthusiasm. Here was a tool that could automate the most tedious part of NLP research. Initial studies showed strong correlations with human judgments, and the field adopted them rapidly.
However, as with any new tool, the early enthusiasm outpaced the evidence. The reliability problems documented in this study are not isolated incidents—they're symptoms of a broader issue. We've been using a measurement tool without properly characterizing its measurement properties.
This study sits alongside a growing literature:
The pattern is clear: LLM evaluators are powerful but fragile. They can be biased, manipulated, and—as this study shows—simply inconsistent.
What would standardized practice look like? At minimum:
These practices won't make LLM evaluators perfectly reliable, but they'll make the unreliability visible and manageable.
The tension is real. LLM evaluators enable research that would otherwise be impossible—evaluating thousands of outputs at scale, comparing models across languages, building automated feedback loops. Abandoning them entirely would be a step backward.
But adopting them uncritically is equally problematic. The field needs to treat LLM evaluators like any other measurement instrument: characterize their reliability, understand their biases, and use them within their known limitations.
This preregistered study did everything right. Clean prompts. Clear rubrics. Standardized procedures. Repeated measures. And the result was unambiguous: black-box LLM observers on shared endpoints produce unstable measurements. Krippendorff's alpha fell below acceptable thresholds across all conditions. ICC values indicated poor to moderate reliability. Temperature=0 helped marginally but didn't solve the problem.
The name of the paper says it all. You can engineer the evaluation protocol perfectly and still get unreliable measurements, because the instability isn't in your protocol—it's in the infrastructure you can't control.
The fix isn't technical; it's cultural. Researchers need to treat LLM evaluators with the same skepticism they'd apply to any other measurement tool. That means testing reliability, reporting it honestly, and designing studies that acknowledge the noise floor.
LLM evaluators aren't going away, and they shouldn't. They're too useful for that. However, their utility depends on understanding their limits. This study provides a clear picture of those limits: with shared endpoints and black-box access, reliability is poor to moderate at best.
The path forward is not to abandon LLM evaluators but to use them with appropriate caution—and to demand that every paper using them reports the reliability evidence that would let readers judge whether the measurements can be trusted.
Key Takeaway: A measurement tool that can't measure consistently isn't measuring—it's guessing. Until LLM evaluators are proven reliable under the conditions they're used, every conclusion based on them carries an asterisk.
The paper found that black-box LLM evaluators accessed through shared API endpoints produce unstable measurements. When the same text was evaluated multiple times under identical conditions, the scores varied significantly. Reliability metrics (ICC and Krippendorff's alpha) fell below acceptable thresholds across all conditions tested.
Reliability is a prerequisite for validity. If an evaluator gives different scores for the same input, you can't trust any single score. This is particularly problematic for model comparisons, where the measurement noise can exceed the actual performance differences between systems.
Preregistration means the researchers specified their hypotheses, methods, and analysis plan before collecting data. The document was timestamped and publicly available. This prevents questionable research practices like changing hypotheses after seeing results or selectively reporting favorable outcomes.
Shared endpoints are the API infrastructure that routes user requests to LLM models. When you call GPT-4 through OpenAI's API, you're using a shared endpoint—the same infrastructure used by thousands of other developers. This shared infrastructure introduces variability through load balancing, batching, hardware differences, and model updates.
Researchers can run evaluations multiple times and average the scores, use temperature=0 (which helps but doesn't solve the problem), report reliability metrics in their studies, supplement LLM evaluation with human judgments or traditional metrics, and design protocols that are robust to measurement noise.
No. The study found that temperature=0 improved reliability only marginally. This is because variability comes not just from sampling randomness but also from infrastructure factors—hardware differences, batching, and model versioning—that no parameter setting can control.
The findings threaten the reproducibility of NLP research. If evaluation measurements are unstable, published conclusions about model performance may be based on noise rather than signal. The study calls for researchers to report reliability metrics and design studies that account for measurement variability.
Reliability refers to consistency: if you measure the same thing twice, do you get the same result? Validity refers to accuracy: does the measurement actually measure what it claims to measure? An unreliable measure cannot be valid, because you can't trust any single measurement.
Alternatives include human evaluation (which has its own reliability challenges but is well-studied), traditional automatic metrics like BLEU and ROUGE (which measure specific aspects of text quality), task-specific metrics, and hybrid approaches that combine LLM evaluation with human spot-checking.
The paper is titled "Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints." Check preprint servers like arXiv, as well as the proceedings of major NLP and AI conferences for the published version.
Read the full paper to dive deeper into the data and methodology, and share your own experiences with LLM evaluators in the comments below.