The Science of Calibrated Trust

The Overconfidence Problem

Large language models are systematically overconfident. When a model says 95% confidence, the answer is right about half the time. This is not a rounding error. It breaks every system downstream that trusts those numbers.

The gap is worse than people think. GPT-4o mini self-reports 90-100% confidence on entity extractions where actual accuracy sits around 35%. Single judge models assign confidence 1.0 to 86% of entities they extract, but only 40% of those entities are correct. This is not noise. It is structural.

Published guard models are no better. Llama Guard, Shield Guard, and Wild Guard, the standard open-source safety classifiers, have Expected Calibration Error (ECE) of 14-28% [2]. ECE measures the average gap between predicted confidence and observed accuracy. An ECE of 20% means the model's confidence is, on average, 20 points away from reality:

If you know how to set a threshold, you set a threshold. But if your model's confidence scores bear no relationship to correctness, you are filtering on noise. You will miss errors the model is confidently wrong about and flag correct outputs it happens to be uncertain about. The threshold gives the appearance of quality control while providing none.

There is also a selection bias that compounds the problem. Extraction models do an implicit top-K filtering before confidence is ever estimated: they only output entities they believe are correct. The confidence model never sees clearly wrong extractions. It evaluates a pre-filtered distribution where most inputs are plausible. A perfect extractor paired with a perfect confidence model would just output "correct" for everything, trivially calibrated but completely uninformative. The practical consequence: precision can be improved after the fact through calibration, but recall cannot. If the extraction model suppresses uncertain outputs before they reach the confidence estimator, those potential corrections are lost permanently.

RLHF makes this worse. The GPT-4 technical report documents calibration degradation after human feedback training [5]. RLHF optimizes for outputs that sound confident and authoritative, because human annotators reward those qualities. The model learns to express certainty as rhetoric, decoupled from actual accuracy. This is not a bug in RLHF. It is a direct consequence of optimizing for human preference rather than calibrated truthfulness.

Three Methods for Confidence Estimation

The methods divide by access level: how much of the model's internals you can see. Each level trades deployment complexity for calibration quality. Which method you use depends on how much access you have to the model's internals.

Knowledge Graph Uncertainty Propagation

Extraction produces entities and relations, each with a confidence score. These scores are not independent. A relation between two entities depends on both entities being correct. The joint probability follows a simple chain: $P(\text{relation} \mid e_i, e_j) \cdot P(e_i) \cdot P(e_j)$. If either entity is wrong, the relation built on them is almost certainly wrong too.

Factor graphs [8][9] provide the framework for modeling these dependencies. A factor graph is a bipartite graph connecting variable nodes (entities, relations) to factor nodes (the functions defining how variables depend on each other). It generalizes Bayesian networks to handle arbitrary dependency structures without requiring a directed acyclic graph.

Our extraction confidence work uses evaluation metrics (CEAF [7]) and datasets from the AdaIE framework [1], which spans 15 datasets, 497 entity types, and 747 relation types, as the testing ground for calibration methods.

Why extraction is different from other NLP tasks

The obvious question: how is confidence for information extraction different from translation, sentiment analysis, or any other NLP task? The answer is the topology of dependencies. In question answering, questions are independent. In translation, dependencies are autoregressive. In information extraction, the dependency structure is a graph. Relations depend on entities. Entities depend on each other through relations. Corrections propagate non-locally through the graph in ways that sequential dependencies never do.

Entity types function as predicates over the universe of extracted mentions. "Software engineer" is a function: pass in a text span, it returns a probability. This connects extraction to Laplace's framework where probability extends logic. The predicate returns not true or false but a value between 0 and 1. Schema-constrained extraction makes this tractable: with known entity types, extraction decomposes into individual classification decisions (does this span have type X?) rather than unconstrained generation.

Belief propagation

When a human corrects one entity, confirming it or marking it wrong, messages propagate through the factor graph to update neighboring beliefs. Confirm entity A and every relation involving it gets a confidence boost. Reject entity A and every relation involving it drops. The update is local: you only re-evaluate the Markov blanket of the corrected node, not the entire graph.

Active correction ordering

The order in which you correct entities matters. Correcting a highly-connected entity first propagates information to more neighbors. Same insight as active learning and Bayesian optimization: choose the query that maximizes expected information gain across the graph, not just locally. A single well-chosen correction can shift confidence scores for dozens of downstream relations.

Active Learning and Human-in-the-Loop

You cannot query a human for every extraction. Humans are expensive and slow. The problem is a multi-objective contextual bandit: minimize calibration error while minimizing the number of human queries. These objectives conflict. More human feedback always improves calibration, but you have a fixed budget.

The acquisition function is expected information gain. For each candidate entity, estimate how much total uncertainty reduction you would get from a human label. Entities with confidence near 0.5 are not automatically the best candidates. An entity at 0.5 confidence with 50 downstream relations is worth more than an entity at 0.5 confidence with zero downstream relations. The graph structure determines query value.

Even at high confidence, the system should occasionally verify for systematic drift. A model that was 95% accurate last month may have degraded. Periodic spot-checks on high-confidence outputs detect this before it compounds. This is the exploration-exploitation tradeoff: mostly exploit your confidence estimates to focus human effort on uncertain items, but explore enough to detect when the estimates themselves have gone stale.

The Bayesian extension replaces point estimates with full distributions. Instead of "this entity has confidence 0.73," the system maintains a $\text{Beta}(14, 5)$ distribution with mean 0.74 and 95% credible interval [0.55, 0.89]. Distributions capture not just the model's best guess but how much evidence backs it. An entity with confidence 0.73 from three observations is genuinely less certain than one with confidence 0.73 from three hundred observations, even though the point estimates are identical.

The Safety Application

The same calibration architecture applies directly to agent safety. "Is this entity correct?" and "Is this action safe?" have identical mathematical structure. Both require a classifier, both suffer from overconfidence, both benefit from ensembles, and both need calibrated confidence to make threshold decisions meaningful.

Multi-specialist ensemble

Instead of a single monolithic safety model, the system deploys per-category specialists: one for privacy violations, one for financial risk, one for security threats, one for ethical concerns. Each specialist is a small model fine-tuned on category-specific data. This mirrors the black-box ensemble from confidence estimation, but with domain specialization.

Specialist diversity improved detection recall from 63% (single general judge) to 73% (four specialists). The improvement comes not from more parameters but from more perspectives. A general safety model treats "agent emails SSH keys to an external address" and "agent provides unsolicited investment advice" as the same class of problem. A specialist architecture treats them as fundamentally different failure modes requiring different training data, evaluation criteria, and confidence thresholds.

Turn off individual specialists and you see their value directly. Without the finance specialist, unauthorized purchase actions pass through undetected. Without the privacy specialist, credential exfiltration slips by. Each specialist catches failures the others miss, because each is trained on a different slice of the risk taxonomy.

Benchmark results

On the R-Judge benchmark [3], approximately 500 human-annotated instances across five categories (finance, IoT, software, web, program) with 27 distinct risk scenarios, the multi-specialist ensemble achieves 66% recall for unsafe actions and 62% specificity, with ECE of 9%. This matches GPT-4's recall on the same benchmark using only small open-weight models. The 9% ECE means confidence scores are off by no more than 10 percentage points at any confidence level, compared to 14-28% for published guard models [2].

The ensemble used four models (Llama, Qwen, Gemma, Mistral) with two prompt variants each, producing eight independent judgments per input. Confidence scores exist only in the 0.5-1.0 range by construction, since majority vote is always at least 50%. Variation comes from model architecture, prompt formulation, and chain-of-thought vs. direct prompting.

LoRA adapter distillation

Cloud-based specialist judges are expensive to run in production. The distillation pipeline compresses them into LoRA adapters on a single small backbone model (Qwen 0.6B or Llama 1B). Each adapter is 5-25MB. Multiple adapters share one base deployment, making per-specialist inference cheap.

The pipeline has four phases: (1) synthetic data generation from 10-50 seed examples per category using frontier models, (2) teacher evaluation to validate the task is solvable at the target accuracy, (3) LoRA fine-tuning on the small backbone, (4) integration into the gateway. This pipeline is task-agnostic. The same four phases apply whether the target is safety detection, entity extraction, PII redaction, or any classification task. The same four-phase pipeline works for any classification task.

Published benchmarks from comparable distillation approaches (Distill Labs, Berlin) show that 1-4B parameter models can match 120B model accuracy on targeted classification tasks. Their PII redaction benchmark shows a 1B model matching GPT-4. Turnaround is approximately 8 hours at approximately 100 USD per distillation.

Conservative aggregation

The aggregation rule is deliberately asymmetric. If any specialist flags an action, the action is flagged. The highest confidence among flagging specialists is reported. If all specialists say safe, the minimum confidence is reported. This is conservative by design. False positives (blocking safe actions) are recoverable: the user can override or the system can escalate. False negatives (allowing unsafe actions) are not recoverable. The cost function is asymmetric, so the aggregation must be.

When multiple specialists flag the same action, combined unsafety scores climb rapidly via the product-of-complements formula: $P(\text{unsafe}) = 1 - \prod_i (1 - P(\text{unsafe}_i))$. Individual scores of 0.8, 0.9, and 0.85 yield a combined score of 0.997. The independence assumption is imperfect (deployment safety and secret leakage are correlated), but it works in practice for the same reason naive Bayes works for spam: the model's errors are tolerable because the aggregation is conservative.

For more principled handling of policy dependencies, you can apply the chain rule over a directed acyclic graph of policy relationships: $P(A, B \mid X) = P(A \mid X) \cdot P(B \mid A, X)$. An LLM can generate the dependency graph given a set of policies, enabling joint probability computation that accounts for correlations. This is the standard approach in probabilistic graphical models [8], but is unnecessary for initial deployment where naive independence suffices.

The mixture-of-opinions architecture

A related approach, inspired by Sanskrit epistemological traditions, organizes reasoning dimensions into specialized perspectives. The concept draws from a framework of approximately 208 reasoning dimensions derived from classical Indian schools of thought. For a given domain or task, a subset of these dimensions (typically 8-12, grouped into clusters) is compiled into specialized critics. Each dimension acts as a thinking hat: fallacy detection, irreversibility checks, sequencing analysis, causal reasoning, temporal consistency, logical coherence.

The architectural distinction from the ensemble is how the critics communicate. In the standard ensemble, each judge produces a text output and votes are aggregated. In the mixture-of-opinions architecture, critics share an embedding space. Queries flow to all critics as vectors in a common latent space, critics debate at the embedding level rather than through text, and synthesis happens in high-dimensional space before decoding back. This reduces context loss compared to text-based inter-model communication, where nuance is compressed through the tokenizer at every exchange.

The multi-stage synthesis: parallel debate (all critics analyze simultaneously), reconciliation (common conclusions drawn), contradiction resolution (disagreements adjudicated), then final answer generation. The underlying models can be open-source LLMs with LoRA adapters for each dimension, making the approach feasible on commodity hardware.

The critical difference: the mixture-of-opinions architecture does not produce calibrated confidence scores. It produces better answers, but cannot tell you how confident it is in those answers. The ensemble approach produces calibrated confidence as a first-class output. Combining the two, diverse perspectives communicating in latent space with calibrated uncertainty estimation on top, is an open problem worth solving.

Per-organization calibration

Safety thresholds are not universal. What counts as unsafe depends on the organization, the domain, the regulatory environment, and the specific workflow. A marketing agency and a healthcare provider have fundamentally different risk profiles. Calibration must be per-organization.

Every approve/deny decision by a human reviewer becomes training data for that organization's calibration model. Over time, the system learns their specific risk tolerance, common edge cases, and policy interpretations. The longer an organization uses the system, the better it calibrates to their needs. The data dependency that initially looks like a weakness (you need organization-specific data to calibrate) turns into a lock-in effect. No competitor can replicate another organization's accumulated calibration data.

Information Geometry and Fine-Tuning

Model knowledge can be understood as a high-dimensional manifold in parameter space. Points on this manifold represent probability distributions over outputs. The geometry of this manifold, its curvature and geodesics, determines how fine-tuning moves through distribution space.

Two kinds of geodesics matter. E-geodesics (exponential family) preserve mean parameters during interpolation. M-geodesics (mixture) preserve natural parameters. The choice between them determines what is conserved during fine-tuning and what drifts. Standard gradient descent follows neither. It follows Euclidean straight lines in parameter space, which are curved paths in distribution space.

Natural gradient descent operates in Fisher information space rather than Euclidean parameter space. The Fisher information matrix measures how sensitive the model's output distribution is to each parameter. Moving in the natural gradient direction makes equal-sized changes in distribution space, regardless of how the parameters happen to be scaled. This connects directly to catastrophic forgetting: standard fine-tuning can destroy previously learned distributions because Euclidean steps in parameter space correspond to wildly unequal steps in distribution space.

Programs as Weights

Natural language instructions can be compiled into LoRA adapters: small weight modifications (approximately 5-25MB) that encode behavioral programs into a base model. The base model can be as small as GPT-2 124M or Qwen 0.6B. These adapters run in the browser via WebAssembly with no server round-trip.

This is policy enforcement at the weight level. Instead of prompting a model with safety rules at inference time (which can be bypassed), you compile the rules into the model's weights. The policy is not an input that can be jailbroken. It is part of the model itself.

The gap is calibration. Programs-as-weights can enforce behaviors but cannot currently report how confident they are in their enforcement. A compiled policy adapter can block an action but cannot say "I am 73% sure this action violates the policy." Connecting the calibration methods described above to compiled policy adapters is an open problem. Solve it and you get tamper-resistant policies that can also tell you how sure they are.

References

1. Xu, J. et al. "AdaIE: Guidelines for Universal Information Extraction." Anonymous ACL submission, 2026. 15 datasets, 497 entity types, 747 relation types, CEAF-based evaluation metrics.
2. "On the Calibration of LLM-based Guard Models for Content Moderation." 2026. Llama Guard, Shield Guard, Wild Guard ECE of 14-28%.
3. R-Judge benchmark. ~500 human-annotated instances, 5 categories (finance, IoT, software, web, program), 27 risk scenarios.
4. "Calibrating LLM Judges: A Linear Probe for Fast Reliable Uncertainty Estimation." Linear probe achieving ~0.02 ECE on MMLU Pro.
5. OpenAI. "GPT-4 Technical Report." 2023. Documents calibration degradation after RLHF.
6. Schellhammer & Blazer, Vector Institute. "Asymmetric Viewless Sidekicks Improve Uncertainty."
7. Luo, X. "On Coreference Resolution Performance Metrics." 2005. CEAF metric used in AdaIE evaluation.
8. Bishop, C. Pattern Recognition and Machine Learning. Chapter 8: Graphical Models. Factor graphs, belief propagation.
9. Frey, B. et al. "Factor graphs and the sum-product algorithm." Co-inventor of factor graphs, founder of Deep Genomics.
10. "High Fidelity Information Extraction from Black Box LLMs" (Safe Passage). Three-step pipeline with fuzzy alignment.
11. "Towards Trustworthy Knowledge Graph Reasoning." Conformal prediction for knowledge graph retrieval paths.