The Science of Calibrated Trust

The Overconfidence Problem

Language models can express more confidence than their accuracy warrants. The size of that gap depends on the model, task, prompt, and evaluation data. Confidence must be measured in the intended setting before it is used to route consequential work.

The extraction experiments summarized here reported large gaps between self-reported confidence and observed correctness. Those figures describe particular experimental conditions, not a universal error rate for language models. Reproducible comparisons need the dataset split, model and prompt versions, labeling rules, and evaluation code.

Liu et al.'s ICLR 2025 study evaluated nine guard models across twelve benchmarks [2]. WildGuard had the lowest average prompt-classification Expected Calibration Error (ECE), at 14.4%. ECE is a weighted average gap between confidence and accuracy across bins; it does not bound the error of an individual prediction:

A poorly calibrated score makes a numerical threshold difficult to interpret. It may still rank some outputs usefully, which is a separate property to measure. Evaluation should check both ranking quality and calibration, including errors the model makes with high confidence. A system that is certain and wrong can steer attention away from the very mistake a reviewer needs to catch.

There is also a selection bias that compounds the problem. Extraction models do an implicit top-K filtering before confidence is ever estimated: they only output entities they believe are correct. The confidence model never sees clearly wrong extractions. It evaluates a pre-filtered distribution where most inputs are plausible. A perfect extractor paired with a perfect confidence model would just output "correct" for everything, trivially calibrated but completely uninformative. The practical consequence: precision can be improved after the fact through calibration, but recall cannot. If the extraction model suppresses uncertain outputs before they reach the confidence estimator, those potential corrections are lost permanently.

Training that changes a model's behavior can also change its calibration. The GPT-4 technical report [5] reports degradation after post-training in its evaluation. That observation does not establish a universal mechanism or rate. Calibration should be rechecked after changes to training, prompts, or deployment conditions.

Three Methods for Confidence Estimation

The methods divide by access level: how much of the model's internals you can see. Each level trades deployment complexity for calibration quality.

Knowledge Graph Uncertainty Propagation

Extraction produces entities and relations whose correctness can be dependent. The chain rule is $P(r,e_i,e_j)=P(r\mid e_i,e_j)P(e_j\mid e_i)P(e_i)$. Replacing the conditional entity probability with $P(e_j)$ assumes independence and should be justified rather than silently applied. Here the events denote correctness of the extracted items.

Factor graphs [8][9] provide the framework for modeling these dependencies. A factor graph is a bipartite graph connecting variable nodes (entities, relations) to factor nodes (the functions defining how variables depend on each other). It generalizes Bayesian networks to handle arbitrary dependency structures without requiring a directed acyclic graph.

Our extraction confidence work uses evaluation metrics (CEAF [7]) and datasets from the AdaIE framework [1], which spans 15 datasets, 497 entity types, and 747 relation types, as the testing ground for calibration methods.

Why extraction is different from other NLP tasks

The obvious question: how is confidence for information extraction different from translation, sentiment analysis, or any other NLP task? The answer is the topology of dependencies. In question answering, questions are independent. In translation, dependencies are autoregressive. In information extraction, the dependency structure is a graph. Relations depend on entities. Entities depend on each other through relations. Corrections propagate non-locally through the graph in ways that sequential dependencies never do.

Entity types function as predicates over the universe of extracted mentions. "Software engineer" is a function: pass in a text span, it returns a probability. This connects extraction to Laplace's framework where probability extends logic. The predicate returns not true or false but a value between 0 and 1. Schema-constrained extraction makes this tractable: with known entity types, extraction decomposes into individual classification decisions (does this span have type X?) rather than unconstrained generation.

Belief propagation

A correction changes the evidence available to a probabilistic graph. Messages can propagate through connected factors, potentially changing distant beliefs. The direction and size of those changes depend on the factors; confirming one entity need not increase every related probability. Exactness and convergence depend on graph structure and the inference algorithm.

Active correction ordering

The order in which you correct entities matters. Correcting a highly-connected entity first propagates information to more neighbors. Same insight as active learning and Bayesian optimization: choose the query that maximizes expected information gain across the graph, not just locally. A single well-chosen correction can shift confidence scores for dozens of downstream relations.

Active Learning and Human-in-the-Loop

Human review has a limited budget. One research formulation is to select labels that are expected to improve the system most, while accounting for review cost and consequences. More feedback helps only when its quality, relevance, and use in training support that conclusion. Keep an independent evaluation set rather than treating every approval as ground truth.

The acquisition function is expected information gain. For each candidate entity, estimate how much total uncertainty reduction you would get from a human label. Entities with confidence near 0.5 are not automatically the best candidates. An entity at 0.5 confidence with 50 downstream relations is worth more than an entity at 0.5 confidence with zero downstream relations. The graph structure determines query value.

Even at high confidence, the system should occasionally verify for systematic drift. A model that was 95% accurate last month may have degraded. Periodic spot-checks on high-confidence outputs detect this before it compounds. This is the exploration-exploitation tradeoff: mostly exploit your confidence estimates to focus human effort on uncertain items, but explore enough to detect when the estimates themselves have gone stale.

The Bayesian extension replaces point estimates with full distributions. Instead of "this entity has confidence 0.73," the system maintains a $\text{Beta}(14, 5)$ distribution with mean 0.74 and 95% credible interval [0.55, 0.89]. A distribution captures both the model's best guess and how much evidence backs it. An entity with confidence 0.73 from three observations is genuinely less certain than one with confidence 0.73 from three hundred observations, even though the point estimates are identical.

The Safety Application

Calibration methods can help estimate whether an action meets a specified safety criterion. Defining that criterion remains a separate governance problem. A classifier can model judgments under a policy; it cannot settle whether the policy is legitimate, whether the available evidence is complete, or whose interests it should protect.

Multi-specialist ensemble

Instead of a single monolithic safety model, the system deploys per-category specialists: one for privacy violations, one for financial risk, one for security threats, one for ethical concerns. Each specialist is a small model fine-tuned on category-specific data. This mirrors the black-box ensemble from confidence estimation, but with domain specialization.

Specialist diversity improved detection recall from 63% (single general judge) to 73% (four specialists). The improvement comes from more perspectives, not more parameters. A general safety model treats "agent emails SSH keys to an external address" and "agent provides unsolicited investment advice" as the same class of problem. A specialist architecture treats them as fundamentally different failure modes requiring different training data, evaluation criteria, and confidence thresholds.

Turn off individual specialists and you see their value directly. Without the finance specialist, unauthorized purchase actions pass through undetected. Without the privacy specialist, credential exfiltration slips by. Each specialist catches failures the others miss, because each is trained on a different slice of the risk taxonomy.

Benchmark results

The experiment summarized in this draft reports 66% recall for unsafe actions, 62% specificity, and 9% ECE on R-Judge [3]. These are experimental results, not production safety guarantees. An ECE of 9% is a weighted average gap across confidence bins; individual bins or groups may have much larger gaps. It does not mean every score is within ten percentage points of the true rate. Comparisons with other systems require matching evaluation conditions and uncertainty estimates. See Guo et al. on ECE and maximum calibration error.

The reported ensemble uses four model families with two prompt variants each, producing eight judgments. These judgments may be correlated. The vote share for the winning class lies between 0.5 and 1.0, but the unsafe-class vote share can range from 0 to 1. Neither quantity is automatically a calibrated probability of harm.

LoRA adapter distillation

Cloud-based specialist judges are expensive to run in production. The distillation pipeline compresses them into LoRA adapters on a single small backbone model (Qwen 0.6B or Llama 1B). Each adapter is 5-25MB. Multiple adapters share one base deployment, making per-specialist inference cheap.

The pipeline has four phases: (1) synthetic data generation from 10-50 seed examples per category using frontier models, (2) teacher evaluation to validate the task is solvable at the target accuracy, (3) LoRA fine-tuning on the small backbone, (4) integration into the gateway. This pipeline is task-agnostic. The same four phases apply whether the target is safety detection, entity extraction, PII redaction, or any classification task.

Published benchmarks from comparable distillation approaches (Distill Labs, Berlin) show that 1-4B parameter models can match 120B model accuracy on targeted classification tasks. Their PII redaction benchmark shows a 1B model matching GPT-4. Turnaround is approximately 8 hours at approximately 100 USD per distillation.

Conservative aggregation

An aggregation rule may favor escalation when any specialist raises a concern. That is a policy choice whose costs need evaluation. Blocking a legitimate urgent action can also cause harm, and some permitted actions are reversible. Thresholds should reflect both kinds of error, available review capacity, and the specific consequences. An escalation score should not be presented as a calibrated probability unless it has been validated as one.

For independent events, $1-\prod_i(1-p_i)$ gives the probability that at least one occurs. Applying that formula to correlated opinions about the same action can inflate confidence. Treat it as a heuristic unless its assumptions are supported, and evaluate or calibrate the aggregate on held-out outcomes. More agreement is not automatically more independent evidence.

A probabilistic graph can represent dependencies if its variables, conditional probabilities, and structure are supported by evidence. The chain rule $P(A,B\mid X)=P(A\mid X)P(B\mid A,X)$ does not itself supply those estimates. A graph proposed by a language model is a hypothesis to test, not a validated causal or probabilistic model.

The mixture-of-opinions architecture

A related approach, inspired by Sanskrit epistemological traditions, organizes reasoning dimensions into specialized perspectives. The concept draws from a framework of approximately 208 reasoning dimensions derived from classical Indian schools of thought. For a given domain or task, a subset of these dimensions (typically 8-12, grouped into clusters) is compiled into specialized critics. Each dimension acts as a thinking hat: fallacy detection, irreversibility checks, sequencing analysis, causal reasoning, temporal consistency, logical coherence.

One proposed distinction is how critics communicate: through text or through learned representations in a shared latent space. Different models do not automatically share a compatible representation space. Alignment, information preservation, interpretability, and actual quality gains need to be demonstrated rather than inferred from the use of vectors.

The multi-stage synthesis: parallel debate (all critics analyze simultaneously), reconciliation (common conclusions drawn), contradiction resolution (disagreements adjudicated), then final answer generation. The underlying models can be open-source LLMs with LoRA adapters for each dimension, making the approach feasible on commodity hardware.

The proposed mixture-of-opinions system needs evaluation of answer quality as well as uncertainty. Neither latent communication nor text-based voting automatically produces calibrated confidence. Comparing these architectures on the same tasks, including correlated errors and review costs, is an open research direction.

Per-organization calibration

Safety thresholds depend on the task, stakes, and applicable policy. Calibration estimates predictive reliability; choosing an acceptable risk threshold is a separate decision. Organization-specific evidence may help, but the evaluation should also examine relevant populations and use cases so a good aggregate score does not hide uneven performance.

Review decisions can provide organization-specific evidence, including disagreements and later corrections. The organization should retain usable access to its permitted labels, policy history, and evaluation records, with documented export formats and privacy controls. A replacement provider should be able to rebuild calibration from that evidence where technically and legally feasible. AceTeam should earn retention through useful service and interoperability, rather than make accumulated learning a barrier to leaving.

Information Geometry and Fine-Tuning

Model knowledge can be understood as a high-dimensional manifold in parameter space. Points on this manifold represent probability distributions over outputs. The geometry of this manifold, its curvature and geodesics, determines how fine-tuning moves through distribution space.

Information geometry offers ways to study changes in distributions rather than only distances between parameter vectors. Translating that perspective into better fine-tuning is a research question: the parameterization, statistical model, geometry, and objective must be specified before claims about preserved knowledge can be tested.

Natural gradient descent operates in Fisher information space rather than Euclidean parameter space. The Fisher information matrix measures how sensitive the model's output distribution is to each parameter. Moving in the natural gradient direction makes equal-sized changes in distribution space, regardless of how the parameters happen to be scaled. This connects directly to catastrophic forgetting: standard fine-tuning can destroy previously learned distributions because Euclidean steps in parameter space correspond to wildly unequal steps in distribution space.

Programs as Weights

Natural language instructions can be compiled into LoRA adapters: small weight modifications (approximately 5-25MB) that encode behavioral programs into a base model. The base model can be as small as GPT-2 124M or Qwen 0.6B. These adapters run in the browser via WebAssembly with no server round-trip.

Encoding a policy into an adapter may change behavior and reduce dependence on an editable prompt. It does not prove compliance or immunity to adversarial inputs. Learned detectors should operate alongside permissions, isolation, and approval gates enforced outside the acting model.

A policy adapter needs evaluation for both detection quality and calibration, including adversarial inputs and changes in deployment context. Versioned weights can support reproducibility; they do not make policy judgments infallible or eliminate the need for independent controls. Combining compact detectors with measured uncertainty remains a research direction.

References

1. Xu, J. et al. "AdaIE: Guidelines for Universal Information Extraction." Anonymous ACL submission, 2026. 15 datasets, 497 entity types, 747 relation types, CEAF-based evaluation metrics.
2. Liu, H. et al. "On Calibration of LLM-based Guard Models for Reliable Content Moderation." ICLR 2025. Table 1 reports calibration by model, task, and dataset.
3. R-Judge benchmark. ~500 human-annotated instances, 5 categories (finance, IoT, software, web, program), 27 risk scenarios.
4. Radharapu, B. et al. "Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation." December 2025. Probe-based calibration evaluated across models and judging tasks.
5. OpenAI. "GPT-4 Technical Report." 2023. Documents calibration degradation after RLHF.
6. Schellhammer & Blazer, Vector Institute. "Asymmetric Viewless Sidekicks Improve Uncertainty."
7. Luo, X. "On Coreference Resolution Performance Metrics." 2005. CEAF metric used in AdaIE evaluation.
8. Bishop, C. Pattern Recognition and Machine Learning. Chapter 8: Graphical Models. Factor graphs, belief propagation.
9. Frey, B. et al. "Factor graphs and the sum-product algorithm." Co-inventor of factor graphs, founder of Deep Genomics.
10. "High Fidelity Information Extraction from Black Box LLMs" (Safe Passage). Three-step pipeline with fuzzy alignment.
11. "Towards Trustworthy Knowledge Graph Reasoning." Conformal prediction for knowledge graph retrieval paths.