Probability Calibration, Decision Thresholds, and Abstention
Ten consecutive outputs of 0.99 do not establish reliability. First identify the event being predicted, then compare predictions with observed frequencies across similar cases. Classification accuracy, calibration, and whether to execute an action are related but different questions.
A probability needs a defined event
Suppose a fictional classifier predicts that a support message needs human handling, with p = 0.8. Good calibration means that among messages from data resembling the deployment environment and assigned probabilities near 0.8, approximately 80% actually need human handling. It neither guarantees this particular prediction nor means the model understood 80% of the content. The scikit-learn calibration guide explains this frequency interpretation.
In multiclass systems, confidence may mean the largest class probability. Elsewhere it may denote entropy, a margin, or a separate prediction head. Read the definition first: concentrating all probability on a wrong choice can still produce high confidence. Choice probabilities in Jev and Laya must be interpreted within their interfaces and target tasks rather than assigned a common meaning based only on field names.
Open full-size imageIn the upper plot, perfect calibration lies on the diagonal. At a given predicted probability, a curve below it indicates fewer positives than predicted. The lower panels show how many predictions occupy each range. These are results on one example dataset, not a universal ranking of classifiers.
Accuracy and confidence can disagree
Imagine 100 independently labeled examples. The model assigns 0.9 to every selected class but is correct only 70 times. Accuracy is 70%; average confidence is 90%. Changing every confidence to 0.7 can leave all selected classes unchanged while improving this group's frequency agreement.
Conversely, always predicting the dataset's positive rate may be calibrated overall while failing to distinguish individual cases. Examine discrimination as well as calibration. Guo et al. studied overconfidence in neural networks and temperature scaling of their outputs.
A positive temperature T replaces softmax(z) with softmax(z / T) (how temperature reshapes a softmax is covered in softmax regression). One shared positive temperature preserves class ordering but changes concentration. Fit it on calibration data rather than lowering confidence by intuition. It cannot repair missing candidates, incorrect labels, or an unlearned task.
Find the probability before calibrating an LLM
When a large model classifies or routes, it does not hand you a p ready to calibrate. There are three common sources, each less reliable than the one before.
The first is token probability. When the model outputs only a label (say yes or no), the exponential of that label token's log probability (logprob) is the probability the model gave it. OpenAI's Chat Completions API returns each output token's log probability with logprobs, and top_logprobs lists up to 20 candidates at each position; the same page notes that parameter support differs by model, particularly for newer reasoning models. A Gemini API response carries avgLogprobs and logprobsResult fields. If the label is split into several tokens, or the model writes an explanation before the label, combine the probabilities of the relevant tokens, or restrict the output format to the label alone.
The second is sampling agreement. Sample the same question N times at a nonzero temperature and use the share of the most common answer as its score. The self-consistency method picks the answer by majority vote in this way; if 8 of 10 samples say yes, the score is 0.8. It works with any API, at N times the number of calls.
The third is asking the model to report "confidence: 0.8" itself. Xiong et al. (2023) found that large models tend to be overconfident when they verbalise their confidence, and that agreement across several answers mitigates part of this. So treat a self-reported number as an uncalibrated score.
All three give scores, not yet probabilities. Before setting a threshold on any of them, split the data as the next section describes, draw a reliability diagram on a labelled calibration set, and fit a mapping when needed.
Assign separate roles to data
Training data learns the model; calibration data fits the probability mapping; validation data can select thresholds and abstention rules; a final test set evaluates the fixed system. Cross-validation can help with scarce data, but labels evaluating a prediction must not have participated in fitting or selecting that prediction.
Near-duplicate messages from one customer, or passages from one document, should not be scattered randomly across sets and treated as independent examples. A future deployment may require a temporal split. If the positive rate changes after deployment, old probabilities and thresholds can fail even when ranking remains good.
A reliability diagram compares mean probability with observed frequency in bins and should show bin counts. ECE summarizes bin gaps but depends on binning and sample size. Brier score and log loss evaluate overall probabilistic predictions; neither alone proves better calibration. This fictional binary example computes Brier score:
probabilities = [0.9, 0.8, 0.7, 0.1]
labels = [1, 1, 0, 0]
brier = sum((p - y) ** 2 for p, y in zip(probabilities, labels)) / len(labels)
print(round(brier, 4)) # 0.1375
The confidently wrong third example contributes 0.49, most of the total squared error. Four examples explain the calculation; they cannot estimate a deployed system's calibration.
Derive thresholds from consequences, not a default of 0.5
Suppose an unnecessary action costs 9, a missed necessary action costs 1, and correct decisions cost 0. With a reliable probability p that the action is needed, acting costs 9(1-p) in expectation, while not acting costs p. Act when the former is smaller: p > 0.9.
This is a simplified binary cost model. Add delay, human review, and alternative actions when relevant. Probabilities estimate outcomes, costs express consequences, and permissions determine allowed actions. None substitutes for another.
Report coverage alongside abstention quality
A system can defer uncertain requests to a person or retrieve more evidence. If it reports 99% accuracy on automatically accepted requests, it must also report how many requests it accepts. Covering the easiest 1% differs substantially from covering 90%.
Raise the threshold gradually and examine the relationship between coverage and error among accepted requests, broken down by language, length, and task type. Confident out-of-distribution errors can still pass a threshold; abstention does not replace input-scope checks. When deployment data shifts, collect representative labels and reevaluate the whole decision policy instead of merely increasing the threshold.