Hand “which one, what score, is it true” judgments to a cheap decision model with typed output that can be tested on its own: how to ask, how to read the probabilities, and which hosted and local options exist now.
A reproducible protocol for checking whether one exact local or API model configuration clears the threshold of your real workflow.
Measure evidence recall, ranking, faithfulness, and answer quality separately, using counterfactual experiments to locate a RAG system’s bottleneck.
Specifications, vendor claims, benchmark results, observations and recommendations prove different things; when reading someone else's conclusion, check what each one actually supports.
Choose the model tier by task, count what caching and batch jobs save, check each provider's data defaults, and keep a short list of free APIs for prototypes.
Distinguish prediction accuracy, reliable probabilities, and worthwhile actions, and use independent data to set calibration and abstention rules.
Distinguish model artifacts, evaluation conditions, and real task benefits to make a model-selection decision that can be checked.