Reading Model Cards, Licenses and Benchmarks
One model family often ships a base version, an instruction-tuned version and 4-bit quantized files at the same time, and these are three different things. With the wrong chat template, or an input silently truncated, a good model answers badly and looks incapable. Model selection therefore asks whether a specific version can perform your task under your constraints. Names, parameter counts and aggregate rankings are indexes; the useful evidence concerns inputs and outputs, training and evaluation conditions, and reproducible artifacts.
Model Cards proposed documenting intended uses and performance under different conditions alongside models. A card is a starting point for investigation, not a quality certificate.
Identify the Artifact
A family may publish base weights, instruction variants, distilled variants and several quantizations. Similar names do not establish identical templates, behavior or hardware needs. The Hugging Face model-card documentation explains how descriptions and structured metadata coexist. Resolve these questions for an exact version:
Separate total from active parameters, particularly for MoE. Read maximum context alongside the actual configuration and cache cost.
Read Software and Weight Licenses Separately
A repository can contain software, weights and data references. A displayed license field helps locate the terms, but open the license and release documentation to identify the artifact covered. The repository-license documentation describes this metadata. A wrapper's license does not establish identical terms for weights, upstream data or a hosted service.
Record a version or revision with the files used. If a model changes under the same name, a download link alone cannot reproduce the original comparison. For hosted APIs, retain the date and whatever model-version information is available.
One Ranking Does Not Answer Every Question
HELM evaluates language models across scenarios and metrics. Read the evaluation target before looking for a winner. Accuracy, calibration, latency, throughput and cost answer different questions; combine them only after defining the application objective.
Suppose two candidates face the same 100 examples:
A scores 90%, B 92%. The whole two-point gap comes from the 10 examples where they disagree: A wins 4 and B wins 6. On the other 90 they are both right or both wrong. If the two models were in fact equally good, those 10 examples would behave like 10 coin tosses. An exact 5–5 split has probability 252/1024, about one in four; in the remaining three quarters of cases the split is 4–6 or more lopsided (sign test p≈0.75). The two-point gap on its own therefore shows nothing.
The cost of errors can also change the conclusion. If A's six misses are minor formatting slips while B's four misses lose essential facts, the two-point gap is no reason to choose B. If error costs are comparable, the difference still needs replication on new examples.
Distinguish one model answer from a system's final result. Was retrieval used? Tools, multiple candidates, retries or additional reasoning budget? More computation can change both quality and latency; differences between systems cannot all be attributed to weights.
Read benchmark results that others publish the same way. First ask who ran them: the vendor, or a third party reproducing them. Then ask which system was measured: are the harness, the number of retries and the reasoning budget stated? Next check whether the competitors ran on the same items under the same configuration, and whether the gap is larger than the same-items coin-toss noise shown above. Finally, consider whether the items may already have been in the training data. If any of these questions has no answer, the score is a lead, not yet a basis for a decision. The full list of common biases, such as contamination, selection bias and vendor bias, and a way to write a claim as a checkable record, are in Judging Evidence in AI Experiments.
Narrow Candidates with Real Tasks
Select tasks spanning difficulty and error cost, excluding sensitive material or obtaining appropriate authorization. Fix the input and success conditions and retain an inexpensive baseline. Classification may use rules or a small classifier; retrieval can start with keyword search; code needs tests and human inspection.
Examples used to select thresholds, modify prompts or tune retrieval belong to development, not independent validation. Reserve new test examples until settings are chosen; see data splits and leakage.
Keep an explainable decision: why this version was selected, which errors remain, which inputs lie outside tested conditions, and what change requires another evaluation. Current candidates belong in model and API selection; use local model evaluation for experiments on your hardware.