Judging Evidence in AI Experiments

A note can contain ten links and still be mostly opinion. To judge it, check what each source actually supports.

A specification can establish how an API is defined. It cannot establish that every implementation works. A benchmark can report performance in one setup. It cannot choose the best tool for every repository. A personal test can guide my own workflow, but one run does not become a general result.

Match the evidence to the claim

If the note claims...	It should usually have...	That still does not prove...
A protocol or API behaves a certain way	A versioned specification or current official documentation	That implementations are complete, secure, or interoperable
A model or product can do a task	A model card or documentation plus a task-specific test	General superiority outside the tested setting
A benchmark result	The task and harness versions, model, date, configuration, and result record	Transfer to another repository, language, or machine
A personal observation	A reproducible run record and retained failures	Population-level performance
A recommendation	Criteria, alternatives, costs, and a failure threshold	One best choice for everyone
A forecast	Assumptions, scenarios, and uncertainty	A verified future fact

Official documentation is the right source for an interface and for what a vendor says its product does. It is interested evidence about quality. Independent benchmarks reduce that bias, but introduce their own choices of tasks, configurations, and publication thresholds. How to check task versions, configuration and per-item differences when reading a benchmark result is covered in model cards and benchmarks.

Bias is easier to manage when it has a name

Before trusting a comparison, I check what it leaves out. The omissions are often more important than the last decimal place.

Temporal bias is the easiest to show: it turns a dated snapshot into a durable ranking. OpenRouter's own June comparison of free APIs was out of date on two entries by August (the model and API radar has the current state): Cerebras's free tier had become a 30-day trial with $5 of credit that requires a verified payment method, and GitHub Models had been fully retired on July 30.

The other biases have recognizable shapes too. Selection bias appears when the test excludes inconvenient models, languages, tools, or failed runs. Vendor bias appears when product positioning is repeated as a measured result. Benchmark bias includes contamination, narrow tasks, metric choice, and confusion between model and harness quality.

Hardware bias hides assumptions about memory, power, accelerators, or network access. Language and domain bias generalizes English or Python results to settings that were never tested. Automation bias treats fluent output or an agent's completion message as proof that the work is correct. Workflow bias turns one person's preference for terminals, IDEs, or automation into advice for everyone.

This list does not make a note neutral. It makes the remaining perspective visible enough to challenge.

Write recommendations as decisions, not slogans

A compact claim record makes the missing pieces hard to hide:

claim: MiniCPM5-2B is a usable local model for triage and simple tool steps
kind: recommendation
as_of: 2026-10-29
facts:
  - model card lists 2.5B parameters, a 131,072-token context and Apache-2.0
  - official GGUF, MLX and GPTQ builds exist
  - model card recommends SGLang with the minicpm5 parser for tool calls
vendor_claims:
  - LiveCodeBench v6 score of 69.1
assumptions:
  - my runtime parses its tool-call template correctly
  - the 4-bit build keeps enough triage quality
missing_evidence:
  - accepted-result rate on my triage tasks
  - tool-call validity outside SGLang
alternatives:
  - Qwen3-4B
  - hosted small model for escalation
falsifier: label or tool-call failures above the task threshold

The facts in this record come from the MiniCPM5-2B model card. The vendor's score sits on its own line, the local assumptions on another, and the last line names the result that would overturn the recommendation, which also sets the next review date. It is more useful than writing that a model is "powerful" or "promising."

My reading