Evidence and Bias in AI Notes

An AI note is not neutral merely because it has links and a comparison table. It should tell the reader what kind of claim is being made, what evidence supports it, and what remains a judgment.

Claim and Evidence Ladder

Claim type	Minimum evidence	What the evidence does not prove
Protocol or API fact	versioned specification or current official documentation	that implementations are complete, secure, or interoperable in practice
Model or product capability	model card/documentation plus a task-specific test	general superiority or reliability outside the tested setting
Benchmark result	result record, task version, model, harness, date, and configuration	transfer to a different repository, language, hardware, or risk profile
Personal observation	reproducible run card and retained failure examples	population-level performance
Recommendation	explicit criteria, alternatives, costs, and failure threshold	a universal best choice
Forecast	assumptions, scenarios, and uncertainty	a verified future fact

Official sources are authoritative about an interface and the vendor's stated behavior. They are interested evidence about quality. Independent benchmarks reduce vendor bias but introduce task, configuration, and publication bias.

Bias Register

Every substantial AI note should consider at least these lenses:

• selection bias: which models, tools, papers, languages, and failures were omitted;
• vendor bias: whether product positioning was repeated as an empirical conclusion;
• benchmark bias: contamination, narrow tasks, metric choice, harness/model confounding, and missing uncertainty;
• temporal bias: a snapshot presented as a durable ranking;
• hardware bias: assumptions about NVIDIA, Apple Silicon, memory, power, or network access;
• language and domain bias: English/Python performance generalized to other languages or high-risk domains;
• automation bias: fluent output or an agent's “done” message treated as evidence of correctness;
• personal-workflow bias: terminal coding preferences generalized to IDE, team, or production use.

A bias register does not make a note unbiased. It makes the remaining perspective visible and testable.

Worked Example: A Compact Claim Record

claim: gpt-oss-20b can be a practical local agent model on a 16 GB machine
kind: recommendation
as_of: 2026-08-10
facts:
  - provider says the MXFP4 model runs within 16 GB of memory
assumptions:
  - runtime supports Harmony and the exact quantization
  - context and KV cache stay within the remaining budget
missing_evidence:
  - accepted-result rate on my coding tasks
  - latency and peak memory on my hardware
alternatives:
  - smaller dense model
  - larger model on 24 GB
  - hosted model for escalation
falsifier: repeated tool-call or patch failures above the task threshold

The factual statement, deployment inference, and personal recommendation remain separate.

Review Procedure

1. Extract claims before polishing prose.
2. Label each as fact, reported result, observation, inference, preference, or forecast.
3. Match evidence strength to consequence; a purchasing or security claim needs more than a product page.
4. Search for a credible counterexample or competing framing.
5. Split durable concepts from dated product facts.
6. Preserve failed cases, not only successful demos.
7. State what would change the conclusion and set review_after from volatility.

Use this method as a reading lens for AI Systems and the Frontier Radar. It is a discipline for better disagreement, not a promise of view-from-nowhere objectivity.