Skip to main content

Evaluating Local Models

A model name is not an experimental condition. Quantization, runtime, chat template, context, sampling, and tool parser can each change the result.

Freeze the Run Card​

Record at least:

model: organization/model
artifact: exact quantization filename and hash
runtime: name and version
template: exact chat or reasoning format
context: input limit and KV-cache settings
generation: temperature, top-p, max output, seed if supported
hardware: GPU, VRAM, RAM, offload

Without this card, two “runs of the same model” may not be comparable.

Three Evaluation Layers​

1. Capability smoke tests​

Check instruction following, structured JSON, tool calls, multilingual input, refusal behavior, and one long-context retrieval case. These catch template and parser failures before expensive testing.

2. Personal task replay​

Use 10–30 representative tasks with deterministic acceptance checks where possible. For coding, include defect repair, cross-file change, test diagnosis, and an out-of-scope action. For knowledge work, include extraction with citations, contradiction detection, and a known “insufficient evidence” case.

Run important tasks more than once. Reasoning and sampled models can have high variance, while one lucky output hides operational fragility.

3. System performance​

Measure:

  • accepted-result rate and severity-weighted failures;
  • time to first token and decode tokens/second;
  • end-to-end wall time, including tool calls and retries;
  • peak VRAM/RAM and CPU offload;
  • prompt, generated, and cached tokens;
  • human interventions and recovery effort.

Tokens per second without accepted-result rate rewards fast nonsense. Benchmark score without memory and latency may select a model that does not fit the intended loop.

Compare Quantizations Properly​

Hold prompts, runtime, context, and generation settings fixed while changing one artifact. Compare at least a high-quality reference quantization with the smallest candidate. If the smaller build loses tool-call validity or exact code edits, the saved memory may cost more loop retries than it saves.

Public suites such as lm-evaluation-harness and Aider's code benchmark provide useful external anchors, but they do not reproduce a private repository, local runtime, or harness policy. Use them to detect implausible results, not to outsource the final decision.

Decision Rule​

Choose the smallest configuration that clears the task-specific acceptance threshold with stable latency and memory headroom. Keep the run card and failing examples. Re-evaluate after changing model, quantization, runtime, prompt template, tool schema, or context policy—not merely after a new leaderboard appears.