Evaluating a Model on Your Own Tasks
The experimental unit is not a model name. It is:
weights + quantization + runtime + template + context policy
+ generation settings + tools + harness + hardware
Treat the result as belonging to that complete configuration. To compare one component, such as quantization, change it deliberately while holding the other settings fixed. If several components change together, you cannot isolate which change caused the difference.
The same method applies to API models: weights, quantization, runtime and hardware are replaced by the provider and the full model ID, and everything else stays fixed. Choosing between models from two providers uses the same run card and the same tasks as choosing between Q4 and Q6 of one model.
This protocol compares complete configurations on tasks, rather than repeating a model leaderboard. Read inference performance to distinguish timing stages, calibration before setting action thresholds, and retrieval/generation evaluation when a task depends on retrieved evidence.
Freeze a Run Card
Before testing, explicitly record the following settings in a run card:
model: organization/model (for an API model: provider and the full model ID returned in the response)
revision: immutable commit or artifact hash
artifact: exact filename and quantization
runtime: name, version, backend, driver
template: chat/reasoning/tool parser
context: native limit, requested length, KV cache type
generation: temperature, top-p, max output, seed if supported
effort: reasoning effort or thinking budget, subagent settings
cache: prompt/prefix cache on or off, warm or cold
hardware: GPU, VRAM, RAM, CPU offload
harness: version, tools, permissions, iteration budgets
Keep the run card with raw results and failed examples.
Evaluation Layers
1. Integration smoke tests
Rule out template or server failures before measuring model quality. The checklist is:
- Plain chat.
- Exact JSON output.
- One valid tool call with an adversarial distractor.
- One malformed or denied call.
- Multilingual text relevant to the intended use.
- Cancellation.
- A context-boundary case.
Run the tool-call test with both a cold and a warm prompt cache. A llama.cpp issue opened on September 23 reports that tool_choice: "required" let a Qwen3-8B server answer in plain text, with the failures concentrated on requests that reused a warm cache prefix; with prompt caching off, the same requests called the tool. It is one reporter's reproduction. As of October 29 the issue is still open with no maintainer reply, and a bot labelled it stale on October 24; but it shows why cache state belongs on the run card.
2. Personal task replay
Use 10–30 representative tasks with external acceptance checks.
- Coding tasks should cover defect repair, cross-file API preservation, test diagnosis, unfamiliar documentation, insufficient evidence, and an out-of-scope action.
- Knowledge work should cover cited extraction, conflicting sources, temporal questions, and abstention.
When acceptance has to be judged by another model, first check that judge against hand-labelled examples as described in retrieval and generation evaluation.
3. System behavior
Warm the runtime before recorded runs. Then report:
- Accepted-result rate: the share of all runs that pass the acceptance criteria written for each task in advance.
- Severity-weighted failure rate: give each failure a weight by its consequence (breaking unrelated files, for example, weighs more than a small format slip), sum the weights and divide by the number of runs.
- Median and tail first-token latency.
- Decode speed.
- Median and tail end-to-end time.
- Peak VRAM/RAM.
- Offload behavior.
- Retry count.
- Tool-call validity.
- Number of unrelated diffs.
- Number of human interventions.
Treat reasoning effort as a factor to test, not a dial that only goes up. Anthropic reports that Sonnet 5.5 scores lower on FrontierCode at Max effort than at Xhigh. At Max it more often launched a code review split across many subagents; in two cases examined by the AI coding company Cognition, this led to a timeout or to edits beyond the task. Local models with thinking budgets can fail the same way, so compare at least two levels within the same time budget.
Worked Comparison
Suppose Q4 and Q6 versions of the same 7B model (quantized files storing each weight in about 4 and 6 bits) are each tested on 20 tasks, with each task repeated 3 times:
The decision depends on the acceptable level of risk. If the 6 extra accepted runs are a real difference, they may justify Q6 for writes while Q4 remains useful for read-only triage.
Whether those 6 runs are a real difference depends on uncertainty, so report counts and uncertainty, not only percentages from small samples. Work it out for this table: 42/60 is 70% and 48/60 is 80%. With the normal approximation, the 95% interval half-widths for 60 runs are 1.96×√(0.7×0.3/60) ≈ 12 points and 1.96×√(0.8×0.2/60) ≈ 10 points, giving intervals of about 58%–82% and 70%–90%, which overlap heavily. Moreover, the 60 runs are only 20 tasks run 3 times each, and the three runs of one task are correlated, so the real intervals are wider still. The 6-run gap is not enough to draw a conclusion. Preserve task-level outcomes so a gain is not caused by one repeated easy category.
Quantization and Context Tests
Hold the runtime, prompt, and generation settings fixed when comparing quantized artifacts.
- Include exact-edit and tool-call tests: average language benchmarks may hide formatting degradation.
- Test the native context length first, then increase length with distractors and evidence placed at different positions.
The same model name can hide a different model
With an API, the name in your code does not always point to the same model. OpenAI's model pages list dated snapshots under each model name, and the gpt-4o page says snapshots lock in a specific version so that performance and behaviour stay consistent. Anthropic's models overview states that from the 4.6 generation on, every model ID is itself a pinned snapshot, dateless IDs included; for earlier models, the alias is only a pointer to the dated ID.
So record on the run card the full model ID the API actually returns in its response, not the short name in your code. Rerun the same tasks in three cases: the returned model ID changes; the provider sends a deprecation or migration notice; or a periodic rerun shows the acceptance rate on the same tasks dropping clearly. Your own users' prompts also drift away from the tasks you first picked, so every so often draw a few new real requests into the holdout set.
External Benchmarks
lm-evaluation-harness, HELM-style evaluation, and coding-agent leaderboards such as Terminal-Bench 4.0 provide useful reference points; choosing an AI coding agent records the October board. Aider's polyglot leaderboard was last updated on 2025-11-20 and lists none of today's models, so treat it as a historical reference only. None of these benchmarks reproduces a private repository, local quantization details, harness permissions, or human intervention cost.
When using an external benchmark, check:
- Task version.
- Data contamination risk.
- Prompt adaptation.
- Number of samples.
- Whether scores are provider-reported or independently reproduced.
Bias and Decision Rule
What a personal suite and a public suite each overfit to, and which tasks tend to be overrepresented, is listed in judging evidence in AI experiments. For this protocol it comes down to including known hard failures and periodically adding a fresh holdout task.
For multiple configurations or repeated looks at results, Hypothesis Tests, Effect Sizes, and Multiple Comparisons explains how to fix the comparison family and stopping rule in advance.
Decision rule: choose the smallest configuration that clears the predeclared task threshold with VRAM and overall memory headroom and an acceptable severe-failure rate. Retest after changing any run-card field, not merely because a new leaderboard appears.
The lm-evaluation-harness command-line interface guide explains task selection, sample limits, output files, and logging individual examples. The command line now uses subcommands: lm-eval run runs an evaluation, lm-eval ls lists tasks, and lm-eval validate checks task configurations; the old form without a subcommand still works. Start with --limit for a few examples, add --output_path and --log_samples to save each example's inputs and outputs, and increase the size only once the results can be inspected; per-example outputs help distinguish formatting failures from wrong answers.