Test Strategy and System Confidence
Different tests answer different questions. Classify them by the boundary they exercise and the evidence they provide, not only by their filename.
| Layer | Primary question | Typical trade-off |
|---|---|---|
| Unit | Does focused domain behavior hold? | fast, limited realism |
| Integration | Do real components agree on a contract? | more setup and failure modes |
| System/end-to-end | Does a representative user flow work? | slow, broad, harder diagnosis |
| Contract | Can independent producer/consumer versions interoperate? | requires shared version policy |
| Performance/load | Does behavior hold under a defined workload? | environment-sensitive |
| Security | Can known abuse cases cross a boundary? | needs evolving threat models |
Black-box tests assert behavior through public inputs and outputs. White-box tests use implementation knowledge to target branches or states. A healthy suite uses both without making every refactor rewrite the tests.
Select cases from risk
Test ordinary behavior, boundary values, invalid inputs, partial failures, retries, concurrency, and recovery where the system actually faces those risks. A regression test preserves a previously missed behavior. A smoke test checks a small critical path after build or deployment; it is not a substitute for the full suite.
Property-based and fuzz testing explore broad input spaces when invariants can be stated more clearly than handpicked examples. They complement, rather than replace, readable examples for important domain scenarios.
CI as a reproducibility check
Continuous integration should start from a declared environment, install from locked or constrained dependencies, and run deterministic required checks. Keep the fast signal early and isolate expensive or environment-specific suites with clear ownership.
A failing required test must block the change or be handled through an explicit exception process. Do not normalize “rerun until green.” A flaky test indicates nondeterminism in the test, product, or environment; record it, assign ownership, and fix or quarantine it with a deadline and preserved visibility.
Evidence, not a score
Line and branch coverage reveal unexecuted code but do not measure assertion quality, realistic data, or missing requirements. Mutation testing can reveal checks that execute code without detecting changes, at additional computational cost.
Production observability, staged rollout, rollback capability, and incident review cover conditions a test environment cannot fully reproduce. Reliability comes from the whole feedback system, not a single test pyramid or coverage percentage.