debugging-with-evidence

# Debugging with evidence

An observed failure establishes that something happened under particular
conditions. Find the mechanism that explains it before changing another thing.
Keep the investigation tied to the user's outcome and authorized file scope.

## Capture the failure

1. Record expected and observed behavior, exact command or user action,
   revision, dirty inputs, environment, relevant order or seed and evidence path.
2. Read the complete error and inspect the affected artifact and relevant diff.
   Missing paths, absent tools, empty test collection and permission errors
   may explain an unrunnable check; they do not establish successful behavior.
3. Reproduce with the smallest scope that still preserves the failure. Retain
   the original run: narrowing can remove order, shared state or boundary effects.
4. Register cleanup before an experiment that creates temporary resources.
   Use read-only inspection to answer state questions; do not reset or destroy
   the working tree to learn what was changed.

## Test explanations

Write a small set of plausible hypotheses based on the observation. For each,
name the predicted result and the observation that would contradict it. Choose
an experiment whose outcomes separate at least two explanations. Repeating the
same failed run without changing a discriminating condition adds little evidence.

Change one relevant condition where feasible and capture the actual outcome.
For an intermittent failure, retain repetitions and their conditions; one passing
retry is not proof of a fix. Instrument only what is needed and remove temporary
instrumentation when it no longer contributes to the delivered behavior.

When evaluating a detector or classifier, establish that a known-good case can
produce a negative result. A check that flags everything may be measuring itself.
Inspect differences in inputs, configuration and failure signatures before
assigning cause from test names, counts, labels or timing alone.

If branch versus baseline attribution matters, use `whose-red` at the location
the host actually exposes. If unavailable, compare identified trees with matched
commands, environment and failure signatures, using a clean control when feasible.
Report extra branch failures and unmatched conditions separately. A shared failure
on one baseline supports a limited historical finding, not innocence of the change.

## Fix the supported mechanism

When evidence supports a cause, make the smallest change that addresses its
class within the authorized scope. Inspect related call sites and affected inputs;
one repaired example may leave the same defect elsewhere. Where warranted, add
a regression check that would reject the old behavior for the observed reason.
Avoid tests that only repeat implementation values without an independent basis.

If a planned fix fails, compare its prediction with the new observation and
revise the hypothesis. Do not stack unrelated guesses. If experiments stop
distinguishing explanations, summarize what is known, seek a better observation
or name the concrete unavailable prerequisite. Continue independent work if useful.

## Verify the explanation and result

Rerun the reproduction and relevant regression scope, then the agreed acceptance
and broader checks on the candidate. Inspect the actual output and capture each
command's own exit code. Review the diff for scope and temporary artifacts.
Use discovered `reviewing-changes` when useful; self-review remains explicit.
After review fixes, run affected checks and review changed or unresolved findings.

Before reporting success, confirm evidence describes the final inputs and
rerun final acceptance yourself. State the supported cause, rejected alternatives,
changed files, observed checks and evidence paths. Keep unavailable comparisons,
remaining intermittent behavior and untested boundaries explicit.

Shown under the source repo’s Apache-2.0 license. Read the full folder on GitHub.