Beyond Green

Your tests passed.
We check what they missed.

A self-correcting loop that investigates the evidence a passing end-to-end test leaves behind, and decides whether the system actually behaved correctly.

6 sper investigation

$0.00054per investigation

0false positives in 24 runs

1real shipped bug reproduced

The problem

A green test hiding a destroyed row

One line changes in Linkding: archiving a bookmark deletes it instead. The project's own browser test still passes, the request returns 200, and the final screenshots are identical down to the byte.

test: test_active_bookmarks_partial_update_on_archive (unmodified)
upstream test: PASSED
archive request: POST /bookmarks/action → 200
screenshots: byte-identical, sha256 3ed76c3e…
clean bookmarks: 9 → 9
defective bookmarks: 9 → 8 ← one row destroyed
tag links removed: 1

Nothing a person would look at has changed.

The loop

Four roles, and a gate before any verdict

Nobody writes prose. Every answer is a choice from a list with a probability attached, so every branch is ordinary code reading an enumerated value.

The gate

Watch the loop refuse itself

assessment.choice === "supported"
  && assessment.confidence >= minConfidence
  && complete
  && hypothesis !== "unknown"

1Scouthypothesis = unknown · 0.57
2Criticassessment = insufficient · 0.78, next = database_state
3Evidenceretrieved database_state
4Investigatorhypothesis = suspected_violation · 0.62
5Criticassessment = supported · 0.31, next = operation_contract The Critic agreed, and the loop kept going anyway, because 0.31 does not clear the gate. A loop that read the word and ignored the number would have stopped one source early and been right by luck.
6Evidenceretrieved operation_contract
7Investigatorhypothesis = suspected_violation · 0.98
8Criticassessment = supported · 0.89, next = none
9Verifierverdict = regression · 0.98, grounding = sufficient · 0.99

Credibility

What the model is never allowed to see

Labels reach Weave only after an investigation returns, and only for scoring. Artifacts are hashed before and after every run, so both providers demonstrably read the same bytes.

Measured

Two models, one engine, identical evidence

Two runs each	TypeSafe jev-1.13.0	DeepSeek-V4-Pro
Correct verdicts	4/6 · 4/6	6/6 · 6/6
Abstentions	2 · 2	0 · 0
Wrong verdicts	0 · 0	0 · 0
False positives	0 · 0	0 · 0
Latency	5.3 – 6.1 s	78.8 – 92.5 s
Output tokens	382	9,900 – 11,200

No incorrect verdict in 24 investigations. Both runs abstained on exactly the same two defect shapes, for the same recorded reason, with every required source already in hand. That is a property of those defects, not noise.

The cascade

One engine. The judge is what changes.

Escalation re-runs the whole investigation with a different judge, from a fresh state and fresh retrievals. Same engine, same evidence catalog, same typed questions, same gate. Only the model answering changes, which is why any number of tiers composes and why comparing two providers is fair.

The cascade

The expensive model's accuracy, at a third of the price

Replayed across two recorded studies, twelve investigations per provider. Running it live confirmed the replay exactly: 6 of 6, escalating 2, flipping both.

Policy	Correct	Cost per 250 tests
TypeSafe alone	8 / 12	$0.14
Cascade	12 / 12	$5.36
DeepSeek alone	12 / 12	$15.67

The obvious design escalates a confident finding for a second opinion. Our measurements say the opposite. TypeSafe issued no wrong verdict and no false positive in 24 investigations, so every miss is an abstention. Escalate what the loop gives up on, not what it concludes.

Policy

Two more levers, measured not asserted

The threshold was conservative

Re-running the suite at seven thresholds costs two cents. Dropping 0.80 to 0.50 recovers a case with no wrong verdict at any value.

The traces label themselves

Across four studies the providers never contradicted each other. All six disagreements were one giving up, spotted without any ground truth.

Neither touches the operation contract, which is the part that needs a human. Tuning a threshold on the same six cases you measure is overfitting, so the sweep is a lead rather than a setting to ship.

Economics

Cheap enough to run on every green test

measured end to end projected range, never run

Investigating 250 passing tests once each, logarithmic because the range spans four orders of magnitude. Both measured providers read a similar volume of input, so the gap is the rate, not efficiency.

Held out

A real bug nobody here designed

Documenso #2485 shipped in 2.6.0. With next-signer dictation, a recipient was marked sent, with a timestamp, while no signing request was ever queued. Its own Playwright test loads that exact row and asserts name and email.

	Defective build	Fixed build
Upstream tests	2 passed	2 passed
Rows claiming a dispatch	2	0
Signing-request jobs	0	0
Emails delivered	0	0

And the investigator does not yet find it. TypeSafe abstains on both builds, DeepSeek reports a regression on both, which makes its finding a coincidence rather than a detection. Only the clean control caught that. We did not rewrite the contract while watching the verdicts, because that manufactures the result the case exists to test.

Honest limits

What this does not establish

Four mutations of one workflow are not four regression classes, and those cases were designed alongside the investigator. Two clean controls cannot characterise a false-positive rate.
Confidence measures how concentrated a distribution is, not how often the answer is right. Tuning the threshold on the same six cases we measure is overfitting.
The loop investigates one recorded operation per run, and it is not yet a CI gate.
Its quality rests on the contract it is given. A vague contract buys silence rather than a wrong answer, which is the safe failure but still a failure.
No invoice reaches this harness. Every dollar figure is an estimate carrying its source.

Try it

Read the investigations yourself

Twelve recorded investigations, both providers over all six captures, including the two abstentions. Inspect before and after state, the operation contract, and every recorded agent choice. No keys, no uploads, no model calls.

Open the viewer Source on GitHub Integration notes

Linkding and Documenso are unmodified dependencies, pinned at eb98e67d and 3cef238f. The engine, capture harness, evidence layer, evaluation and viewer were built for the hackathon, and every role, retrieval and verdict is traced in W&B Weave.

Your tests passed. We check what they missed.

Your tests passed.We check what they missed.