Beyond Green · integration notes

Two notes on `jev-1.13.0`

Both turned up while wiring the typed-choice API into a post-test investigation loop. Both are measured from recorded responses, not inferred.

—

We run a five-role investigation loop over CI results: a Scout proposes, a Critic attacks and picks what evidence to read, an Investigator revises, and a Verifier rules. Each role is one POST /v1/systemone call with a JSON state and a set of closed questions. The loop only concludes when a Critic marks the case supported at or above a confidence threshold, with every required source retrieved.

Across the runs behind this note, — answers came back from jev-1.13.0. Two things in them are worth passing on. The first is a small interoperability trap. The second changed how we read our own benchmark, and it is the reason we are writing.

Note oneTwo-decimal distributions do not always sum to 1.00

Probabilities come back rounded to two decimals, so a well-formed distribution can miss 1.00 by a rounding unit. In our sample that is —:

Model	Answers	Sum 1.00	Sum 0.99

Every deviation we recorded is downward. We have not seen a sum above 1.00:

—

This matters only if the client validates strictly. Ours did, with a 0.001 tolerance, and rejected those responses as malformed. The symptom was an intermittent “invalid response” on roughly one call in a hundred, which is an unpleasant thing to chase because retrying usually works. The fix on our side was to size the tolerance to the rounding rather than to zero — with n values each carrying at most half a unit of error, n × 0.005:

const roundingUnit = 0.01;
const sumTolerance = (options) => options * (roundingUnit / 2) + Number.EPSILON;

Nothing here needs fixing at your end if the two-decimal rounding is deliberate. It would help to state the convention in the docs, since the natural client-side check is equality to 1.

Note two`confidence` is a margin, not a probability

The confidence field is not the probability of the chosen option. Against the recorded answers it matches the chosen option’s distance above chance, normalised to 0..1:

confidence = (p_max − 1/n) / (1 − 1/n)

On a real Scout answer — {consistent: 0.17, suspected_violation: 0.25, unknown: 0.58} — that gives (0.58 − ⅓) / (⅔) = 0.37, which is what came back. Classifying every answer by which formula it matches:

Model	Answers	Margin only	Top probability only	Both	Neither

“Both” is where the two formulas nearly coincide, which happens as the top probability approaches 1. “Neither” is mostly two-decimal rounding at the margins. The split is unambiguous in the columns that separate them: for jev-1.13.0 the top-probability-only count is —.

This is a reasonable definition — arguably a better one, since it says how much better than guessing the answer is, which is what a gate usually wants to know. The problem is only that it differs from the other provider we run, which reports the top probability directly. Our threshold was a single number applied to both.

What that cost us

A threshold of 0.80 is a different bar on each scale. To clear it:

Options	Needs p_max (margin scale)	Needs p_max (top-probability scale)
2	0.900	0.800
3	0.867	0.800
4	0.850	0.800

So jev-1.13.0 was held to a materially stricter standard than the model we were comparing it against, by a threshold we thought was neutral. Re-scoring every gate decision in the recorded runs on a common scale, — differ — and all of them are the same model on the same case, each one an assessment our gate rejected that a top-probability gate would have accepted:

Case	Role	Choice	Reported	p_max

Looking at every run that gave up because the Critic was never convinced, and the last supported assessment it made before giving up:

Case	Reported	p_max	Our gate	Top-probability gate

—

What this does not show

That those runs would have reached a correct verdict. Clearing the Critic’s gate only lets the loop continue to the Verifier, which holds an independent veto and was never reached in any of them. We have not re-run the suite on a common scale, so the outcome is unknown rather than favourable. The honest claim is narrower: on this case the abstentions sit inside the gap between the two conventions, so they are partly an artefact of our threshold rather than a limit of the model.

For contextThe record these notes come from

Six seeded cases on an open-source bookmark manager, each a real state mutation behind a passing UI test. Every policy was repeated; the ranges below are across repeats.

Policy	Runs	Correct	Wrong	Abstained	Mean latency

—

The column worth dwelling on is Wrong. Across every run of every policy the loop never produced an incorrect verdict — no false positives and no false negatives. Where jev-1.13.0 could not establish a case it declined to answer, which is the behaviour we wanted and the reason it is the first tier in our cascade. Note two is about the threshold we set, not about that behaviour.

ReproducingEverything above from the receipts

The public repository includes sanitized inputs from the ten recorded runs. From a fresh checkout with Node.js 24 or newer, regenerate this page’s data without API keys or model calls:

npm ci
npm run report
git diff --exit-code -- site/findings-data.js

The final command checks an exact match against the published data. The input receipts and provenance retain the recorded choices, confidence values, distributions, outcomes, and timings. Answer statistics cover each run’s final tier; the original cascade receipts omit earlier-tier answers. Running a new model investigation is a separate, billable workflow described in the repository.

The scale check is scripts/provider-report.ts; the tolerance change is src/investigation/choice-response.ts. Full source and the step-by-step traces are at github.com/SashaSkind/safeci, and the loop is walkable step by step at the trace viewer.