Confidence & Calibration: When to Trust the Number

Jev doesn't just return a verdict — it returns how sure it is. Reading that signal correctly is the difference between an automation you can trust and one that silently misroutes 15% of your traffic.

#probability vs confidence

The two fields are easy to confuse and they are not the same thing:

For Noul, the probability is the whole signal: 0.97 is a confident yes, 0.52 is a coin flip. If you need a single decisiveness number from a Noul, the community convention is abs(p - 0.5) * 2 — 0 at total uncertainty, 1 at certainty. For Choice and Score, read confidence first and use the distribution shape as backup: a Choice at [0.34, 0.33, 0.33] is undecided whatever the top label says.

#What "calibrated" means — and why RLCD exists

A model is calibrated when its stated probabilities match reality: among all answers it gives at 0.8, about 80% are right. Calibration is what makes a threshold meaningful — "auto-approve above 0.95" only works if 0.95 actually means ~95% correct, not "a highish number the model likes."

Jev is trained with RLCD — Reinforcement Learning for Calibrated Decisions — specifically targeting this property, rather than RLHF-style "produce text people prefer." That's the theory, and it's a genuinely better objective for decision models.

Now the honesty clause. Calibration claims are launch-material claims, and independent testers have reported accuracy below official numbers, varying significantly by task — one early third-party benchmark was even withdrawn. Treat any official accuracy figure as a hypothesis, and verify on your own data before you automate against a threshold. How:

1. Collect 200+ real items where you know the correct answer.
2. Run your exact production question over them; log every probability.
3. Bucket the results (0.5–0.6, 0.6–0.7, …) and compare predicted vs actual hit rates.
4. If 0.8-bucket answers are only right 65% of the time on your task, your thresholds come from your curve — not from anyone's blog post, including this one.

#Designing a confidence gate

The production pattern is three zones, not two:

p ≥ T_high          → act automatically (approve, route, send)
T_low ≤ p < T_high  → fall back: human review queue, or escalate to an LLM
p < T_low           → act on the negative automatically (discard, archive)

• Set T_high from your measured curve at the precision you can live with — not from vibes. For irreversible actions (deleting, publishing, paying), demand a bucket that measured ≥99%.
• The middle zone is not a failure state; it's the design. It's where Cascade Routing hands off to an LLM, or where a human spends the 30 seconds the model just saved everyone else.
• Re-measure on a schedule. Input distributions drift; a gate tuned in September can be miscalibrated by December.

A worked, code-level version of this pattern lives in Confidence Gating.

#Where to go next

• Confidence Gating recipe — the three-zone gate in code
• The Three Primitives — which fields each primitive returns
• Models & Pricing — the honest-performance caveats in full

#Sources

• learnjev.com — Cost & benchmarks (community documentation; independent accuracy re-tests and the withdrawn benchmark).
• jevai.wiki — API reference (community documentation; response fields per primitive).
• jev101.com — 什么是 Jev（中文） (community documentation, Chinese; RLCD).

Unofficial fan-made handbook. Not affiliated with TypeSafe AI or jev.com.