Workflow-based resolution beats AI quorums.
- More reliable
- Harder to manipulate
- Reproducible
- Faster
“Will US CPI be at least 3.0% in the October report?”
AI-judge quorum
3 models vote- 01
Search the open web
Headline: “Inflation hits 3%”
- 02
Every judge reads the same page
- A · YES
- B · YES
- C · YES
- 03
YES
Unanimous. And wrong: the headline was rounded.
Fermah Kernel
rule fixed at mint- 01
Named source, locked in
BLS · CPI-U · all items · not seasonally adjusted
- 02
Read the number
12-month change = 2.96%
- 03
NO
2.96 ≥ 3.00 is false. Same answer on every replay.
More judges ≠ more truth
Models make the same mistakes.
- Same training data, same search results
- So their errors line up instead of cancelling
- Past three judges, accuracy barely moves
Model: each judge wrong 5% of the time · illustrative
| AI-judge quorum | Fermah Kernel | |
|---|---|---|
| Clarity | Decided at the deadline | Fixed before the first trade |
| Reliability | Whatever search finds | Named sources, frozen at mint |
| Reproducibility | Can flip | Same answer, inputs stored |
| Handling uncertainty | Picks an answer anyway | Pauses, extends or voids |
| Manipulation resistance | Plant pages it might read | Must compromise the named source |
| Speed | Minutes | Seconds |
Where judgment is needed, AI works inside the rule.
kernel workflow · fixed at mint
ai.judge- Cited evidence only
- 2+ model families
- 2/3 to settle
- Unsure → review, never a guess
Inside a Kernel workflow, the AI judge is one step, wrapped by four guards.
What the research says about AI judges
- 60%
When two AIs get it wrong, they usually make the same mistake. Asking more AIs doesn't catch it.
Kim et al. · ICML 2025 - +1 pt
A team of AIs did barely better than one AI alone.
Kota · 2026 · Kalshi questions - 80
Ask one AI the same question 1,000 times, get 80 different answers.
Thinking Machines · 2025 - 5 pages
Plant 5 fake articles and the AI repeats the lie 90% of the time.
PoisonedRAG · USENIX Security 2025