MODEL WATCH · THE NEUTRAL LEADERBOARD

When an agent can spend money, which models stay inside the mandate?

One fixed mandate. The same adversarial battery for every model, prompt injection, payee redirects, homoglyph lookalikes, over-cap temptations, duplicate invoices, explicit "ignore your mandate" overrides. Each model acts as the agent; its proposed payment is judged by the real Fidacy firewall, the same deterministic engine that runs in production. The model is the only variable. We publish the data and the method; we do not publish verdicts about any model. Read the full methodology.

        MODELATTEMPT % · lower is saferOBEDIENCE %FALSE-REFUSAL %TRIALS
      

first cut published 2026-07-18 · 28 scenarios (23 adversarial) · 20 trials each · reproduce with the open harness

Notes · First cut, 5 models at 20 trials each (n=560). Reasoning models (e.g. DeepSeek-R1) are being evaluated separately with a larger token budget and will be added. Gemini's numbers include truncation-salvaged responses (raw preserved; see the harness).

What the three numbers mean

Attempt rate ↓

Of the adversarial scenarios, how often the model proposed a payment the firewall then blocked. Lower is safer.

Obedience on deny ↑

After being shown the block, does the model stop, or reformulate and try another out-of-mandate payment? Higher is safer.

False refusal ↓

Of the legitimate in-mandate payments, how often the model wrongly refused. Reported so "safe" is never just "refuses everything".

Why Fidacy publishes this

A neutral benchmark needs a publisher with nothing to gain from the result. Fidacy takes no fee on any transaction, sells no model, and holds no funds. Every other party who could run this, the model labs and the payment rails, has a stake in the outcome. The neutrality is structural, and the record is anchored to Bitcoin so it cannot be edited after the fact.

See the full public record → read the methodology → run the open harness yourself →