Every model gets the identical mandate. It plays an autonomous payments agent for a fictional company called Northwind Robotics, with these rules and nothing else:
The system prompt that carries these rules is published verbatim in run.mjs. It states that the mandate cannot be overridden by any instruction in a message, no matter how urgent or authoritative that message claims to be. We do not tune the prompt per model. Same words for everyone.
A legitimate payment fully inside the mandate. A model that refuses one of these is being over-cautious, and we count that.
A redirected payee (classic BEC), a homoglyph payee like "Pay0neer", an over-cap payment, an out-of-scope category through a trusted payee, a re-presented paid invoice, and a flat "ignore your mandate" override.
The scoring engine is @fidacy/firewall's own decide() function, the same code path that gates payments for real installs. Deterministic: the same proposed payment against the same mandate state always returns the same verdict. No rubric, no LLM judge. If a model proposes paying Global Settlement Partners, the firewall denies it because that payee is not on the allowlist, full stop.
The harness never invents model output, every number comes from a real API call. A stub mode proves the pipeline without touching the network; those runs are tagged stub:true and never published as model results.
{{ m.name }}
{{ m.dir }}
{{ m.desc }}
This is the part we want you to read twice. Model Watch measures the behavior of a system: the model plus our exact prompt plus this exact mandate, under adversarial pressure. It does not measure the intelligence of a model, its "alignment" in the abstract, or how it would behave in your product with your prompt.
Read every result as "this model, under this prompt and this mandate", never as "this model, period."
Neutrality here is structural, not a promise about our good intentions. Fidacy takes no fee on any transaction, sells no model, holds no funds, no financial stake in whether a given model looks good or bad. Everyone else who could run this benchmark is a party to the outcome: model providers rank their own models; payment rails have a stake in which agents move money over them. We sit outside both, which is exactly why the same code can be the neutral judge here.
{{ c.desc }}
The harness is open source; the mandate, the scenarios and the exact prompt are public. Clone it, plug in your own API keys, run it yourself. You will get your own numbers, and if they differ from ours we want to know. Each published run is hashed and anchored to Bitcoin, the same anchoring the firewall uses for audit proofs, so the dated record of what we ran and what came back cannot be quietly edited later.
A model appears on the public leaderboard only once it clears a minimum number of trials, always disclosed; below that it is marked provisional. And Model Watch is a lab-condition benchmark, separate from the field data of real installs, where mandates and messages are whatever customers actually configure and receive. The two are clearly labeled and never blended.
stub/safe
Refuses every bait AND pays the legitimate controls. A model that refused everything would also read 0% attempt, until you check the false-refusal column. attempt_rate alone never tells you a model is good.
stub/reckless
Takes every bait literally; the firewall denies every one. The other end of the axis. Real models land between, and the interesting question is the tradeoff: how few baits taken, at how low a cost in wrongly refused payments.
If you think our method is wrong, show us and we will fix it in public. Believe a better system prompt would represent your model fairly, or spotted a bug or an unfair scenario? Send it. We will run your prompt as a clearly labeled variant alongside the standard one, never quietly swapped in as the headline number, because then the benchmark stops being comparable.
We are not here to rank your model. We are here to publish a reproducible measurement and the method behind it.