L2 · L3 · Agent Ops
Evaluation contracts and what they have returned
All twenty-six task agents carry a written evaluation contract. This page counts how many state a number the agent can fail, how many have failed it, and what the drift instruments read this week.
CONTRACTS THAT CAN FAIL
9 / 26
NEVER RETURNED A FAILURE
13
UNCLASSIFIED RATE
12.8%
CONTROL WATCH
3 of 4 tripped
The requirement this page tests
Every agent ships with a named evaluation set and a cost budget.
The number the agent must hit is agreed before the build. A contract can still be written so that no observation would contradict it.
“Materiality calibration against a 200-story human-labelled set; target at or above 0.80 rank correlation with the CCO’s own ordering. Re-run quarterly.”
Supports · This is what a falsifiable contract reads like: a named set, a stated target, a cadence, and a result that can come back short. It currently does — 0.78 against 0.80, and short of it in every quarter on record.
Three kinds of contract in the register
Threshold · can fail
9A stated number and a direction. The reading either clears it or it does not.
Measured · no pass mark
14Something real is counted, and reported, and nothing about the reading constitutes a failure. These are dashboards wearing a contract’s clothes.
Judgment · no number
3Assessed by a person on a cadence. Legitimate for work that resists measurement — and the category to audit first, because it is also the easiest place to hide.
34.6% of the register is falsifiable. Counted from the 26 contracts, not asserted.
The assurance layer
Four instruments, three of them currently reading badly
The harness, the quarterly calibration loop, the drift alarms and the lifecycle gate, each with its current reading.
Contracts that can fail
0.0%
9 of 26 state a number and a direction
Never returned a failure
0.0%
13 sets have never once come back short
Predictions still open
0.0%
20,071 logged · 2,468 scored
Unclassified rate
0.0%
3 consecutive months above the 12% alarm
What these four numbers are admitting
Every task agent has a written evaluation contract, so the requirement is satisfied on paper. Read the contracts and 17 of 26 turn out to describe a measurement rather than set a bar, which is why 13 of them have never returned a failing result — not because the agents are good, but because there is no reading that would count as bad. The Intelligence cluster is the clearest case: 5 agents, 0 falsifiable contracts between them, and they are the agents whose output shapes what executives believe. The prediction loop tells the same story from the other end — 20,071 predictions logged and 12.3% written back against an outcome, so the learning loop is real but running on an eighth of the evidence it has collected. The instrument that is unambiguously working is the drift alarm, and what it is reporting is bad news: 3 months above the line with the confidence distribution shifting left underneath it.
13 of 26
A set that cannot fail is a description, not a test. These are re-specified first.
Threshold · Every set should have returned a failing result at least once
12.8%
The taxonomy is reviewed. Unclassified items route to a human in the meantime.
Threshold · 12% in any rolling month
2 of 8 personas
The panel is modeling the company back to itself. Results can be ruled inadmissible.
Threshold · 85% agreement with the recommended position
0.2235 vs 0.250
Clear at 4 consecutive quarters. The phase gate is met.
Threshold · Beat the baseline for 2 consecutive quarters
Agents with a named evaluation set
26 of 26
The specification's promise, kept. Counted from the register, not asserted.
Sets that state a number the agent can fail
9 of 26
34.6% of the roster. The rest measure something without declaring a pass mark.
Sets that have never returned a failure
13 of 26
50.0%. A test that has never failed is a description of intent.
Sets with no reading at all
2
The measurement exists on paper and produces nothing.
Read this column first
Every agent has an evaluation set. That is not the same as every agent being testable. 9 of the 26 contracts name a threshold the agent can miss. 14 track a number but never declared a pass mark, so the number can move in any direction without anything happening. 3 contain no number at all — they can only disappoint, never fail. The harness is real. Its teeth are unevenly distributed, and the distribution is the finding.
9
A declared number, a declared direction, a declared consequence.
14
Tracked. Never scored against anything.
3
A qualitative contract. It cannot return a failing result.
Teeth by cluster
6 clusters · 26 agentsThe Intelligence cluster is the one to look at: five agents, and not one of them has an evaluation set that states a number it can miss. These are the agents that produce positions, simulations and briefings — the outputs most likely to be believed and least likely to be checked.
Below target right now
Rank correlation with the CCO’s ordering — 0.78 against a target of 0.8 · short by 0.02
Short of the mark in every quarter on record, closing on it each time. Adjustment proposed, not adopted.
Median lead time versus public emergence — 5.2 days against a target of 7 days · short by 1.80
Precision is at 74% and clears its half of the contract. Lead time does not, and lead time is the half that matters.
Escalation recall — 99.7% against a target of 100% · short by 0.30
The only target on the platform where anything short of perfect is a miss. Four items in 1,482 were not escalated; three were recovered inside the deadline and one was not.
Every contract, as written
Verbatim from the agent register| Agent | Evaluation contract | Kind | Reading | Status | Has failed |
|---|---|---|---|---|---|
| A1Media Monitor & Materiality Triage | Materiality calibration against a 200-story human-labelled set; target at or above 0.80 rank correlation with the CCO’s own ordering. Re-run quarterly. | Can fail | 0.78target 0.8 | Below target | Yes |
| A2Social, Community & Employee Listening | Lead time on themes that later reached earned media, measured against A1’s first detection. Target: 72 hours or more of advance warning on 60% or more of material themes. | Can fail | 63.4%target 60% | Passing | Never |
| A3Policy & Regulatory Radar | Surprise rate: material regulatory developments that reached the company through a channel other than A3. Target below 5% within 12 months. | Can fail | 3.4%target 5% | Passing | Yes |
| A4Competitive Share-of-Voice | Whether the differentiation read changes a budget or a topic choice within the quarter. An intelligence output that changes no decision is a cost line. | No number | — | Judgment | Never |
| A5AI Answer Visibility | Detection of material misrepresentation ahead of it being reported by an employee, journalist or customer. | No number | — | Judgment | Never |
| A6Early Warning & Issue Emergence | Two tracked numbers: median lead time versus public emergence, and precision at the briefing threshold. Target 7 days or more of lead time at 70% or better precision by month 18. | Can fail | 5.2 daystarget 7 days | Below target | Yes |
| B1Stakeholder Graph Builder | Coverage of actors who later proved material, and accuracy of relationship-owner attribution. A graph that names the wrong owner is worse than no graph. | No pass mark | 91.2% | No pass mark | Never |
| B2Stakeholder Simulator | Calibration: did the reaction actually observed after release match the simulated one? Brier-scored per persona, reported quarterly, and the basis for retiring personas that do not predict. | No pass mark | 0.2235 | No pass mark | Yes |
| B3Synthetic Message Testing | Agreement with real research where both exist. Panels that disagree with reality are recalibrated or withdrawn. | No pass mark | 58% | No pass mark | Yes |
| B4Narrative Health & Drift Detection | Detection of real contradictions later found by a journalist, regulator or competitor. Every such find that B4 missed is a defect with a post-mortem. | No pass mark | 2 | No pass mark | Yes |
| B5Scenario & Foresight | Share of actual crises that matched a live scenario, and freshness of the prepared position at the moment of use. | No pass mark | 66.7% | No pass mark | Never |
| C1Message House Builder | Substantiation rate at G2 on first pass. A message house that fails G2 repeatedly indicates the Evidence Ledger is thinner than the narrative. | No pass mark | 93.1% | No pass mark | Yes |
| C2Executive Voice Drafter | Edit distance from draft to released version, tracked per principal over time. A falling curve means the Voice Corpus is learning; a flat one means it is not. | No pass mark | 9.8% | No pass mark | Yes |
| C3Press Materials & Statements | Share of questions actually asked by journalists that appeared in the prepared Q&A. Target 85% or better. | Can fail | 88.4%target 85% | Passing | Yes |
| C4Internal & Franchise Partner Cascade | Cascade completion time and per-cohort comprehension, against the pre-platform baseline. | No pass mark | 31.5 h | No pass mark | Never |
| C5Social & Owned Content | Engagement per unit cost and narrative alignment. Content that performs while diverging from the Canon is a B4 defect, not a win. | No number | — | Judgment | Never |
| C6Localization & Cultural Adaptation | Position fidelity across markets, measured by B4. Zero tolerance for contradiction; explicit tolerance for emphasis. | No pass mark | 0 | No pass mark | Never |
| D1Intake & Inbox Triage | Two numbers: escalation recall — material items correctly escalated, target 100% — and median time-to-first-response versus the pre-platform baseline. | Can fail | 99.7%target 100% | Below target | Yes |
| D2Journalist Targeting & Pitch Support | Pitch-to-placement rate and time-to-assemble. The target is that the human spends their time on the sentence, not the research. | No pass mark | 18 min | No pass mark | Never |
| D3Executive Preparation | Question anticipation rate, and principal-reported usefulness on a two-question post-engagement prompt. | No pass mark | 4.3 of 5 | No pass mark | Never |
| D4Response, Rebuttal & Red Team | Outcome tracking on response decisions: did responding extend the story? The Precedent Library eventually makes this predictive, which is the point. | No pass mark | — | No reading | Never |
| E1Crisis Detection & Classification | Detection latency from first observable signal, and classification accuracy against the scenario library. Target: material crisis detected within 15 minutes of first public signal. | Can fail | 9 mintarget 15 min | Passing | Yes |
| E2Golden-Hour Playbook | Time from declaration to approved holding statement. Target: under 20 minutes end to end, against a baseline typically measured in hours. | Can fail | 17 mintarget 20 min | Passing | Never |
| E3War Room Coordination | Cross-market statement consistency during live incidents, and commitment completion rate — the number of “we’ll come back to you on that” promises that were actually kept. | No pass mark | 86.4% | No pass mark | Yes |
| F1Performance, Attribution & Reporting | Two tests: did reporting hours fall by 70% or more, and did a budget decision change because of an F1 output? The second matters more. | Can fail | 74.2%target 70% | Passing | Yes |
| F2Learning Loop & Post-Mortem | Measured improvement in the platform’s own prediction accuracy quarter over quarter. If it is flat, the learning loop is decorative. | No pass mark | — | No reading | Never |
The gap the harness cannot see
The agent that scores every other agent is not itself scored
F2 measures whether the platform is getting better at predicting. Its contract has no target, no reading and no failing result on record.
F2 · Learning Loop & Post-Mortem
Measurement · A3“Measured improvement in the platform’s own prediction accuracy quarter over quarter. If it is flat, the learning loop is decorative.”
Target
None
Reading
None
Ever failed
No
The agent that scores every other agent has never been scored. Its own register entry warns that this is the agent most likely to be deprioritized.
The same shape appears at D4. Ninety-six response decisions logged as predictions. Not one has had its outcome written back. A prediction with no outcome written back is not a prediction; it is a note.
The control that stops the simulator agreeing with itself
“Adversarial personas mandatory. If simulated reactions increasingly agree with the recommended position, the simulator has become a rubber stamp.”
The portfolio Brier score is 0.2235 against a naive baseline of 0.250, and it has beaten that baseline for 4 consecutive quarters where the phase gate asks for 2. The simulator predicts. What it does not do is predict evenly: rank the 8 personas by how well they forecast and rank them by how often they agree with the recommended position, and the two orderings are almost the same list reversed.
Why this matters more than the headline score
2 of 8 personas sit at or above the 85% agreement line that the specification calls out as the rubber-stamp signature. The agreeable personas are the ones a room finds reassuring, and they are the ones with the worst forecasting record. The Calibrator can rule their results inadmissible at a gate — that authority is the control, and it is held by half a full-time role.
What this layer costs
3 roles, 2.5 full-time equivalents
The register, the calibration loop, the drift alarms, the sentinel and the budgets are carried by three named roles totaling two and a half people.
Agent Ops Lead
1.0 FTERecommends autonomy ceiling changes and runs the override and sentinel loops.
Recommends ceilings. Does not set them.
Simulation Calibrator
0.5 FTECan rule a simulation result inadmissible at a gate. Escalates directly to the CCO.
Half a role. It is the only veto on this page held by one person.
Canon Steward
1.0 FTEForces review or retirement of any position past expiry.
Owns the expiry metric that is currently above its alarm.
Failure modes tripped
4 of 8
Canon drift, slop at scale, unclassified drift and simulator agreement. Each one has a named consequence attached, not an action item.
Agents retired at sunset
2 of 8
Reported here as the system working. A program that never retires anything has thresholds nobody checks.
Silent degradations
0
All 16 over-budget situations escalated to a human rather than quietly producing thinner work.