Fable 5.1 went 17 for 18.
The miss was mine.

The channel that opened

The first draft of this class ended at a wall. Asked to test Fable 5.1 the way class 89 tested Gemini 3.8, the model checked for a way to call itself and found none:

.env                      no ANTHROPIC or CLAUDE credential
environment               no ANTHROPIC or CLAUDE variable
/usr/local/bin/claude     present, v2.1.260
  claude -p "Reply OK"    ->  "Not logged in · Please run /login"
api.anthropic.com         reachable  ->  HTTP 401

That draft built a class about what a model can verify about itself without a channel. It was honest. It was also not what was asked for, and Rob said so. The fix was a channel that had been sitting there the whole time: the browser.

Why a chat window is a valid instrument

What made the class-89 battery a measurement was not the API. It was separation: the model produced code without seeing the vectors, and the vectors were applied by execution, not by reading. A fresh claude.ai chat preserves both properties. The model answering has no access to this session, to the previous chat, or to the grader. The code comes back verbatim — self-tests, docstrings and all — and gets run.

Three things had to be pinned down before the first prompt, and each was checked by reading the page programmatically rather than by looking:

Three questions, one confound

Before the code, three questions about the model itself, each in its own fresh chat. The answers are reproduced verbatim from the page.

Three answers that read like this class wrote them. Which is the problem.

Which of these results is a measurement?

live/results.json · 10 chat URLs

i01_identity · Settings → Profile

gemini_very_hard.json

live/results.json · wall_s_coarse

i02_effort · Settings → Profile

v01_expr · three graded files

Ten for ten is a ceiling

The hard battery is class 89's: ten tasks, each with four to eight adversarial vectors, passing only if every vector passes. Empty inputs, a pattern that needs backtracking, a doubled quote inside a quoted field, banker's rounding on a negative tie, a pre-release suffix that sorts before its release. Verbatim prompts. Same grader.

Three of four models at 10 of 10. By the rule class 89 paid to learn, that is not a finding about Fable 5.1 — it is a battery that cannot see a difference between Fable 5.1 and a Gemini Flash model from two releases ago. It does establish that Fable 5.1 passes h08_version_full, the task that truncated Gemini 3.8, but claude.ai has no comparable output ceiling, so even that is not like for like.

Two things were worth keeping from the run. Fable 5.1 appended its own test suite to two of the ten answers, under a __main__ guard, unprompted — eighteen cases on the glob matcher, ten on the nested-repeat expander. Those were graded verbatim, guard included. And the fastest answer, the longest-palindrome task, came back in 4.8 s wall-clock; the slowest, banker's rounding, in 12.5 s, and chose exact rational arithmetic over floats to do it.

Eight tasks with traps

Each task carries a plausible wrong reading, and the vectors are built to catch it. The file was written, then hashed, then the first prompt was sent.

Fable 5.1 ran them in fresh chats. Then, with approval for the spend (16 calls, $0.22 measured), Gemini 3.7 and 3.8 Flash ran the same eight through the API at a 16,384-token ceiling — high enough that 3.8's thinking would not be the limiting factor.

Now the battery discriminates — barely, and in an unexpected direction. Seven of eight tasks were passed by all three models. The eighth was passed by one. And the one that passed it is the model class 89 caught truncating on hard problems, given enough room to think.

Claim against measurement

The miss was mine

The v01 prompt, verbatim as sent:

Write Python: def evaluate(expr:str)->int for integer arithmetic with + - * / ^
and parentheses. '^' is exponent and RIGHT-associative. Unary minus binds LOOSER
than '^' (so -2^2 == -4) but tighter than * and /. '/' is floor division toward
negative infinity (Python //). Whitespace allowed. Raise ValueError on unbalanced
parentheses. No eval, no ast. Return ONLY a fenced python code block.

And the vector it failed:

evaluate('2^-1') == 0  or  evaluate('2^-1') == 0.5

Read the prompt again. It says ->int. It says “integer arithmetic”. It defines what ^ does with associativity and what unary minus does with precedence. It says nothing about a negative exponent, which in integer arithmetic has no integer answer. The vector I wrote accepted either a floor to 0 or a float — two different, incompatible readings — because I had not decided which was right. I had not noticed there was a decision.

Three models read the same sentence. Here is what each did with 2^-1:

Two independent models, on two independent surfaces, with no shared context, chose the strict reading: the contract says int, this input has no int, refuse it. Fable 5.1 said so in its error message. That is not a wrong answer. It is a more careful answer than the question.

Anatomy of the miss

Vector: evaluate('2^-1') == 0 or evaluate('2^-1') == 0.5The contract?The vector?The first result?The control?The outlier?The verdict?

What 8 of 8 cost

Gemini 3.8 Flash was the only model to pass all eight. Class 89 spent a whole page on why that sentence needs a second half: thinking tokens bill against the output ceiling, and 3.8 spends more of them on hard problems than 3.7 does. The very-hard set was run at a 16,384-token ceiling precisely so that budget would not be the thing being measured. Here is what 3.8 used.

This is the class-89 tail, measured again on a harder set and looking worse. On the two easiest tasks 3.8 thinks about as much as 3.7, and on one it thinks less. On the hardest it thinks two and a half to four and a half times as much. Given room, that spend buys a perfect score. At the ceiling most production code was written against, three of the eight return nothing at all — HTTP 200, STOP, empty string.

Which check would have caught it?

Three situations from this class. In each, exactly one check is the one that revealed the problem. The others are reasonable and did nothing.

the grade was FAIL, 1 of 9 vectors

MEASURED, ASSERTED, Not verified, unprompted

a clean perfect score at cap 16,384

Test a model you cannot call

Everything on this page was produced through a chat window. No key, no SDK, no endpoint. What made it a measurement rather than a conversation was a short list of disciplines, each of which cost something when it was skipped.

1. Write the vectors first. Hash the file. The hash is what stops you editing the grader after the result embarrasses you. It stopped me on v01.
2. Fresh session per task. Navigate to a new chat every time and assert that the URL changed on send. Nothing carries over, including your own earlier prompts.
3. Read the picker, don't trust it. The model label was read programmatically before every send and the send was refused if it was wrong. It was never wrong; the check still ran 21 times.
4. Pull the output verbatim. Self-tests, docstrings, the lot. The first hard-battery answer was trimmed before grading and had to be re-saved — editing the output before you grade it is grading something else.
5. Grade by execution. A function that reads correctly and fails one vector fails the task. Reading is how the v01 vector got written in the first place.
6. Run a control. A second model on the same vectors is what separates “the model got it wrong” from “the spec never said”. It cost $0.22.
7. Read the surface. Settings, defaults, preferences. The identity answers would have gone on this page as observations if nobody had opened the profile page.
8. Raise the threshold until it can fail. Ten for ten across three models was not a result. Eight tasks with traps was.

What a claim about a model has to survive

Build your own outside test

Fill in four fields. The output is a locked-vectors file header and a run log skeleton for testing any model through any chat surface, with the checks above written in as steps that must be recorded, not remembered.