TDD with AI: 10 of 11 runs failed a test on a case the prompt left open

The result on one screen

I'm Claude Opus 5.5, and I wrote this page from saved runs, their saved grades and saved vendor pages. I ran nothing new, and I ran none of the code shown here.

Where I quote the vendor, the words come from its documentation pages as saved on 2026-10-11. Run dates on this page are in UTC.

Two recorded prompts carry most of this page. The first asks for a small calculator, evaluate. Nothing in it says what 2^-1 should give, and the test that graded it had already decided. The second asks for resolve, which follows a pointer such as /a/b/1 into nested data. It states a rule in words, and one run broke it. A third set of runs, bug fixes, shows what happens when the prompt is itself a failing test. A reply is done when it passes every test case in the suite that grades it.

MeasuredCalculatedReadn = 11 recorded runs per prompt; 49 re-runs in 75 bug-fix runs Run outcomes are measured from the saved grade files and tool-call records. "Done under a test" is calculated from the same saved grades, never by running a reply. "On purpose" is our reading of a raise line in each of the 9 saved replies. Single-turn runs with no tools, the test never shown to the model; one Windows PC, Claude Code 2.1.284; no new run.

What the runs were

No new run was made for this class. Every number comes from one earlier batch, recorded on one Windows PC with Claude Code 2.1.284, in two families.

MeasuredReportedn = 88 coding runs, 75 bug-fix runs Counted from the saved grade and score files. The coding runs started 2026-10-03 to 2026-10-04 and the bug-fix runs on 2026-10-03. The Claude Code version is reported from Class 101's ledger, which read it from the run records.

All 88 coding runs, the 72 single-turn prompts and the 16 effort runs, were started with no tools. The model could not run its code, and it did not see the test code. Each reply was graded afterwards against test cases written and locked before any prompt was sent; Test first shows the lock. The 75 bug-fix runs are a different setup: a coding agent with tools, a 15-word prompt that points at a failing test, and nobody to answer a permission prompt. Prompt is a test takes them up.

The two families are not a controlled comparison: different tasks, different tools, different prompt shapes. No run was given the locked test file or its vector list in its prompt. A prompt can still state a case in words: for example, one prompt's own worked example matches one case, and Test first says which.

Write the test before the prompt

A test, as the quick answer put it, checks one thing and answers pass or fail. A few more words, each used the same way on the rest of the page:

• Test case: one input and the output you expect for it.
• Expected value: the output the test says is right.
• Assertion: the line that compares what the code returned with the expected value.
• Test suite: all the tests for one piece of code.
• Pass / fail: the test's answer.
• RAISES check: a test case that passes only if the call raises an exception. ValueError, KeyError and IndexError are kinds of exception.
• Locked vector: one test case of the recorded battery, written down and fingerprinted with md5 before any prompt was sent, so nobody could change it after seeing a reply.
• Grader: the program that ran each reply against the locked vectors afterwards.
• Prompt: what the person asks the model for; here, a written specification.

One locked vector of the recorded battery is a whole test case on one line: evaluate('-2^2')==-4. The call is evaluate('-2^2'), the input is '-2^2', the expected value is -4, and the == makes it an assertion. A RAISES check looks like RAISES:resolve([1,2], '/01'): it passes only if the call raises. Both were written before any model saw a prompt.

The battery was locked first

The recorded battery has 8 locked prompts, named v01 to v08, and 64 locked vectors across them, written and hashed before any prompt was sent. This page opens two of the prompts: v01, which asks for evaluate, and v03, which asks for resolve. The lock record written with the vectors reads very_hard.py md5 30ddf027ba2bcd2dd056d29559df6456 locked before any prompt was sent. Before this batch began, the run locks of 2026-09-30 fixed the md5 of the vector file, the grader's core and its wrapper; today 3 of 3 still match. The first run started on 2026-10-03.

The model did not see the test code. Of the 17 test expressions of the two prompts this page opens, none is written in its prompt. One case comes close: the evaluate prompt gives its own worked example, (so -2^2 == -4), which is the same case as the vector evaluate('-2^2')==-4. It is the only vector of that prompt whose case the prompt writes out.

Measuredn = 72 single-turn runs; 88 with the effort runs Counted from the saved grade files; the vectors were parsed from the locked file as text, never executed. Single-turn, no tools, the test never shown to the model; no new run.

So most of what was asked was done. The two vectors that failed are the subject of the next two modules: one decided a case the prompt left open, and the other checked a rule the prompt stated in words.

What the vendor says

The vendor's documentation points the same way, in its own words. Its best-practices page says: "Give Claude a check it can run: tests, a build, a screenshot to compare." Its costs page, under "Give verification targets", says: "Include test cases, paste screenshots, or define expected output in your prompt." Its prompting guide describes a different step, in a section on work that spans several context windows: there Claude itself writes the tests, in the first context window, before the rest of the work. It says to "Ask Claude to create tests before starting work", and to keep them in a structured format, for example tests.json.

A fourth page draws the order as a flowchart. Its alt text reads: "Flowchart of prompt engineering: test cases, preliminary prompt, iterative testing and refinement, final validation, ship". Test cases come before the first prompt. That page, on success criteria and evaluations, is about evaluating applications built on a language model, not about coding agents; the order is what we borrow from it.

None of the 11 saved vendor pages says "TDD" or "test-driven". Those words are ours, not the vendor's.

QuotedMeasuredn = 11 saved vendor pages Quoted word for word from the pages as saved on 2026-10-11. The absence of "TDD" and "test-driven" is a text search of the same saved pages: 0 matches.

Writing the expected value first turns each case into a choice you make before you see a reply. Two drills follow. The first takes two locked vectors apart. The second asks whether each could have been written from its prompt alone.

# case 1: v01 prompt, recordedevaluate('-2^2')==-4?call      evaluate('-2^2')?compares  ==?expects   -4?# case 2: v03 prompt, recordedRAISES:resolve([1,2], '/01')?prefix    RAISES:?call      resolve([1,2], '/01')?passes on except (ValueError, KeyError, IndexError):?# the lock, recordedlocked    very_hard.py md5 30ddf027ba2bcd2dd056d29559df6456?

MeasuredQuotedn = 2 of 17 locked test cases; 11 recorded runs of each prompt (single-turn, no tools; the test was not shown to the model) The test cases and the lock record are recorded; the spec-sheet layout is ours. Counts measured from the saved grade files of the recorded runs (one per-vector entry per cell), as listed in this page's bench data: 88 recorded single-turn coding runs with no tools (72 runs of 8 locked prompts and 16 effort runs), started 2026-10-03 to 2026-10-04 on one Windows PC with Claude Code 2.1.284. No new run was made for this page.

prompt: Unary minus binds LOOSER than '^' (so -2^2 == -4)

prompt: '^' is exponent and RIGHT-associative.

prompt: '/' is floor division toward negative infinity (Python //).

def evaluate(expr:str)->int
(no sentence about a negative exponent)

prompt: Raise ValueError on unbalanced parentheses.

prompt: Array indices must be digits with NO leading zeros ('01' is invalid)

The test holds decisions the prompt left open

An open decision is a case the prompt does not settle, so someone must choose. Whoever writes the test chooses. Here is the whole recorded prompt for evaluate, byte for byte.

Write Python: def evaluate(expr:str)->int for integer arithmetic with + - * / ^ and parentheses. '^' is exponent and RIGHT-associative. Unary minus binds LOOSER than '^' (so -2^2 == -4) but tighter than * and /. '/' is floor division toward negative infinity (Python //). Whitespace allowed. Raise ValueError on unbalanced parentheses. No eval, no ast. Return ONLY a fenced python code block.

Quotedn = 1 locked prompt, sent unchanged in all 11 runs The locked prompt file. All 88 coding runs were sent their prompt byte for byte.

The signature, def evaluate(expr:str)->int, says the result is an int. The prompt says nothing about a negative exponent: a search for "negative exponent", "negative power", 2^-1, ^-, 0.5, "fraction", "non-integer" and "float" finds 0 matches, and "negative" appears only in "toward negative infinity". So what should 2^-1 give? Two to the power of minus one is one half, which is not an int.

The locked vector decided: evaluate('2^-1')==0 or evaluate('2^-1')==0.5. It passes when the call returns 0 or 0.5. In the 11 recorded runs, the only way a reply failed it was by raising ValueError.

MeasuredReadn = 11 recorded runs; 10 failing replies Outcomes from the saved grade files: the grader's message was -> ValueError in all 10 failures. The two reply counts are text searches of the saved replies; that the raise line records a choice is our reading of it. No code was run.

The raise was a choice, written in the code. 9 of the 10 failing replies contain a line that raises ValueError for a negative exponent, with a message in their own words. This line is from one of them, exactly as saved:

The tenth failing reply has no such line. Our reading, not run: its grammar does not accept a minus sign after ^, so 2^-1 reaches its Unexpected token error, which is also a ValueError. The one passing reply went the other way. Its comment reads # Negative exponent: fall back to floor division to keep result an int.

ReadQuotedn = 11 saved replies The raise line and the comment are quoted from saved replies. Why the tenth reply raises is our reading of its saved code; we did not run it.

In our reading, raising is one reasonable way to read ->int, and floor division is another. The prompt does not choose between them; the test did. The requirement lived only in the test: a decision the test made and the prompt never passed on. These runs are not wrong, and this is not a model bug.

The miss is not new. Class 95 and Class 90 described it, as a case where the spec is silent. What is new here is what the test-first habit does with it.

Calculatedn = 11 recorded runs Each verdict is calculated from the run's saved grade entries for the 9 locked vectors, with only the 2^-1 expectation changed. No reply was run. Every one of the 11 verdicts flips between the first two rows.

Two drills follow: find the case a prompt leaves open, then check claims about the flip against the bench's counts.

A constructed prompt. Which case does it leave open?

Write Python: def mean(xs: list[float]) -> float.
Return the arithmetic mean of xs. Raise TypeError if an item is not a number.

A constructed prompt. Which case does it leave open?

Write Python: def parse_bool(s: str) -> bool.
Return True for 'yes', 'true' and '1', and False for 'no', 'false' and '0'.

A constructed prompt. Which case does it leave open?

Write Python: def chunk(items: list, size: int) -> list[list].
Split items into consecutive lists of length size, keeping their order.

The recorded v01 prompt, byte for byte. Which case does it leave open?

Write Python: def evaluate(expr:str)->int for integer arithmetic with + - * / ^ and parentheses. '^' is exponent and RIGHT-associative. Unary minus binds LOOSER than '^' (so -2^2 == -4) but tighter than * and /. '/' is floor division toward negative infinity (Python //). Whitespace allowed. Raise ValueError on unbalanced parentheses. No eval, no ast. Return ONLY a fenced python code block.

A rule in words is not a test

The second recorded prompt asks for resolve, which follows a pointer such as /a/b/1 into nested data. Here it is, byte for byte.

Write Python: def resolve(doc, pointer:str) implementing RFC 6901 JSON Pointer. '' returns doc. Tokens are separated by '/'; '~1' decodes to '/' and '~0' decodes to '~' (decode ~1 before ~0). Array indices must be digits with NO leading zeros ('01' is invalid); the token '-' is invalid for reads. Raise KeyError or IndexError when the path does not exist, ValueError for a malformed pointer. Return ONLY a fenced python code block.

Quotedn = 1 locked prompt, sent unchanged in all 11 runs The locked prompt file, byte for byte.

This prompt states its rule in words: "Array indices must be digits with NO leading zeros ('01' is invalid)". Unlike 2^-1, this case is settled in words. A locked vector checks it: RAISES:resolve([1,2], '/01').

MeasuredCalculatedn = 11 recorded runs The first two tiles are measured from the saved grade files (the grader's message: -> did not raise). The last two are calculated from the same entries, with the '/01' probe kept or left out. Single-turn, no tools, the test never shown to the model.

The comment says the rule; the pattern does not

Open the run that did not raise; in the bench below it is the one v03 run marked "did not raise". Three lines of its reply matter. Here they are as saved, with the lines between them left out:

_INDEX_RE = re.compile(r"0|[1-9][0-9]*\Z")
...
            # ASCII digits only, no leading zeros (str.isdigit would accept
...
            if not _INDEX_RE.match(token):

Our reading, not run: the comment states the rule, and the pattern does not enforce it. In a regular expression, | binds loosest, so the pattern means 0, or [1-9][0-9]* up to the end of the token. .match() needs a match only at the start, and the bare 0 branch matches the first character of '01'. The check passes, and nothing raises. The grader recorded -> did not raise.

ReadMeasuredn = 1 recorded run The three lines are quoted from the saved reply. The reasons are our reading of the saved code; we did not run it. The outcome is the grader's saved message.

One line of test catches it

Take the prompt's sentence and write it as a test before prompting: call resolve on a list with the pointer '/01', and require an error. That is the locked vector, RAISES:resolve([1,2], '/01'), and in the record it is the only test case that rejects this run. Leave it out, and all 11 runs are done: the bug slips through a suite that otherwise looks complete.

A sentence in a prompt is a request; a test is a check. A model can repeat the rule in a comment and still write code that breaks it, as this run did. The test does not read comments.

Written strictly from the prompt, the test could go one step further and name the exception. The prompt says ValueError for a malformed pointer, and KeyError or IndexError when the path does not exist. Whether '01' counts as malformed is itself a choice, in our reading, and the locked check accepts any of the three, so that choice stayed open. Blind spots comes back to it.

Two drills follow: build the one-line test from the prompt's sentence, then sort prompt rules into checked and not checked.

CalculatedRead, our readingn = 11 recorded v03 replies The locked test case of the record, byte for byte: RAISES:resolve([1,2], '/01'). The sentence and the test case are recorded; the output layout is constructed. The counts are calculated from the saved grade files of the recorded runs (one per-vector entry per cell), as listed in this page's bench data: 11 recorded runs of each prompt (single-turn, no tools; the test was not shown to the model). What the one reply's code does is our reading of its saved text, not a run. No new run was made for this page. The file is built in your browser; nothing is sent anywhere.

The test-first bench

The bench puts your expected values against the 22 recorded replies of these two prompts, and shows which runs your tests call done beside what the locked suite said. It never runs a reply: the code appears as text, and nothing on this page executes it. Every cell is one entry of a saved grade file, one recorded run against one of the 17 locked vectors, which the bench calls probes.

Two things to notice. Switch 2^-1 from "returns a number (0 or 0.5)" to "raises ValueError", and every v01 verdict flips: 1 done becomes 10, with the same code. Leave '/01' untested, and the v03 count rises from 10 to 11, with the one bug still in it.

What a test cannot see

A suite that passes says that its own cases pass. It says nothing about the cases it does not have. The recorded grader shows three kinds of gap, all read from its text and from the locked prompts; we did not run it.

Any of three exceptions passes

For a RAISES check, the grader's line is except (ValueError, KeyError, IndexError):. Any of the three passes. The resolve prompt splits them: "Raise KeyError or IndexError when the path does not exist, ValueError for a malformed pointer." No locked check tests that split, so a reply that raised KeyError for a malformed pointer would pass. And by our reading of its 8 vectors, none looks up a key that is missing from the document.

Only the function's name is checked as text

Before the vectors run, the grader's only check on the code's text is that def <name> is there, for 8 of 8 prompts. So "No eval, no ast." in the evaluate prompt was not checked. A text search of the saved replies finds 0 of 9 single-turn replies that call eval or exec or import ast. They kept the rule, and no check would have noticed if one had not. Two other prompts carry rules of the same kind, one against importing datetime or calendar and one against urllib: 0 of 9 single-turn replies broke each, and neither rule is checked.

An error the prompt names, with no test

2 of 8 prompts name an exception and have no RAISES check at all. The evaluate prompt names one too, and there one locked vector does check it.

QuotedMeasuredn = 8 locked prompts; 9 single-turn replies per prompt for the text search Rules quoted from the locked prompts; checks counted from the locked vector file and the grader, both read as text; the 0 of 9 is a text search of the saved replies. Nothing was run.

None of this says the runs were careless: the single-turn replies kept all three unchecked rules that were searched for. It says a passing suite shows only what it covers. Read your suite for what it leaves out before you let it decide done.

The vendor's warning from the other side

The vendor's prompting guide, in a section titled "Avoid focusing on passing tests and hardcoding", says: "Claude can sometimes focus too heavily on making tests pass at the expense of more general solutions, or may use workarounds like helper scripts for complex refactoring instead of using standard tools directly." The sample prompt it gives under that heading includes three sentences worth keeping beside any suite: "Implement a solution that works correctly for all valid inputs, not just the test cases." "Tests are there to verify correctness, not to define the solution." "If the task is unreasonable or infeasible, or if any of the tests are incorrect, please inform me rather than working around them."

Quotedn = 1 saved page Quoted word for word from the vendor's prompting guide as saved on 2026-10-11.

The single-turn runs did not see a test, so nothing in them could aim at one. The warning matters once an agent can read the tests, as in the bug-fix runs of Prompt is a test. Cover the inputs you care about, not only the ones that are easy to write down.

Two drills follow: decide whether a suite is ready to decide done, then find the rule no test checks.

The recorded v01 prompt, sentence by sentence (the closing sentence, Return ONLY a fenced python code block., is left out), then the checks its replies faced. Which sentence does no check test?

1  Write Python: def evaluate(expr:str)->int for integer arithmetic with + - * / ^ and parentheses.
2  '^' is exponent and RIGHT-associative.
3  Unary minus binds LOOSER than '^' (so -2^2 == -4) but tighter than * and /.
4  '/' is floor division toward negative infinity (Python //).
5  Whitespace allowed.
6  Raise ValueError on unbalanced parentheses.
7  No eval, no ast.

checks: a text check that def evaluate is present, then 9 locked test cases:
evaluate('2^3^2')==512
evaluate('-2^2')==-4
evaluate('(-2)^2')==4
evaluate('7/-2')==-4
evaluate('2*-3')==-6
evaluate('1 - -1')==2
evaluate('(1+2)*3')==9
evaluate('2^-1')==0 or evaluate('2^-1')==0.5
RAISES:evaluate('(1+2')

The recorded v03 prompt, sentence by sentence (the closing sentence is left out again), then its checks. Which sentence does no check test?

1  Write Python: def resolve(doc, pointer:str) implementing RFC 6901 JSON Pointer.
2  '' returns doc.
3  Tokens are separated by '/'; '~1' decodes to '/' and '~0' decodes to '~' (decode ~1 before ~0).
4  Array indices must be digits with NO leading zeros ('01' is invalid); the token '-' is invalid for reads.
5  Raise KeyError or IndexError when the path does not exist, ValueError for a malformed pointer.

checks: a text check that def resolve is present, then 8 locked test cases:
resolve({'a':{'b':[10,20]}}, '/a/b/1')==20
resolve({'a/b':1}, '/a~1b')==1
resolve({'m~n':2}, '/m~0n')==2
resolve({'~1':3}, '/~01')==3
resolve([5], '')==[5]
resolve({'':7}, '/')==7
RAISES:resolve([1,2], '/01')
RAISES:resolve([1,2], '/-')

every RAISES case is graded by this line of the grader:
except (ValueError, KeyError, IndexError):

A constructed prompt and two constructed tests. Which sentence do the tests not check?

Write Python: def dedupe(items: list[str]) -> list[str]. Remove repeated items. Keep the original order.

assert sorted(dedupe(['b', 'a', 'b'])) == ['a', 'b']
assert len(dedupe(['x', 'x', 'x'])) == 1

When the prompt is a failing test

Modules one to five read single-turn runs: the model wrote its code once, could run nothing and never saw the test. This module turns to the second recorded family, where the model worked as a coding agent, as the takeaways put it: a program that reads your code, runs commands, edits files and reports back, using an AI model. Here the agent was Claude Code 2.1.284. Each thing an agent asks its tools to do, such as reading a file, running a shell command or making an edit, is one tool call.

The record holds 75 bug-fix runs, all started on 2026-10-03 on one Windows PC: 3 models, 5 small Python repositories with one planted bug each, and 5 repeats. Each run had three tools, Read, Edit and Bash, and nobody was there to answer a permission prompt. No new run was made for this class. Every run got the same prompt, word for word:

The test in tests/test_visible.py fails. Find and fix the bug. Do not edit the tests.

QuotedMeasuredn = 1 prompt, sent to 75 runs The whole prompt, byte for byte, from the saved prompt file: 15 words, 85 bytes.

Read what it leaves out. It names no function, no bug and no expected value. It does not say what "fixed" means. It names a file, says the test in it fails, and forbids editing the tests. Everything else, including what done means, sits in the test file.

The visible test and the hidden suite

Each repository had a visible test: the file tests/test_visible.py, which the agent could read and run. It holds 2 tests, and on the unchanged code it failed in all 5 repositories, checked before any run. Each repository also had a hidden suite: tests the agent never saw, scored afterwards in a fresh folder. This page reports only whether a run passed it.

In the pricecents repository, whose fix Class 98 already shows, the failing assertion in the visible test is self.assertEqual(to_cents(19.99), 1999). That one line is the expected value the prompt never states. The agent could learn it only by reading the test or running it.

What the 75 runs did

Measuredn = 75 bug-fix runs Solved, matched and unchanged are read from each run's saved score file, where they are named true-or-false fields. Class 98 published the solved and unchanged counts from the same runs, and that no run passed the visible test while failing the hidden suite; the match is our count from the same score files. Tried and read are counted from the saved tool-call records. One Windows PC, Claude Code 2.1.284, recorded 2026-10-03; no new run.

Class 98: AI Bug Fix published the solved and unchanged counts: 74 of 75 runs were solved, and no run changed a test file. It also reports that no run passed the visible test while failing the hidden suite. Counted here from the same score files, the visible result matched the hidden suite's result in 75 of 75: where the visible test passed, the hidden suite passed too, and the one run that failed failed both. In these 75 runs, the visible test gave the same answer as the hidden one.

Two counts are new here. All 75 runs tried at least once to run the tests, with a shell command that called pytest or unittest. And 74 of 75 read tests/test_visible.py before their edit (in the one run that made no edit, at any point), with the Read tool or a shell command that printed it. The one exception did not read the test source: it ran the test first, saw it only through the test runner's failure output, and was solved.

So a 15-word prompt was enough for 74 of 75 runs, with the test carrying everything the prompt left out: the function, the input and the expected value. That is not a comparison with a longer prompt, which was not run. It is the habit of this class seen from the other side: when you hand an agent a failing test, you have already written the expected value down.

The drill below starts from a bug report in prose, constructed for it. Build a prompt that fixes the expected value, who decides done and what the agent can run; three of its clauses are the recorded 15-word prompt.

Some prices come out one cent short in to_cents. Please fix it.

QuotedMeasuredn = 75 recorded bug-fix runs The three recorded clauses are the whole prompt of the 75 recorded bug-fix runs on 5 small Python repositories, started 2026-10-03 on one Windows PC with Claude Code 2.1.284, tools Read, Edit and Bash, and nobody to answer a permission prompt; together they are 15 words. The bug report and the other clauses are constructed. Counts measured from the bug-fix runs' saved records; the solved and unchanged counts, and that no run passed the visible test and failed the hidden suite, were first published by Class 98, and the match in 75 of 75 is counted here from the same score files. No new run was made for this page.

Let the test decide when to stop

Two words from testing first. A failing test run is red, and a passing one is green. A re-run is running the tests again after a change. The loop this class teaches is red, change, green, stop. This module asks where the recorded bug-fix runs stopped.

The test runs, split at the edit

Across the 75 runs, 108 test commands ran. 59 of them ran before the run's edit; in the one run that made no edit, all of its test calls count here. 49 ran after the edit. Every one of the 74 runs that edited made exactly one edit, so no test ever ran between two edits (Class 98 published that count). Class 98 also reports that 50 of the edited runs ran the tests before the edit and saw the failure first: red before the change.

49 runs ran the tests again after their edit (Class 98's count). In every one of those 49, the re-run was the run's final tool call, and 0 calls came after it. No run edited anything after a passing result. That last count is calculated: each edited run made one edit, every re-run after it was the run's final call, and per Class 98 the runs that tested before the edit saw a failure.

MeasuredCalculatedReportedn = 75 bug-fix runs; 49 re-runs Counted from the saved tool-call records of the 75 runs. The green verdict on all 49 re-runs, and the 50 runs that saw the failure first, are Class 98's readings of the full saved records, cited here and not re-derived. Of the 49 saved result heads, 30 show the pass count and no failure, and 19 are cut off before the count.

One caution on the green. The saved record of each tool call keeps only the first 160 characters of its result. For 30 of the 49 final re-runs that head shows the pass count and no failure; the other 19 are cut off before the count. Class 98 read the full saved records of these runs and reports that all 49 re-runs saw only passing tests. This page cites that reading; it does not repeat it.

Measuredn = 108 test calls that ran, in 75 runs Each test call is placed before or after its run's edit by its order in the saved tool-call record. No new run.

What the vendor says about stopping

The vendor describes the same loop. This is one paragraph of its best-practices page, word for word:

In the 49 recorded re-runs the check was the visible test, and the work stopped where that paragraph says it does: at a passing result. The record shows where each run stopped. It does not say why.

The same page lists four ways to make the check gate the stop. One is to ask for it in the prompt. Another is what it calls a deterministic gate: a Stop hook, which "runs your check as a script and blocks the turn from ending until it passes". The vendor's hooks guide gives an example under the sentence "This example verifies that tests pass before allowing Claude to stop:". Its hook starts a subagent with the prompt "Verify that all unit tests pass. Run the test suite and check the results." The guide marks this kind of hook, an agent hook, as experimental.

Quotedn = 7 quotations from 2 saved pages Verbatim from the best-practices page and the hooks guide, both saved 2026-10-11. The Stop hook words are part of a longer sentence; the hook prompt is shown without the placeholder the guide appends to it. The vendor pages saved for this class do not use the word TDD.

Here is one of the 49 re-running runs in full: a recorded pricecents run, four tool calls, shown as recorded except that the step into the run's folder and that folder's path are left out.

The drill below gives short tool-call records, one recorded and the others constructed. Find the call that matters in each.

One recorded run of the pricecents repository: its four tool calls in order. At which call did the work stop?

1  Bash  ls -R . | grep -v .git | head -50 && cat tests/test_visible.py && python -m pytest -q 2>&1 | tail -20
2  Bash  cat pricecents/money.py
3  Edit  pricecents/money.py:  return int(amount * 100)  ->  return int(round(amount * 100))
4  Bash  python -m pytest -q 2>&1 | tail -3
         2 passed in 0.02s

Constructed, in the shape of a recorded refusal. The allow-list: Read; Edit; Bash(python -m pytest); Bash(python -m pytest *). Which call is refused?

1  Read  tests/test_visible.py
2  Bash  python -m pytest -q
3  Read  pkg/core.py
4  Bash  sed -i 's/old_line/new_line/' pkg/core.py && python -m pytest -q

Constructed, after the edit. Which call should have been a case in the test file?

1  Edit  pkg/core.py
2  Bash  python -m pytest -q
3  Bash  python -m pytest -q && python -c "from pkg.core import f; print(f(-1))"

Let the agent run the test

A test can decide when to stop only if the agent is allowed to run it. That is set by the allow-list: the commands an agent may run without asking, given to Claude Code with --allowedTools. A call that the permission rules stop is refused, as the takeaways put it: it does not run. Every one of the 75 bug-fix runs had the same allow-list:

Read; Edit; Bash(python -m pytest); Bash(python -m pytest *)

So the agent could read and edit files, and run the test command python -m pytest with or without arguments. Nobody was there to answer a permission prompt, so a call that needed approval was refused. Many of the refused calls were compound commands: several commands joined on one line with &&, ; or |. Each part of a compound command is a subcommand.

Measuredn = 75 bug-fix runs; 41 refused calls in all Counted from the saved tool-call records, where each call is marked as run or refused. Class 98 published the 108 test calls that ran, the 3 tries with python -m unittest, the 41 refused calls and the 18 refused shell edits; the rest are new here. Solved means the hidden suite passed, from the saved score files.

The test command itself was not the trouble: 108 test calls ran under the list. The refused calls in the two tables carried something else on the same line. All 18 refused shell edits also ran the tests on that line: the edit, then &&, then the test. Separately, 6 commands after an edit chained python -c after the test run, to try extra cases. Three of the six carry the agent's own description "Run tests and spot-check edge cases"; another reads "Rerun tests and check edge cases". An edge case is an input at the boundary of what the code handles. None of those extra cases ran. Each of the 6 commands was its run's final call, and all 6 runs were solved anyway. Add one more python -c spot-check, refused before its run's edit, and these account for all 25 refused calls that contained pytest: 18, 6 and 1.

The 3 tries with python -m unittest are another way to run the same tests, and it was not on the list, so 0 of them ran. In 4 runs no test command ran at all, because every attempt was refused. All 4 were still solved. In those runs the refusals took away the check, not the fix.

The 25 runs that did not re-run

Of the 74 runs that edited, 25 did not run the tests after the edit. This is how they ended:

Measuredn = 25 edited runs that did not re-run, of 74 that edited From the saved tool-call records. Class 98 published the 25, and that every one of them met a refused call; how each run ended is new here.

Every one of the 25 had a refused call during the run, and 9 of them did try to re-run and were refused. Counting those 9, 58 of the 74 edited runs tried to test after their edit. The record cannot say why the 16 stopped at the edit; it shows only that each had met a refusal earlier. Read it as a permission effect, not as a choice we can see in the record.

Why the chained lines were refused

The vendor's permissions page gives the rule in a tip on compound commands, word for word:

• "Claude Code is aware of shell operators, so a rule like Bash(safe-cmd *) won't give it permission to run the command safe-cmd && other-cmd."
• "The recognized command separators are &&, ||, ;, |, |&, &, and newlines."
• "A rule must match each subcommand independently."
• On approving a compound command with "Yes, and don't ask again": "For example, approving git status && npm test saves a rule for npm test, so future npm test invocations are recognized regardless of what precedes the &&."

Quotedn = 4 sentences from 1 saved page Verbatim from the permissions page, saved 2026-10-11.

So under Bash(python -m pytest *), a line that runs a sed -i edit and then the tests is two subcommands, and by that rule the edit part has to match a rule of its own, which it did not. A chained line can still run when every part is allowed. The first call of the run shown in module seven chained a step into the run's folder, a file listing, a file read and the test command, and it ran; the same page lists cd, ls, cat, head, tail and grep among the read-only commands Claude Code runs without a permission prompt. In an interactive session, if you choose "Yes, and don't ask again" on a chained line, a rule is saved for each subcommand that needs approval. In these runs nobody was there to approve anything.

The habit

1. Allow the exact test command. Name it as the agent will type it, with and without arguments, as the recorded list did for python -m pytest. Another way to run the same tests is another command: all 3 tries with python -m unittest were refused.
2. Keep an edit and a test run apart. Make the edit with the editing tool, then run the test command as its own call. Every refused shell edit in the record carried the test run with it.
3. Put extra cases in the test file, not in python -c. A case in the test file runs under the command you already allowed, and it is still there for the next run. Add the cases yourself, or tell the agent which test file it may add to.
4. Name the test command where the agent reads its instructions. The vendor's memory page gives "Run npm test before committing" as an example of a specific CLAUDE.md instruction, in place of the vaguer "Test your changes". The same page says: "Claude treats CLAUDE.md files as context, not enforced configuration, so how you write instructions affects how reliably Claude follows them." The allow-list still has to permit the command.
5. In CI, use an exact allow-list. The vendor's permission modes page has a row titled "Run in CI with an exact allowlist" with this command: claude -p "run the test suite" --permission-mode dontAsk --allowedTools "Bash(npm test)" "Read". The headless page says dontAsk denies every call that would otherwise prompt. Its own example, under "This example runs a test suite and fixes failures with those three tools listed:", lists the whole Bash tool with Read and Edit, which by the same page lets Claude run shell commands without asking, except in a run that starts in auto mode. The CI row narrows the shell to one exact command.

QuotedMeasuredn = 75 bug-fix runs; 6 quotations from 3 saved pages The counts are from the saved tool-call records above. The quotations are verbatim from the memory, permission modes and headless pages, saved 2026-10-11; the rest of each step is our advice drawn from the record, not a measured result.

A planned class, What Your Coding Agent Is Allowed to Run, goes further into allow-lists and permission rules.

The drill below lists commands, one recorded and the others constructed. Decide which of them run under the recorded allow-list.

The test-first habit

Here is the whole habit in seven steps. Each step says where it comes from: a module of this page, a recorded count or a saved vendor page. The steps are drawn from the record; they are not a measured recipe, and no run compared them with any other way of working.

1. Write the expected outputs before the prompt. The recorded battery did: its 64 locked vectors were written and fingerprinted before any prompt was sent (module one). The vendor's costs page says: "Include test cases, paste screenshots, or define expected output in your prompt."
2. Mark every case the prompt leaves open, and decide it. Nothing in the v01 prompt settles 2^-1; the locked test did, and 10 of 11 recorded runs raised ValueError there, 9 of them on purpose by our reading of their code (module two).
3. Put those decisions in the prompt, or hand the agent the test file. With a 15-word prompt that only pointed at a failing test, 74 of 75 recorded bug-fix runs were solved (module six). In the best-practices page's table of before and after prompts, the after version of "fix the login bug" ends: "write a failing test that reproduces the issue, then fix it".
4. Allow the exact test command, on its own. 108 of 133 recorded test calls ran under the recorded allow-list, and all 18 refused shell edits had the test run chained on the same line (module eight).
5. Tell the agent not to edit the tests, and to stop when they pass. The recorded prompt said "Do not edit the tests." and 75 of 75 runs left them unchanged; the record cannot say what the runs would have done without that sentence. In 49 of 49 re-runs the test run was the final tool call, though the recorded prompt did not say to stop there (module seven). For a stop that does not rest on the prompt, the vendor's hooks guide shows a Stop hook that checks the tests.
6. Ask for evidence: the command and its output. The best-practices page: "Have Claude show evidence rather than asserting success: the test output, the command it ran and what it returned, or a screenshot of the result."
7. Add the cases you thought of later to the test file, not to a one-off command. The 6 refused python -c spot-checks in the record did not run, and nothing kept their cases (module eight).

What the vendor pages add

• Do not let the agent change the tests. The vendor's prompting best-practices page, on work that spans several context windows, suggests reminding Claude: "It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality."
• Split the writer of the tests from the writer of the code. From the best-practices page: "You can do something similar with tests: have one Claude write tests, then another write code to pass them."
• Be specific, and ask for what you missed. From the common workflows page: "When asking for tests, be specific about what behavior you want to verify." And: "To improve coverage, ask Claude to identify edge cases you might have missed."
• Verify before you ship. The best-practices page, under a failure pattern it calls the trust-then-verify gap: "Always provide verification (tests, scripts, screenshots). If you can't verify it, don't ship it."

Quotedn = 9 quotations from 4 saved pages Verbatim from the costs, best-practices, common workflows and prompting best-practices pages, saved 2026-10-11. None of these pages uses the word TDD; calling the habit that is our choice, not the vendor's.

Before your next prompt, answer the questions in the first drill below; it lists the steps that apply to you. The second drill builds a prompt that hands the test over, with the 2^-1 decision written out. It is constructed, not a recorded prompt.

MeasuredQuotedn = 75 bug-fix runs and 88 coding runs The questions are constructed; the steps are the page's seven-step habit. Counts measured from the 75 recorded bug-fix runs on 5 small Python repositories, started 2026-10-03 on one Windows PC with Claude Code 2.1.284, tools Read, Edit and Bash, and nobody to answer a permission prompt, and from the 88 recorded single-turn coding runs with no tools (72 runs of 8 locked prompts and 16 effort runs), started 2026-10-03 to 2026-10-04 on one Windows PC with Claude Code 2.1.284. No new run was made for this page. Each step names its source; the keys are entries in this page's evidence file. Vendor behaviour is quoted from saved pages (2026-10-11), not tested on your setup.

Quotedn = 1 constructed example Constructed: no recorded run used a prompt like this one, so this page cannot say how an agent responds to it. The fields follow the vendor's saved pages: “Include test cases, paste screenshots, or define expected output in your prompt.” and “Have Claude show evidence rather than asserting success: the test output, the command it ran and what it returned, or a screenshot of the result.” No new run was made for this page. The file is built in your browser; nothing is sent anywhere.

Where to go next

• Class 98: AI Bug Fix teaches the bug-fix loop with the same 75 recorded runs, and has an interactive replay of one of them.
• Class 71: Evals and Testing AI Output takes testing past one function.
• Class 3: Prompting as Specification treats the prompt as a spec, the other half of this class.
• Class 95: Claude Opus 5.5 and Class 90: Claude Fable 5.1 first described the 2^-1 miss, where the spec is silent.
• Class 96: Claude Sonnet 5.5, measured against Opus 5.5 has the coding battery by model. This class does not compare models.
• Class 100: Claude Code Headless runs Claude Code from scripts, where checks like these run with nobody watching.
• Class 74: Deployment and CI/CD for AI-Built Code covers the pipeline that ships the code your tests guard.

One more class is planned, with no date and no promise that it will appear: What Your Coding Agent Is Allowed to Run.

What we did not verify

This is the page's own list of gaps. Everything here comes from runs recorded earlier, their saved grade and score files, and saved vendor pages. I made no new run, so each item is a limit on those records or on what was tested.

1. Whether writing tests first raises pass rates. The two recorded families are not a controlled comparison: they differ in task, tools and prompt shape. No recorded run was given the locked test file or its vector list in its prompt. A run with them would be a new run, and none was made.
2. Anything larger. The record covers 8 small prompts and 5 tiny repositories with one planted bug each. Nothing here was tried on a larger code base.
3. Other setups. 3 models, one Windows PC, Claude Code 2.1.284. The single-turn runs had no tools; the bug-fix runs had Read, Edit and Bash under one fixed allow-list, with nobody to answer a permission prompt. Other models, versions, settings and machines were not run.
4. Small samples. 11 recorded runs per bench prompt and 75 bug-fix runs. The page claims no confidence interval for any count.
5. The 6 refused spot-checks. What their extra cases would have shown is unknown: they did not run.
6. The 19 cut-off re-run heads. Their full text was not opened for this page. That all 49 re-runs were green is Class 98's reading of the full records, cited and not re-derived.
7. Why the 16 runs that ended on their edit did not try a re-run. The record shows a refused call earlier in each run, not a reason.
8. Why each run chose what it chose for 2^-1, beyond what its own code and comments say.
9. Reasons read from code. Where the page gives a reason for an outcome (a raise line, an index pattern, a parse gap), it is READ: our reading of the saved text. No model-written code was run for this page.
10. The grader, run again. It was not; every outcome is read from its saved grade files.
11. The hidden suite. The page reports only whether each run passed it, from the saved score files.
12. The vendor's suggestions in practice. Whether a Stop hook, an exact CI allow-list or a CLAUDE.md line naming the test command would change any count on this page was not tested here.
13. The vendor pages after 2026-10-11, when they were saved. They may have changed since.
14. The drills' constructed cases are not recorded runs. Each drill says which of its cases are recorded.