Reading and Directing AI Code

Section 01

The Confidence Trap, Revisited

Class 01 named the confidence trap in one sentence: AI-generated code does not look uncertain. This class is the practical answer to it. Every reviewer of AI code carries a small lie in their head, which is "if the code looks clean, it probably is." That belief is the single biggest reason AI-generated code ships with vulnerabilities, and it is the belief this class is built to dismantle.

Consider the asymmetry. A human writes code under physical constraints: every line costs typing effort, and every choice is felt as a decision. An AI writes code without that friction. It produces a complete, well-named, sensibly-commented implementation for any request, including requests it does not have enough information to satisfy. The result looks like the output of someone who knew exactly what they were doing, even when the underlying logic quietly assumes something that will not always be true. The fluency is the trap.

That reframe changes the entire review experience. You are not playing detective against a malicious AI. You are auditing your own specification against the implementation that resulted. Every issue you find is not a model failure; it is feedback about where your specification was thin. The fix is rarely "tell the AI to do better." The fix is to extend the spec and regenerate, the same discipline Class 03 named.

Section 02

The Numbers (Why This Class Exists)

Industry data from 2025 to 2026 makes the case for deeper review of AI-generated code unambiguous. The numbers below are from named studies and named research organizations. They are not vibes; they are the empirical floor you are reviewing against.

Read this table as the architect would read it. Each number is not a reason to avoid AI coding; it is a reason to do the review work. The same productivity gains that produce 3 to 4 times more code can produce 10 times more security issues, or they can produce slightly more code than hand-coding with no extra issues. The variable is the review discipline. The numbers say what happens when teams adopt AI generation without adopting the review practice this class teaches.

Section 03

Slopsquatting and the Hallucinated Supply Chain

Of all the AI-specific failure modes, slopsquatting is the one new architects most often miss because it does not look like a security issue at the diff level. The import line reads cleanly. The package name is plausible. The code compiles. The vulnerability is that the package did not exist until an attacker registered it, anticipating exactly the hallucination the AI produces.

The term was coined by Seth Larson, the Python Software Foundation Developer-in-Residence, and the underlying research is unambiguous. AI models hallucinate package names that do not exist. Attackers monitor those hallucinations, register the hallucinated names on real package registries, and wait for developers to paste the AI-generated import statement into a real codebase. When the install command runs, the malicious package is downloaded and executed with the same permissions as the developer's machine.

The hallucination is not random

What makes slopsquatting exploitable at scale is that the hallucinations are systematic. Research found 43 percent of hallucinated package names are recommended consistently across ten separate queries asking similar questions, and 58 percent appear more than once. Attackers do not need to predict every model output; they need to register the names the model produces predictably. This is fundamentally different from typosquatting, which exploits human typos. Slopsquatting exploits machine output patterns.

What to scan for

Slopsquatting is not the only hallucination class to watch for. AI-generated code routinely invents API methods that do not exist (the call compiles only if the language is dynamic enough), file paths that do not exist (in projects the AI has not been given full visibility into), and configuration keys that match nothing in the actual config schema. The pattern is consistent: the model knows the shape of the thing, so it produces a plausible instance, and the plausible instance is wrong in a way that looks right.

The fourth pass of the four-pass review (Section 06) explicitly looks for hallucinated dependencies, hallucinated APIs, and hallucinated paths. Doing this pass takes about 90 seconds for a typical diff. Skipping it is the move that puts your team in the 1-in-5 breach statistic.

Section 04

The 5:1 Rule (The Price of Velocity)

Spend 5 minutes reviewing AI-generated code for every 1 minute the AI took to generate it. The rule appears across multiple industry sources because it captures a real shift: the time savings of AI generation must be repaid in review effort. A 10-minute AI generation deserves about 50 minutes of human attention before merging. A two-minute one-shot of a small function deserves about ten minutes of careful reading.

The 5:1 rule sounds extreme until you do the math the other way. AI generation is roughly 5 to 10 times faster than hand-coding for the kinds of work it does well. If review takes 5 times the generation time, you are still net 1 to 2 times faster than hand-coding, and the output is safer than unreviewed AI code. The architect's discipline is to convert raw speed into reliable speed by spending the difference on review.

How to budget the time

Block review time on the calendar after a generation block. If you spent the morning prompting and accepting code, spend the afternoon reading it. Most architects find that reviewing fresh is more accurate than reviewing immediately, because the immediate read carries the optimism of having just produced the code. A short delay (an hour, a meal, overnight) introduces the small adversarial gap that catches more issues.

For ongoing work, run reviews in pairs. Two architects each reviewing the other's AI-generated code outperforms the same two architects reviewing their own, by a measurable margin. The pair-review pattern was a hand-coded discipline that became optional as IDEs got better; with AI-generated code, it has become essential again because the author cannot reliably review their own AI output.

Section 05

The Eight Failure Clusters in AI-Generated Code

AI code fails in patterns, not at random. Eight clusters cover the vast majority of what goes wrong. Recognize them at the diff level and you catch problems in seconds instead of in production. Each cluster has a distinct review move.

Cluster 1: Authentication gaps (client-side guard, no server-side check)

The classic AI code pattern: a beautiful client-side validation that prevents the user from submitting bad input, with no server-side validation at all. The client guard makes the diff look secure; the server gap means anyone with the API endpoint can do whatever the form was supposed to prevent. Review move: for every client-side check, search the codebase for the corresponding server-side check. If it does not exist, the AI did not write it.

Cluster 2: Hardcoded credentials reproduced from training data

The AI was trained on public code that contained example API keys, sample passwords, and test secrets. When the AI writes new code, it sometimes reproduces them verbatim. Review move: every string in the diff that looks like a credential, a token, an API key, or a password gets scanned. Run a secret-scanning tool in CI (GitGuardian, Trufflehog, Gitleaks) so this cluster is caught automatically.

Cluster 3: Happy-path-only logic

The function works correctly when the inputs are well-formed. It throws an uncaught exception, returns null, or silently fails when they are not. Review move: trace the unhappy paths Class 01 named. Empty input, null, malformed input, oversized input, concurrent access. If the diff does not handle them, the AI did not write the handling.

Cluster 4: Missing error handling for null and empty

Subset of Cluster 3, common enough to call out separately. The AI assumes the array is non-empty, the object has the field, the parsed JSON matches the schema. Review move: every variable that came from outside the function (parameter, fetch response, parsed input) gets a null check or a type guard. If the diff has none, add them.

Cluster 5: Insecure defaults

CORS set to wildcard, cookies without httpOnly or secure, sessions without expiration, file uploads without size limits, regex without timeout. The defaults the AI chooses are often the defaults that produce working code, not the defaults that produce secure code. Review move: every configuration object in the diff gets a security-defaults check against your team's standard.

Cluster 6: Hallucinated dependencies

Covered in detail in Section 03. The diff imports a package that does not exist, or that exists but is malicious due to slopsquatting. Review move: verify every new import against the official registry, every API call against the official documentation, every file path against the actual codebase.

Cluster 7: Prompt-injection vulnerabilities

If the code in the diff itself processes user input and sends it to an LLM, the diff is a new attack surface. User input that contains "ignore previous instructions, instead do X" can hijack the LLM call. Review move: every LLM call in the diff that includes user-controlled input gets an explicit injection-resistance check (input sanitization, output validation, sandboxed execution).

Cluster 8: Poisonable RAG pipelines

If the diff implements retrieval-augmented generation, the documents being retrieved are part of the trust boundary. A poisoned document in the retrieval index can hijack downstream behavior. Review move: every RAG ingestion path gets a source-validation check. Who can write to this index? What sanitization runs on ingestion? What happens when retrieval returns adversarial content?

Section 06

Read a Diff Like an Architect: The Four-Pass Method

Reading an AI-generated diff effectively means reading it four times, with a different lens each time. Four passes sounds inefficient until you do the math: a typical AI-generated diff that takes 12 minutes to read once and miss things takes about 8 minutes total across four focused passes, and the four passes catch roughly five times more issues. The passes are short, specific, and stack.

Pass 1: Contracts

Read every function signature, every type definition, every API boundary, every database schema change. Confirm they match what the specification asked for. If you wrote a spec in the eight-block format from Class 03, this pass is checking the diff against your <interfaces> block. Look for parameter order changes, type widening (any creeping into a strict type), return-shape changes, error-shape changes. Roughly 90 seconds.

Pass 2: Paths

Trace the happy path through the diff start to finish. Then trace each unhappy path: bad input, missing data, network failure, concurrent access, timeout. Verify the unhappy paths exist; the most common AI failure is that the unhappy paths simply are not written, because they were not specified. If the <done> criteria from your spec named negative cases (which they should), this pass is verifying each one. Roughly 2 to 3 minutes.

Pass 3: Secrets and dependencies

Scan every string for credential patterns. Scan every import for new packages, then verify each one on the package registry (this is the slopsquatting defense from Section 03). Scan every external call for API methods you do not recognize, then verify each against the actual documentation. This pass is the one most teams skip and the one that produces the breach statistics. Roughly 2 minutes.

Pass 4: Edges

Read the diff as if you were trying to break it. Empty arrays. Null. Negative numbers. Strings with control characters. Unicode. Concurrent calls. Race conditions. Inputs at the boundary of the type's range. For UI work, this includes screen reader behavior, keyboard navigation, and reduced-motion mode. The Class 01 happy-path-versus-unhappy-path framing is the lens. Roughly 2 minutes.

Section 07

The Adversarial-Review Prompt (Paste-Ready)

After the four-pass human review, run the diff through a second AI session with an adversarial-review prompt. Industry data suggests this catches an additional 40 to 50 percent of issues beyond standard review alone. The mechanism is simple: the generating AI is committed to its choices and tends to defend them. A separate AI session has no investment and is free to find flaws. The architect's job is to read what the second AI finds and decide which findings are real.

This is the canonical DDS adversarial-review prompt. Copy it, paste your diff into the <code> block, and run it in a fresh AI session. It is intentionally short. Longer prompts add noise; this one earns every line.

<role>
You are a senior security-focused code reviewer. You did not write this code.
Your job is to find what is wrong with it. Be specific. Cite line numbers.
</role>

<context>
This diff was produced by an AI coding agent against the specification below.
The code is intended to ship to production. The reviewer (the human) will
read your findings and decide what to act on.
</context>

<specification>
[Paste the original spec the code was generated from. If you used the
eight-block template from Foundation 03, paste it verbatim.]
</specification>

<diff>
[Paste the unified diff or the full file content here.]
</diff>

<review_checklist>
For each item, either flag a specific issue with the line number, or
explicitly confirm "no issue found." Do not skip items silently.

1. Authentication and authorization: does every server-side action have a
   server-side check, not just a client-side guard?
2. Hardcoded credentials, tokens, keys, or secrets anywhere in the diff.
3. Happy-path-only logic: are unhappy paths (empty, null, malformed,
   timeout, concurrent) handled?
4. Insecure defaults: CORS, cookies, sessions, file uploads, regex.
5. Dependencies: every import verified against an official registry?
   Any package that looks plausible but you cannot confirm exists?
6. Hallucinated APIs: every external call verified against actual docs?
7. Prompt injection: any path where user-controlled input reaches an LLM?
8. Tests: are tests included? Do they exercise the unhappy paths?
9. Performance: any obvious quadratic loops, unbounded recursion,
   missing pagination, or N+1 database queries?
10. Accessibility: keyboard navigation, screen reader, reduced motion,
    color contrast (for UI changes).
</review_checklist>

<output_format>
Return a numbered list of findings. For each finding:
- Severity: critical, high, medium, low.
- Location: file path and line number(s).
- Issue: one sentence describing the problem.
- Fix: one sentence describing the change required.

End with a single-line verdict: "READY TO MERGE" or "DO NOT MERGE: [reason]."
Do not include praise. Do not summarize. Findings only.
</output_format>

Two warnings about using this prompt well. First, use a different model or at minimum a fresh session from the one that generated the code. Same-model same-session adversarial review is much less effective; the model carries forward its own choices. Second, treat the output as findings to evaluate, not commands to execute. Adversarial review produces some false positives. The architect's job is to read each finding and decide which are real.

Section 08

Static Analysis Is Table Stakes, Not the Goal

Before human review, before adversarial review, before any read happens, static analysis runs and the diff either passes or does not. This is non-negotiable. The architect's discipline is to refuse to begin review on a diff that has not passed CI. Doing so trains the AI workflow correctly: the model learns through your iteration that broken code does not advance.

The minimum stack for 2026, by category:

Wire all six into your CI pipeline. The diff cannot be merged until all six pass. This is the deterministic floor below which human review does not begin, which means human review can focus on what the tools cannot see (architecture, judgment, specification alignment) rather than catching what should not be a human's job.

Section 09

The 12-Item PR Checklist (Paste-Ready)

This is the DDS canonical PR checklist for AI-generated code. Drop the markdown into .github/PULL_REQUEST_TEMPLATE.md and every new PR on the repo gets the checklist by default. The author checks each box before requesting review; the reviewer confirms each before approving. The list is twelve items by design. Longer is theater; shorter misses the failure modes this class named.

## Summary

[One paragraph: what this PR does and why.]

## Specification

[Link to the spec or paste the eight-block specification here.]

## AI-generated?

- [ ] This PR contains AI-generated code (add the `ai-generated` label if so).

## Review Checklist

- [ ] Spec is linked or pasted above.
- [ ] Static analysis green: linter, type check, security scan, secrets scan,
      dependency scan, tests, all pass in CI.
- [ ] Tests added for new behavior, including at least one unhappy-path case.
- [ ] Every new import verified against the official package registry
      (no slopsquatting).
- [ ] Every external API call verified against actual documentation
      (no hallucinated methods).
- [ ] No hardcoded credentials, tokens, keys, or secrets anywhere in the diff.
- [ ] Every client-side check has a corresponding server-side check.
- [ ] Empty, null, malformed, and oversized inputs handled explicitly.
- [ ] Performance reviewed: no obvious quadratic loops, no unbounded recursion,
      no N+1 queries, pagination present where needed.
- [ ] Security defaults reviewed: CORS, cookies, sessions, file uploads.
- [ ] Accessibility reviewed for any UI change (keyboard, screen reader,
      reduced motion, contrast).
- [ ] Adversarial-review pass completed (see foot of this file).

## Adversarial Review Findings

[Paste the findings list from the adversarial-review prompt, with notes on
which were acted on and which were dismissed and why.]

Two practical notes on this template. First, the AI-generated label is the most important single line. It lets your team measure (and improve) the ratio of AI-authored vs human-authored PRs, the issue rate per kind, and the time-to-merge per kind. The label is the data infrastructure of an architect-led team. Second, the adversarial-review findings block at the end forces the reviewer to engage with the second-AI critique rather than skipping it. If the field is empty, the review is incomplete.

Section 10

Test-First with AI (TDD Reborn)

Test-driven development was a discipline that lost momentum during the 2010s because writing the test, then writing the code, then refactoring took longer than the productivity it returned. AI generation flips the math entirely. The architect writes the failing test, the AI implements until the test passes, and the architect reviews the implementation. The test is the executable specification. The AI does the typing.

The workflow is short:

1. Write the test first, by hand or with a separate AI session. The test must fail when run, and the failure message must clearly indicate why the desired behavior is missing.
2. Hand the test plus the specification to the implementing AI. Ask for an implementation that makes the test pass and nothing more.
3. Run the test. If it passes, run the four-pass review on the implementation. If it fails, read the AI's output (it usually explains what it tried) and either correct the test or correct the spec, never both at once.
4. Once green, run the adversarial-review prompt as a final safety check.
5. Merge.

The reason this works is that the test removes the most common failure mode of AI code: the implementation that looks correct but does the wrong thing. The test is the architect's acceptance criteria made executable. The AI cannot fake passing it.

Section 11

When to Refine and When to Reject (The Two-Strikes Rule)

Some AI output is one good follow-up prompt away from being right. Other AI output is fundamentally wrong, and refining it costs more than rejecting and starting over. The architect's judgment call between the two is one of the most important skills this class can install. The rule, refined over thousands of architect-hours of practice, is the two-strikes rule.

The two-strikes rule

If two consecutive attempts at the same task both miss the specification in different ways, the specification is the problem, not the AI. Stop attempting the task. Open the spec. Read it as a hostile reviewer. Find the missing constraint, the ambiguous goal, the absent example. Fix the spec. Then regenerate from the corrected spec, not from a follow-up to either of the failed attempts.

The mechanism is straightforward. The same AI failing at the same task in the same way twice in a row is a model issue (try a different model). The same AI failing at the same task in different ways twice is a specification issue (the AI was guessing both times, and guessed differently each time because the spec did not pin the answer). Conversational fixes that try to correct both attempts at once produce code that is the average of two wrong guesses, which is its own kind of wrong.

When to refine instead of reject

• The output matches the spec, but with one missed edge case. Refine: add the edge case to the spec, regenerate.
• The output is correct but uses an unconventional pattern for your codebase. Refine: add the pattern preference to the spec or CLAUDE.md, regenerate.
• The output works, but a small refactor would make it cleaner. Refine: ask for the refactor as a follow-up, accept that the next time you want this output you should put the cleaner pattern in your spec.

When to reject and rewrite the spec

• Two attempts miss in different ways (the two-strikes rule).
• The output introduces an architectural pattern that does not fit the codebase.
• The output shows hallucination (imports, APIs, or paths that do not exist).
• The output handles the happy path but completely ignores the unhappy paths the spec named.
• The output security-by-default fails one of the eight failure clusters in a way that suggests systemic misunderstanding, not a single missed line.

Section 12

The DDS Review Discipline (Five Habits)

Discipline is what you do every day. The five habits below are the operational version of this class. They take roughly 20 percent more time than skipping them. They produce roughly 80 percent fewer issues in production. The math is the architect's entire job.

Habit 1: The pre-flight

Before reading any AI-generated diff, confirm CI is green and the AI-generated label is applied. If either is missing, refuse to start reviewing. This trains the workflow correctly and removes the variability of "did I check that today."

Habit 2: The four passes, every time

Contracts, paths, secrets, edges. No skipping. No "this one looks clean." The four passes take 8 minutes total for a typical diff; they catch issues you would otherwise find in production at 100x the cost.

Habit 3: The adversarial pass

After your four passes, run the adversarial-review prompt against a different model. Read the findings. Document which you act on and which you dismiss. This is the second pair of eyes that the academic data and the industry data both say catches the issues the first pair misses.

Habit 4: Test-first for anything non-trivial

For changes beyond a few lines, write the failing test first. The AI implements until the test passes. The test is the contract. This habit alone catches the entire happy-path-only failure cluster before it can ship.

Habit 5: Spec updates on every issue found

When the review catches an issue, ask: what missing line in the spec let this through? Add the line. The next generation against this kind of work will produce the right thing on the first try. The spec gets better with use; the AI does not need to. Compounding the spec is the long-term productivity move.

Section 13

What You Now Have

You came into this class knowing how to specify what to build. You leave with the second half of the architect's discipline: how to read what comes back, find what is wrong, and direct the correction. Together, the two skills are the entire job. The model is the typist. The architect specifies, reads, judges, and decides.

You also leave with the empirical case. The 45 percent Veracode rate, the 40 percent Stanford finding, the 10x Apiiro multiplier, the 1-in-5 Aikido number, the 6.4 percent GitGuardian leakage rate. These are not abstractions. They are the floor your workflow is judged against. The five-habit discipline in Section 12 is the practical answer to all of them.

Two artifacts in particular are worth keeping at hand: the adversarial-review prompt in Section 07, and the 12-item PR checklist in Section 09. Both are paste-ready. Both produce immediate, measurable improvement the day you adopt them. Both are short on purpose. The architect's discipline is small, repeated, and consistent, not heroic.

The final Foundation class brings it all together. Foundation 05 teaches the loop: how the paradigm, the environment, the specification, and the review combine into the agentic build cycle that produces reliable software at the speed AI generation makes possible. Context engineering, the agentic build loop, knowing when to ship. The end of the Foundation track and the beginning of the architect's ongoing practice.

FAQ

Frequently Asked Questions

The questions newcomers ask most about reviewing and directing AI-generated code. Each answer matches this page's structured data exactly, so a person reading the page and an AI engine extracting the schema receive the same canonical response.