Deployment & CI/CD for AI-Built Code

Module 01

Pipeline Anatomy: Source to Production

Here is the problem. AI-generated code carries 2.7x higher vulnerability density than human-written code, and 81% of enterprise technology leaders reported production failures linked to AI-generated code by mid-2026. You are shipping code you did not write line by line. The pipeline is how you verify it.

The six stages, in order:

1. Source — you commit to a feature branch. The commit triggers the pipeline.
2. Build — the project compiles or bundles. If it fails to build, everything stops.
3. Test — unit tests, integration tests, linting. Fast, deterministic, cheap.
4. Eval gate — LLM evaluation against your golden dataset. This is where Evals (Class #71) plugs in. If the eval score drops below your threshold, the merge is blocked.
5. Security scan — static analysis, secret detection, dependency audit. This is where Security (Class #72) plugs in. In 2025, 28.65 million new hardcoded secrets were added to public GitHub commits — a 34% year-over-year increase (GitGuardian State of Secrets Sprawl 2025).
6. Deploy — the code ships to production. Only after every prior gate passes.

The DORA metrics (DevOps Research and Assessment) define four indicators that separate high-performing teams from the rest: deployment frequency, lead time for changes, change failure rate (elite teams hold this below 15%), and failed deployment recovery time (elite teams recover in under one hour). A fifth metric, deployment rework rate, was added in 2025 to capture the hidden cost of fixing post-deployment bugs. For AI-built code, rework rate is the one to watch: AI can inflate throughput metrics while silently increasing the volume of code that needs to be patched after it ships.

Module 02

The DDS Deploy Discipline

I will teach the general pipeline in Modules 03 through 08. But I lead with how I actually do it, because the specifics are more useful than the theory.

The workflow

1. Build on staging. Every new section or template is first deployed to a staging Shopify theme (a separate theme ID, not the live MAIN). The live store is never touched during development.
2. Render-verify, cache-busted. Open the staging theme preview with ?preview_theme_id=STAGING_ID&cache=false appended. Check three viewports: 1440x900 (desktop), 1440x680 (laptop), 390x844 (mobile). If anything renders wrong, fix it on staging. Never promote a broken render.
3. Promote with length parity. The deploy script GETs each file from staging, PUTs it to MAIN, then re-GETs from MAIN and asserts the byte length matches staging. If lengths diverge, the script halts and reports the mismatch. This catches silent truncation, encoding issues, and partial writes.
4. Pre-change snapshot. Before the deploy script writes anything to MAIN, it saves a copy of every file it is about to overwrite. If the deployment causes problems, rollback.ps1 restores from the snapshot in a single command.

This discipline is not GitHub Actions. It is a PowerShell script calling the Shopify Admin REST API. But the principle is identical: source → staging → verify → promote → snapshot rollback. The staging theme is the preview environment. The length-parity assertion is the deployment gate. The snapshot is the rollback. Every deployment pipeline, regardless of stack, needs these four components.

Module 03

A Real GitHub Actions Workflow

This is a complete, valid GitHub Actions workflow for an AI-built project. It covers all six pipeline stages from Module 01.

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read
  id-token: write    # Required for OIDC

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      - run: npm test

  eval-gate:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx promptfoo eval --config evals/config.yml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - run: |
          SCORE=$(cat evals/output.json | jq '.results.score')
          if (( $(echo "$SCORE < 0.85" | bc -l) )); then
            echo "Eval score $SCORE below threshold 0.85"
            exit 1
          fi

  security-scan:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - run: npx semgrep scan --config auto --error

  deploy:
    needs: [eval-gate, security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci && npm run build
      - run: npm run deploy
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

What each job does

• build-and-test — installs dependencies, builds the project, and runs unit tests. If any step fails, the entire pipeline stops.
• eval-gate — runs Promptfoo against your golden dataset. If the aggregate eval score falls below 0.85, the job exits with code 1 and the merge is blocked. This is the Evals (Class #71) gate in action.
• security-scan — runs gitleaks to detect hardcoded secrets in the git history, then runs Semgrep for static analysis. This is the Security (Class #72) gate in action.
• deploy — only runs on pushes to main (not on pull requests) and only after both eval-gate and security-scan pass. Uses a protected environment with its own approval rules.

Module 04

Branch Protection & PR Gating

Required settings for the main branch

1. Require pull requests. No direct pushes to main. Every change goes through a PR so the pipeline runs.
2. Require status checks to pass. Select the specific jobs from your workflow: build-and-test, eval-gate, and security-scan. All must pass before the merge button activates.
3. Require branches to be up to date. The PR must be tested against the latest code in main. This prevents a passing PR from merging when main has changed underneath it.
4. Require conversation resolution. If a reviewer leaves a comment, it must be resolved before merging.

This is where your Git (Class #73) discipline intersects with deployment. A clean commit history, enforced through branch protection and PR gating, means every deployment is traceable to a specific PR, a specific set of changes, and a specific set of passing checks. When something breaks in production, you can trace it back to the exact merge that introduced it.

Module 05

Staging & Preview Environments

For web applications, services like Vercel, Netlify, and Cloudflare Pages create a unique preview URL for every pull request automatically. The reviewer clicks the link, sees the running application, and confirms it works.

For Shopify themes, the pattern is different. You deploy to a staging theme ID and preview it with a URL parameter:

https://your-store.myshopify.com/pages/your-page?preview_theme_id=STAGING_THEME_ID&cache=false

The &cache=false parameter is critical. Without it, you might be verifying a cached version of the page, not the version you just deployed. I have wasted time debugging “broken” pages that were actually fine — I was just looking at the cached old version.

Verification checklist (three viewports)

• 1440 × 900 — desktop monitor. Check layout, typography, and the 3-column grid.
• 1440 × 680 — laptop screen. Check that the hero fits above the fold and the sticky TOC works.
• 390 × 844 — mobile (iPhone 14 / 15 viewport). Check that the layout collapses cleanly, the TOC is accessible, and no horizontal overflow occurs.

Module 06

Staged Rollout: Canary, Blue-Green, Feature Flags

The 2026 industry consensus is to layer all three. Use blue-green for infrastructure stability (the “safe environment”). Use canary for traffic-based validation (the “gradual exposure engine”). Use feature flags for business-logic control (the “fine-grained toggle”). An estimated 80% of large organizations are expected to adopt platform engineering by end of 2026, and these platforms typically bake all three strategies into their deployment templates.

Module 07

Secrets Management

This is where the Security (Class #72) “the AI shipped its own API key” failure mode becomes a deployment concern. If your AI coding agent generates code that contains a hardcoded API key, and your pipeline does not scan for secrets, that key ships to production and potentially to a public repository.

Three levels of secrets management

1. Environment variables. The baseline. Store secrets as encrypted environment variables in your CI/CD platform (GitHub Actions secrets, for example). Reference them in your workflow YAML. They are never exposed in logs or code.
2. OIDC (OpenID Connect). The current standard. Instead of storing a long-lived cloud access key as a secret, your workflow requests a short-lived token from your cloud provider at runtime. The token expires in minutes. If it leaks, the window of exposure is minimal. GitHub Actions supports OIDC natively for AWS, Azure, and GCP. As of July 2026, GitHub uses immutable OIDC claims for new repositories to prevent claim spoofing.
3. Dedicated secrets manager. For production applications with many secrets, use a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager). These provide audit logging, automatic rotation, and fine-grained access control.

Module 08

Rollback Strategy & Post-Deploy Observability

Rollback patterns

• Container image tags. For containerized deployments, tag every image with the git SHA. Rolling back means redeploying the previous tag. The DDS Shopify equivalent: every file overwritten during a deploy is saved to a snapshot directory before the write.
• Database migrations. Every migration must be reversible. If your deployment includes a schema change that cannot be rolled back, you have created a one-way door. Flag those explicitly in your pre-deploy checklist.
• Traffic switching. If you use blue-green deployment, rollback is a traffic switch. If you use feature flags, rollback is a toggle. Both are sub-second operations.

Post-deploy monitoring

The deployment is not done when the script finishes. It is done when you have confirmed that production is healthy. Define guardrail metrics before you deploy:

• Error rate — if the error rate exceeds your baseline by more than a defined threshold (e.g., 2x), trigger an alert.
• P95 latency — if the 95th percentile response time exceeds your SLA, investigate.
• Business metrics — conversion rate, checkout completion, page load time. A deployment that passes all technical checks but drops conversion by 15% is still a failure.

Module 09

The Four Pillars: Correct, Safe, Traceable, Shippable

The deployment pipeline is the final consumer. It does not produce correctness, safety, or traceability. It requires them. If any pillar is missing, the pipeline either rejects the code or ships it with unverified risk. Build all four.

Module 10

Pre-Deploy Checklist & Paste-Ready Prompts

Pre-deploy checklist

1. All CI status checks pass (build, lint, test, eval gate, security scan).
2. No hardcoded secrets in the codebase (gitleaks reports zero findings).
3. Staging environment renders correctly at 1440x900, 1440x680, and 390x844.
4. Deploy script produces matching file lengths between staging and production.
5. A rollback snapshot exists and the rollback script has been tested.
6. Post-deploy monitoring dashboards are open and alert thresholds are configured.
7. Database migrations (if any) are reversible.
8. Feature flags for new features are set to “off” by default.

Paste-ready prompts

Write a GitHub Actions workflow for this project. It should:
1. Trigger on push to main and pull_request to main.
2. Run build, lint, and unit tests in the first job.
3. Run an eval gate (promptfoo eval) that blocks the merge if the score is below 0.85.
4. Run a security scan (gitleaks + semgrep) in parallel with the eval gate.
5. Deploy to production only on push to main, only after eval and security pass.
6. Use OIDC for cloud authentication instead of static secrets.
Output valid YAML. No placeholders.

Scan this entire repository for hardcoded secrets, API keys, tokens,
and credentials. Check: source files, config files, .env files,
MCP config files, docker-compose files, and CI/CD workflow files.
Report every finding with the file path, line number, and the type
of secret detected. If zero findings, confirm explicitly.

Write a rollback script for this deployment. Before any file is written
to production, the script should save a copy of the current production
version to a timestamped snapshot directory. The rollback command should
restore every file from the most recent snapshot. Include error handling
for partial restores. Output the script in the language this project uses.

After deployment completes, verify the production environment is healthy:
1. Fetch the deployed page/endpoint and confirm it returns HTTP 200.
2. Check that the response contains the expected content markers.
3. Measure response time and flag if it exceeds 3 seconds.
4. If any check fails, output a FAIL status with the specific failure.
5. If all pass, output PASS with the response time.

Audit the branch protection rules on this repository's main branch.
Verify: pull requests are required, status checks are required and
include build, eval, and security jobs, branches must be up to date,
and force pushes are disabled. Report any gaps as actionable items.

Frequently Asked Questions

FAQ