Turn a bug report into a failing Playwright test — automatically

Every bug fix ends with the same slightly awkward question: *how do we know it's actually fixed?* Usually the answer is "the person who reported it clicked around again and it seemed fine." That's a vibe, not a verification — and with AI agents writing more of our fixes, vibes are not enough. An agent will happily declare victory on a patch that compiles.

There's an old discipline for this: write a failing test first. Reproduce the bug as an assertion, watch it fail, fix the code, watch it pass. Almost nobody does it for reported bugs, because translating "the order button doesn't work" into a runnable test is twenty minutes of tedious locator archaeology.

So TraceBug does the translation automatically (disclosure: I build it). Every captured bug report embeds a generated Playwright spec that replays the session and asserts the captured failure is gone.

What the generated test contains

TraceBug already captured everything a test needs: the exact click/input sequence, the request that failed, the console error that fired. The generator turns that into a spec:

import { test, expect } from '@playwright/test';

// Captured with TraceBug — free, local-first bug reports
// Generated from bug report: "Place order" Action — API POST Returns 404
//
// This test REPRODUCES the captured bug — expect it to FAIL until the bug
// is fixed, then pass. Point BASE_URL at your running dev server.

test('replays the captured session and asserts the failure is gone', async ({ page }) => {
  // ...navigates to the captured route, replays each action in order,
  // then asserts:
  //  - the captured endpoint (POST /api/orders) no longer fails
  //  - the captured console error is no longer thrown
});

Three details in the generation matter more than they look:

Locator preference is stability-ordered. Each replayed action targets data-testid first, then id, then aria-label, then role + accessible name, and only falls back to the captured CSS selector last — so the test survives a refactor better than a recorded macro would.

The assertions target the captured failure, not a screenshot. The spec collects failed requests and asserts *the specific endpoint that broke* stops failing, and that the captured console error stops being thrown. It's asserting the bug, not the pixels.

Redacted input stays redacted. If the captured session involved sensitive typed values, they arrive in the test as TODO placeholders with a comment — the capture-time masking carries through to the artifact.

Why this is bigger with an AI agent in the loop

Hand an agent a bug report and a failing test together, and the debugging session gets a *definition of done*:

1. Agent reads the report over MCP (console, network, replay — the full-context story) 2. Agent runs the spec → red, reproducing the exact captured failure 3. Agent patches the code 4. Agent runs the spec → green

Step 4 is the difference between "the agent says it's fixed" and "the fix is proven against the failure that was actually captured." It also leaves something behind: commit the spec, and this bug has a permanent regression guard in CI. The bug report stops being a disposable artifact and becomes part of the test suite.

Getting one

Three ways, all free: the Download failing test (.spec.ts) item in the ticket's More menu, embedded inside every exported .html replay, or via the MCP tool get_playwright_test if your agent wants to fetch it itself. Run it like any spec:

npx playwright test bug-place-order-404.spec.ts

The fastest way to see the loop end-to-end: capture a bug on the live sandbox — it has intentional bugs for exactly this — export the test, and watch it fail for the right reason. Then fix the sandbox's bug if you like. The test will tell you when you're done, which is more than most bug reports ever did.