Visual Regression Testing for AI Coding Agents

A coding agent can rewrite a stylesheet, swap a component or bump a dependency in a few seconds. The unit tests pass, the build is green and the diff looks sensible. Nobody has looked at the page, and at 375px the pricing table now overflows the screen.

That is the gap behind questions like “how do I stop Claude Code from shipping broken UI changes?” and “how do I run regression tests from Claude Code or Cursor without leaving the IDE?”. The agent changes things it cannot see. The fix is to give it a visual check it can run on its own, read the result of and report back on, the same way it already runs your test suite.

This post shows how to do that with visual regression testing in Diffy: with the Claude Code plugin, and from any other agent that can run shell commands.

What an agent needs from a visual check

A visual check is useful to an agent only if it has four things:

A stable renderer. The same page has to produce the same screenshot twice, or every run reports noise. Fonts, animations, cookie banners and lazy-loaded images all get in the way, which is why screenshot testing is harder than it looks.
A known-good reference. Production, staging, or a baseline taken before the change.
A result the agent can read. A number and a list of which pages and breakpoints changed, not an image it has to interpret.
Something a human can open. The agent can tell you that /pricing changed by 12% at 375px. You still want to see it before you merge.

Diffy covers all four from the command line. Screenshots are taken in Diffy’s rendering environment with your project’s settings (pages, breakpoints, masks, delays, cookies, login), the comparison runs on Diffy’s side, and the result comes back as a diff ID you can query.

Claude Code: use the plugin

If you use Claude Code, install the open-source Diffy plugin:

/plugin marketplace add DiffyWebsite/diffy-skills
/plugin install diffy@diffy

It gives Claude ten skills. Claude can take a baseline of your local dev server, compare against it after an edit, compare two deployed environments, and summarise the result as a table of pages and breakpoints sorted by how much changed. Local capture runs in Diffy’s screenshot-worker Docker container, so a screenshot from your laptop renders like one from Diffy’s servers. The skills never delete Diffy projects.

The full walkthrough, with every skill and both capture modes, is in Claude Code visual regression testing: Diffy plugin and skills.

Cursor, Codex and other agents: use the CLI

The plugin has only been tested in Claude Code. Every other agent that can run terminal commands, including Cursor’s agent and Codex, can call the Diffy CLI directly. It’s what the plugin uses under the hood.

Install and authenticate

The CLI is a PHP application (PHP 8.2 or newer) distributed as a single file:

wget -O /usr/local/bin/diffy https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
chmod a+x /usr/local/bin/diffy
diffy auth:login <API_KEY>

Create the API key in Diffy under your account’s keys. API access, and with it the CLI, comes with every paid plan.

The three commands an agent needs

Compare a reference environment with the URL the agent’s change is running on:

DIFF_ID=$(diffy project:compare <PROJECT_ID> prod custom \
  --env2Url="https://preview-123.example.com" --wait)

Diffy screenshots both sides on its servers and the command prints a diff ID. --wait makes the CLI poll Diffy before printing it, but on a large project it can return before the comparison has finished, so check the status before reading the result. If the preview is behind HTTP basic auth, add --env2User and --env2Pass.

diffy diff:get-status "$DIFF_ID"                # prints 1 once the comparison is complete
diffy diff:get-changes-percent "$DIFF_ID"       # one number: 0 means nothing changed
diffy diff:get-result "$DIFF_ID" --format=json  # full result, per page and breakpoint

Two details matter for agents. First, the CLI exits successfully whether or not anything changed, so a green exit code means the comparison was created, not that the page is unchanged. Second, a percentage read before the comparison is complete means nothing. The agent has to wait for the status to be complete, then read the percentage or the JSON.

The URL has to be reachable from Diffy

Diffy takes these screenshots on its own servers, so localhost won’t work. Point the comparison at a preview deployment, a staging environment or a tunnel to your machine. Diffy’s docs cover ngrok and local capture with DDEV, Lando, Docksal or a standalone Docker container.

Let the agent reach Diffy’s API

Both Cursor and Codex run commands in a sandbox that blocks network access by default, and the CLI needs to reach app.diffy.website.

Codex. In the default workspace-write sandbox, network access is off. With the default approval settings, Codex asks before running a command that needs it. To allow it without asking, set this in Codex’s config.toml:
```
[sandbox_workspace_write]
network_access = true
```
Cursor. Sandboxed commands can only reach domains on the allowlist: your sandbox.json plus Cursor’s defaults for common package managers. Add Diffy to .cursor/sandbox.json in the project:
```
{
  "networkPolicy": {
    "allow": ["app.diffy.website"]
  }
}
```

Check your agent’s own security settings before changing them. Both tools offer narrower options than the ones above.

Make it part of how the agent works

Running the check once is easy. Getting the agent to run it after every UI change, without being asked, takes a line in its instructions. Codex and Cursor both read an AGENTS.md file in the repository, and by default Claude Code reads AGENTS.md too when the project has no CLAUDE.md. Something like this works:

## Visual checks

After any change to CSS, templates or UI components, run a Diffy comparison
before saying the task is done:

1. Make sure the change is deployed to a preview URL.
2. Run: diffy project:compare <PROJECT_ID> prod custom --env2Url="<preview URL>" --wait
3. Run diffy diff:get-status <diff ID> until it prints 1.
4. Run: diffy diff:get-changes-percent <diff ID>
5. If the result is not 0, run diffy diff:get-result <diff ID> --format=json and
   list the pages and breakpoints that changed, largest first.
6. Report the diff ID. Do not change baselines or project settings.

Instructions are context, not enforcement: an agent can skip a step, and Claude Code’s own documentation says so about CLAUDE.md. So treat this as the fast feedback loop while the agent works, and put the gate somewhere the agent can’t skip.

Put the gate in CI

The same command runs in your pipeline with two extra options:

--commit-sha attaches the result to the commit as a GitHub check. See visual regression testing in GitHub Actions.
--merge-request-iid makes Diffy comment on the GitLab merge request and set a commit status. See visual regression testing in GitLab CI.

diffy diff:get-result <diff ID> --format=junit-xml returns the result as JUnit XML, a format many CI systems can show as test results.

With this setup the agent catches most visual problems while it is still working, and anything it missed shows up on the pull request where a person reviews it.

What to leave to people

An agent is good at running the comparison and summarising the numbers. Deciding whether a change is correct is a different job. A 12% change on /pricing might be the bug, or it might be the redesign you asked for. Let the agent report and link the diff, and have a person approve changes and update baselines.

The other job worth keeping with people is tuning the project. Most bad experiences with visual testing come from false positives: carousels, cookie banners and dynamic content that change on every load. Fix those once in the project settings, and every comparison the agent runs from then on gives a result you can trust.

Create a free Diffy account to set up a project, then move to a paid plan when you want your agent or pipeline to call the API.