Visual Regression Testing in GitHub Actions

GitHub Actions is where most teams already build, test and deploy. With Diffy it is also where visual review starts: every pull request’s preview environment is compared with production, page by page and at every breakpoint, and the result lands on the pull request as a check.

There is no test code to write and no screenshots to commit. The workflow is one CLI command; the pages, breakpoints, masks and login steps live in your Diffy project.

How it works

A pull request is opened or updated, and your host deploys a preview environment for it.
A GitHub Actions workflow picks up the preview URL and runs diffy project:compare with the pull request’s commit.
Diffy screenshots production and the preview on its own servers, at every page and breakpoint in the project, and compares them.
A check named Diffy check “<your project>” appears on the pull request. It completes as success with “No visual changes found”, or as action required with the number of changes that need review. Details opens the comparison in Diffy.
A reviewer goes through the changes in Diffy and marks each one as expected or as a bug. When every change is reviewed, Diffy updates the check to success: “All visual changes reviewed”, with the number of bugs marked, if any.

Optionally, Diffy also posts a comment on the pull request when the comparison starts — pages, breakpoints, estimated time — and updates it with the result when it finishes. If AI Summary is enabled for the project, the comment includes the summary of critical and minor changes.

What you need

A Diffy project with your production URL, the pages you want tested and your breakpoints. See what visual regression testing covers if you’re choosing pages for the first time.
The Diffy GitHub app installed on the repository (github.com/apps/diffy-testing).
An API key from your Diffy account. API access comes with every paid plan.
A preview environment per pull request that Diffy’s servers can reach — Vercel, Upsun, Pantheon Multidev, Tugboat, Cloudflare, or your own setup. If you don’t have preview environments, see comparing staging with production below.

Step 1: Connect the repository in Diffy

In your Diffy project, open Settings → Notifications → GitHub and enter the repository in GitHub project URL (for example https://github.com/acme/website). If you want the pull request comment as well as the check, tick “In addition to the check post a comment to the pull request with diff progress and results”.

Step 2: Add the secret and the variable

In the GitHub repository, go to Settings → Secrets and variables → Actions and add:

a secret DIFFY_API_KEY with your Diffy API key;
a variable DIFFY_PROJECT_ID with the ID of the Diffy project — the number in the project’s URL, as in app.diffy.website/#/projects/12345.

Step 3: Add the workflow

Which workflow you need depends on how your host tells GitHub that a preview is ready.

If your host reports GitHub deployments (Vercel, for example)

Some hosts create a GitHub deployment for each preview and report its status, which fires the deployment_status event with the preview URL in environment_url. Vercel does this by default; for other hosts, check their GitHub integration docs. Create .github/workflows/diffy.yml:

name: Visual regression tests

on:
  deployment_status:

jobs:
  diffy:
    # Run once per successful preview deployment, never for production.
    if: >-
      github.event.deployment_status.state == 'success' &&
      github.event.deployment.environment != 'Production' &&
      github.event.deployment.environment != 'production'
    runs-on: ubuntu-latest
    steps:
      - name: Compare the preview with production in Diffy
        env:
          DIFFY_API_KEY: ${{ secrets.DIFFY_API_KEY }}
          DIFFY_PROJECT_ID: ${{ vars.DIFFY_PROJECT_ID }}
          PREVIEW_URL: ${{ github.event.deployment_status.environment_url }}
          COMMIT_SHA: ${{ github.event.deployment.sha }}
          REF: ${{ github.event.deployment.ref }}
        run: |
          wget -q -O /usr/local/bin/diffy https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
          chmod a+x /usr/local/bin/diffy
          diffy auth:login "$DIFFY_API_KEY"
          diffy project:compare "$DIFFY_PROJECT_ID" prod custom \
            --env2Url="$PREVIEW_URL" \
            --commit-sha="$COMMIT_SHA" \
            --name="Preview $REF"

Check what your host calls its production environment and adjust the if condition to match; the goal is to run on previews only.

If your preview URL is predictable

Some setups give every pull request a URL built from its number or branch. Then you can run on pull_request directly:

name: Visual regression tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  diffy:
    runs-on: ubuntu-latest
    steps:
      - name: Compare the PR environment with production in Diffy
        env:
          DIFFY_API_KEY: ${{ secrets.DIFFY_API_KEY }}
          DIFFY_PROJECT_ID: ${{ vars.DIFFY_PROJECT_ID }}
          PREVIEW_URL: https://pr-${{ github.event.pull_request.number }}.preview.example.com
          COMMIT_SHA: ${{ github.event.pull_request.head.sha }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
        run: |
          wget -q -O /usr/local/bin/diffy https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
          chmod a+x /usr/local/bin/diffy
          diffy auth:login "$DIFFY_API_KEY"
          diffy project:compare "$DIFFY_PROJECT_ID" prod custom \
            --env2Url="$PREVIEW_URL" \
            --commit-sha="$COMMIT_SHA" \
            --name="PR #$PR_NUMBER"

Use github.event.pull_request.head.sha, not github.sha: on pull_request events github.sha is the temporary merge commit, and the check has to attach to the commit the pull request shows. This workflow assumes the preview is already deployed when it runs; if your host takes a few minutes to build, add a step that waits for the URL to respond first.

Host-specific guides

Upsun (formerly Platform.sh): Visual regression testing for Upsun pull requests — a complete workflow that waits for Upsun’s preview environment.
Pantheon: GitHub, Pantheon Multidev and GitHub Actions — creates a Multidev per pull request and compares it. See also our post on Pantheon deployments from GitHub.
Cloudflare: Astro, GitHub and Cloudflare — triggers after a successful Cloudflare deployment check.
Tugboat: GitHub and Tugboat.

What each part of the command does

wget … diffy.phar downloads the latest Diffy CLI. It is a PHP archive; the ubuntu-latest runner already has PHP and wget.
diffy auth:login authenticates with your API key.
diffy project:compare <project> prod custom compares two environments: prod is the production URL from your project settings, custom is the URL passed in --env2Url. Use stage or dev instead of prod to compare against those environments.
--commit-sha tells Diffy which commit to attach the check to, and lets it find the pull request for the comment.
--name names the comparison in Diffy, so you can find it by pull request later.
For previews behind HTTP basic auth, add --env2User and --env2Pass.

Without --wait, the command returns as soon as Diffy has created the comparison, so the workflow run finishes in seconds while the Diffy check stays in progress. Add --wait only if a later step in the same job needs the result. Finding visual changes doesn’t make the command fail, with or without --wait — the pass/fail signal is the Diffy check, not the workflow run.

Make visual review a merge requirement

To stop pull requests from merging before someone has looked at the visual changes, add the Diffy check as a required status check in the branch protection rules (or ruleset) for your main branch. GitHub only allows a merge when required checks are successful, skipped or neutral, so a comparison with unreviewed changes — action required — keeps the pull request blocked until the changes are reviewed in Diffy. GitHub lists checks that have run in the repository recently, so set this up after the first pull request has been through the workflow.

Note that marking a change as a bug still counts as reviewed: the check turns green with the number of bugs marked. The idea is that the author fixes the bug and pushes, which triggers a new comparison.

No preview environments?

You can still use GitHub Actions to catch visual regressions before they reach production. Run a comparison of staging against production whenever you deploy to staging — for example on push to your staging branch, after the deploy step:

diffy project:compare "$DIFFY_PROJECT_ID" prod stage --commit-sha="$GITHUB_SHA"

On push events $GITHUB_SHA is the pushed commit, so the Diffy check appears on it in the repository’s commit history.

Diffy vs Playwright screenshots in GitHub Actions

You can also run Playwright’s toHaveScreenshot() inside the workflow itself — we wrote a guide to Playwright visual regression testing that covers the CI setup. The two approaches answer different questions. Playwright compares the pull request against baseline images committed to the repository, which have to be generated on the same operating system the runner uses. Diffy compares the pull request’s environment against live production, with no baselines in git, and gives reviewers — including people who don’t read code — a UI to approve changes page by page. Diffy vs Playwright goes through the trade-offs in detail.

FAQ

Do I need to write test code to run visual regression tests in GitHub Actions with Diffy? No. The pages, breakpoints, masks and login steps live in the Diffy project settings. The workflow only passes Diffy the preview URL and the commit, with one CLI command.

Can the Diffy check block a pull request from merging? Yes. Add it as a required status check in your branch protection rules. A comparison with unreviewed changes completes as action required, which keeps the pull request blocked until someone reviews the changes in Diffy; once every change is reviewed, Diffy updates the check to success.

Why does the workflow finish before the Diffy check? The workflow only starts the comparison. Screenshots and the comparison run on Diffy’s servers, and the check on the pull request stays in progress until they finish. Add --wait to the CLI command only if a later step in the same job needs the result.

Does the preview environment have to be public? It has to be reachable from Diffy’s servers. For previews behind HTTP basic auth, pass the credentials with --env2User and --env2Pass. Diffy projects can also send custom HTTP headers and cookies with every screenshot.

Which Diffy plan do I need? The GitHub Actions workflow uses the Diffy CLI, which authenticates with an API key, and API access comes with every paid plan. The free plan (500 screenshots a month) lets you set up the project and run comparisons from the Diffy UI.

Get started

Create a Diffy account, add your production URL and key pages, and run a first comparison from the UI. When you’re ready to automate it, add the workflow above. Using GitLab? See visual regression testing in GitLab CI, or browse all Diffy integrations.