Diffy Logo
Sign inGet started

Visual Regression Testing in GitLab CI

If your merge requests get review apps, every one of them can be checked against production before it merges. With Diffy, a GitLab CI job hands the review app URL to Diffy, Diffy compares it with production page by page and at every breakpoint, and the result comes back to the merge request as a comment and to the commit as a status.

There is no test code to write and no screenshots to store in the repository. The job is one CLI command; the pages, breakpoints, masks and login steps live in your Diffy project.

How it works

  1. A merge request is opened or updated, and your pipeline deploys a review app for it.
  2. A job in a later stage runs diffy project:compare with the review app URL, the commit SHA and the merge request IID.
  3. Diffy posts a comment on the merge request — “Visual check started” with a link to the results — and sets a commit status named Diffy check “<your project>” to running.
  4. Diffy screenshots production and the review app on its own servers, at every page and breakpoint in the project, and compares them.
  5. When the comparison finishes, Diffy posts a second comment: Diffy visual check completed — “No visual changes found” or the number of pages with visual differences — with a link to the results. If AI Summary is enabled for the project, the comment includes the summary of critical and minor changes. The commit status becomes success when nothing changed, or failed with the number of changes that need review.
  6. A reviewer goes through the changes in Diffy and marks each one as expected or as a bug. Once every change is reviewed, Diffy updates the commit status to success: “All visual changes reviewed”, with the number of bugs marked, if any.

GitLab attaches the commit status to an existing pipeline for that commit as an external job, or creates a separate external pipeline if there isn’t a suitable one.

What you need

  • A Diffy project with your production URL, the pages you want tested and your breakpoints. See what visual regression testing covers if you’re choosing pages for the first time.
  • A GitLab access token that Diffy can use to post comments and statuses. It needs the api scope, and its user needs at least the Developer role in the project — GitLab requires that role to set commit statuses. A project access token keeps it limited to one project.
  • A Diffy API key. API access comes with every paid plan.
  • A review app per merge request that Diffy’s servers can reach — GitLab review apps, Tugboat, Pantheon Multidev or anything else that gives each merge request its own URL. No review apps? See comparing staging with production below.

Step 1: Connect the GitLab project in Diffy

In your Diffy project, open Settings → Notifications → GitLab and fill in:

  • GitLab project URL — the project’s address, for example https://gitlab.com/acme/website. Diffy uses its host and path to reach the GitLab API, which is why self-managed GitLab works too, as long as Diffy’s servers can reach it.
  • GitLab access token — the token from the previous section.

Step 2: Add the CI/CD variables

In GitLab, go to Settings → CI/CD → Variables and add:

  • DIFFY_API_KEY — your Diffy API key.
  • DIFFY_PROJECT_ID — the ID of the Diffy project, the number in the project’s URL, as in app.diffy.website/#/projects/12345.

Step 3: Add the job to .gitlab-ci.yml

Add a visual stage after the stage that deploys your review app, and a job that runs once the review app is up. Here is a complete example, with a placeholder deploy job standing in for yours:

stages:
  - deploy
  - visual

# Your existing review app job (shown for context).
deploy_review:
  stage: deploy
  script:
    - ./deploy-review-app.sh
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_COMMIT_REF_SLUG.review.example.com
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

diffy_visual_check:
  stage: visual
  image: alpine:3.20
  needs: ["deploy_review"]
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  variables:
    REVIEW_URL: https://$CI_COMMIT_REF_SLUG.review.example.com
  before_script:
    - apk add --no-cache php83 php83-phar php83-curl php83-openssl php83-json php83-iconv
  script:
    - wget -O /usr/local/bin/diffy https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
    - chmod a+x /usr/local/bin/diffy
    - diffy auth:login "$DIFFY_API_KEY"
    - >
      diffy project:compare "$DIFFY_PROJECT_ID" prod custom
      --env2Url="$REVIEW_URL"
      --commit-sha="$CI_COMMIT_SHA"
      --merge-request-iid="$CI_MERGE_REQUEST_IID"
      --name="MR !$CI_MERGE_REQUEST_IID"

Set REVIEW_URL to the same URL your review app job uses. The rule runs the job in merge request pipelines only, which is where GitLab provides CI_MERGE_REQUEST_IID.

What each part does

  • image: alpine:3.20 and apk add — the Diffy CLI is a PHP archive that needs PHP 8.2 or later. On Alpine 3.20 the php83 package installs PHP 8.3 as the php command; the other packages are the extensions listed in Diffy’s GitLab docs, which use the same job setup.
  • diffy auth:login — authenticates with your API key.
  • diffy project:compare <project> prod custom — compares two environments: prod is the production URL from your project settings, custom is the URL in --env2Url. Use stage or dev instead of prod to compare against those environments.
  • --commit-sha — the commit Diffy sets the status on.
  • --merge-request-iid — the merge request Diffy comments on. Comments need both this and --commit-sha; without the IID you get the commit status but no comments.
  • --name — the name of the comparison in Diffy, so you can find it by merge request later.
  • For review apps behind HTTP basic auth, add --env2User and --env2Pass.

Without --wait, the command returns as soon as Diffy has created the comparison, so the job finishes in seconds while the Diffy status stays running. Add --wait only if a later step in the same job needs the result. Finding visual changes doesn’t make the command fail, with or without --wait — the result is reported through the comment and the commit status.

Reviewing the result

The completed comment links straight to the comparison in Diffy. Reviewers go through the changes page by page and breakpoint by breakpoint, with production and the review app side by side, and mark each change as expected or as a bug. When the same change shows up on several pages, it’s grouped so it can be approved once.

Every review action updates the commit status. When all changes are reviewed it turns success — and marking a change as a bug still counts as reviewed. The idea is that the author fixes the bug and pushes, which runs the pipeline and a new comparison.

No review apps?

You can still catch visual regressions before production. Compare staging against production whenever you deploy to staging, for example in a job after your staging deploy:

diffy_staging_check:
  stage: visual
  image: alpine:3.20
  rules:
    - if: '$CI_COMMIT_BRANCH == "staging"'
  before_script:
    - apk add --no-cache php83 php83-phar php83-curl php83-openssl php83-json php83-iconv
  script:
    - wget -O /usr/local/bin/diffy https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
    - chmod a+x /usr/local/bin/diffy
    - diffy auth:login "$DIFFY_API_KEY"
    - diffy project:compare "$DIFFY_PROJECT_ID" prod stage --commit-sha="$CI_COMMIT_SHA"

Change staging to the name of your staging branch. There’s no merge request in a branch pipeline, so Diffy sets the commit status but doesn’t comment.

FAQ

Do I need to write test code to run visual regression tests in GitLab CI with Diffy? No. The pages, breakpoints, masks and login steps live in the Diffy project settings. The pipeline job only passes Diffy the review app URL, the commit and the merge request number, with one CLI command.

What does Diffy post back to GitLab? A comment on the merge request when the comparison starts and another when it completes, and a commit status named Diffy check "<your project>": running, then success if nothing changed or failed if changes need review. Once every change has been reviewed in Diffy, the status is updated to success.

Does it work with self-managed GitLab? Diffy sends comments and statuses to the GitLab host in the project URL you enter, so a self-managed instance works as long as its API is reachable from Diffy’s servers.

Why does the pipeline job finish before the Diffy status? The job only starts the comparison. Screenshots and the comparison run on Diffy’s servers, and the commit status stays running until they finish. Add --wait to the CLI command only if a later step in the same job needs the result.

Which Diffy plan do I need? The pipeline job uses the Diffy CLI, which authenticates with an API key, and API access comes with every paid plan. The free plan (500 screenshots a month) lets you set up the project and run comparisons from the Diffy UI.

Get started

Create a Diffy account, add your production URL and key pages, and run a first comparison from the UI. When you’re ready to automate it, add the job above. On GitHub instead? See visual regression testing in GitHub Actions, or browse all Diffy integrations.