Diffy vs SauceLabs

SauceLabs is a very established company in the browser testing industry. After acquiring Screener, SauceLabs started offering visual regression testing as part of its product portfolio.

The main technical difference between SauceLabs and Diffy is that SauceLabs requires a testing framework, where you insert “capture screenshot” statements to build a set of screenshots for comparison.

We built a Diffy-like testing scenario in Cypress that visits multiple pages and takes screenshots to compare both tools:

const baseUrl = 'https://www.adaptive.co.uk';
const pages = ['/', '/contact-us'];
const breakpoints = [640, 1200];

describe('Sauce Visual Demo', () => {
  it('should be able to open the pages and take screenshots', () => {
    pages.forEach((page) => {
      breakpoints.forEach((breakpoint) => {
        let fullUrl = baseUrl + page;
        cy.on('uncaught:exception', (err, runnable) => {
          return false;
        });
        cy.visit(fullUrl);
        cy.viewport(breakpoint, 1000);
        cy.get('body').then(($body) => {
          if ($body.find('#CybotCookiebotDialogBodyButtonDecline').length) {
            cy.get('#CybotCookiebotDialogBodyButtonDecline').click();
          }
        });
        cy.sauceVisualCheck(fullUrl + ' ' + page, { captureDom: true });
      });
    });
  });
});

Browsers

SauceLabs is designed to support multiple browsers / OS, while Diffy uses only the latest Chrome.

Screenshots quality

We took screenshots from a website with a floating header and saw that the header is duplicated multiple times. So SauceLabs does take screenshots from the entire page, but it works like taking screenshots with a set viewport and then stitching them together. In contrast, Diffy opens a page in a tall browser window to grab the whole page content.

Tools to modify screenshots

If you use SauceLabs, you need to implement your stabilization scripts. Here’s an example for hiding a cookies popup:

// Modify the page before taking screenshots.
cy.get('body').then(($body) => {
  if ($body.find('#CybotCookiebotDialogBodyButtonDecline').length) {
    cy.get('#CybotCookiebotDialogBodyButtonDecline').click();
  }
});

In the same way, you would need to add a delay, scroll the page, and perform other actions as required.

On Diffy’s side, we allow you to apply these modifications in the UI, but you can also inject custom JavaScript.

Ignore elements

SauceLabs has an interesting feature that ignores certain regions on the page. While Diffy has a way to mask elements (draw a green rectangle on top of the element so its content is invisible for the screenshots), in SauceLabs, you can specify the coordinates of the area that should be skipped during comparison.

Baseline

SauceLabs allows you to compare any set of screenshots with the Baselines. There are some rules about how baseline screenshots are selected during comparison, with overrides and different browsers / OS combinations.

At Diffy, you can compare any set of screenshots with any other. You can take them from different environments. There is also a Baseline set, but it is more like a “special” set of screenshots rather than the only one you can compare against.

Review process

Diffy allows you to approve/mark individual changes as bugs. In SauceLabs, you approve/reject the whole screenshot. SauceLabs categorizes the changes by visual, style, dimension, and others. It also analyzes the HTML of the pages.

Conclusion

SauceLabs is very framework-dependent, so you must implement all the manipulations on the web pages yourself to implement things like delay, masking, mocking content, and other things that Diffy offers out of the box.