Diffy vs BackstopJS | Diffy visual testing

The common ground

Both Diffy and BackstopJS are designed to open many pages of your website, take screenshots, and compare them. We were asked multiple times if Diffy uses any parts of BackstopJS behind the scenes, and the answer is no. Diffy was built independently from scratch but uses Puppeteer as well as BackstopJS.

Configuration differences

BackstopJS uses a YAML-based file for configuration. While Diffy is also moving toward this solution, it has not yet been implemented for all the options.

The main difference in configuration is that BackstopJS has a configuration per page. In contrast, Diffy has per project (as an example all pages will get the same delay, masking, and other options).

Environments

BackstopJS supports reference and test environments. The actual URL can be changed in the configuration file. Diffy stores three environments: production, staging, and development, and allows the use of custom environments, too. Interestingly you can map completely unrelated URLs in BackstopJS. I.e. your reference URL could be https://example.com/about-us, and test page could be https://staging.example.com/new-about-us-page. In Diffy, we use base URLs. In the example above, the base URLs would be https://example.com and https://staging.example.com) but URI should stay the same, i.e., either /about-us or /new-about-us-page. This is how Diffy matches pages across different environments.

Both systems support basic authentication for environments.

Speed

BackstopJS is fast. We tested dozens of pages in multiple breakpoints, and generating all the screenshots took under 2-3 minutes. At Diffy, we usually estimate that it will take around 30-60 seconds per screenshot. Remember that you can set the number of concurrent workers (browsers) that will be used in Diffy.

While BackstopJS is fast, it is not always a good idea as you might overload servers or get blocked by CDNs.

Screenshots accuracy

Diffy and BackstopJS take completely different approaches to rendering large pages that have more than one screen. BackstopJS takes screenshots of each screen and stitches them together (github issue), while Diffy opens the page in a very tall browser to grab all the content. These two approaches lead to a different set of problems.

With BackstopJS, if you have a floating element (an example could be a sticky header) you could find it in a very unexpected location. Here is an example of a screenshot from https://www.adaptive.co.uk/.

Diffy, on the other hand, has to deal with elements of relative height (imagine 100vh height in the hero area). To prevent these elements from stretching, Diffy adjusts the CSS behind the scenes to override these styles.

Scroll and delay

Both tools allow you to set a delay before taking the screenshot. BackstopJS also has an interesting feature that allows users to scroll up to a certain element, while Diffy has a checkbox that scrolls the page to the bottom.

Mask and remove

Both Diffy and BackstopJS can remove elements. BackstopJS sets CSS display:none; and Diffy removes elements from DOM. BackstopJS can hide elements (set CSS visibility:hidden). Diffy has masking functionality – this will draw a green rectangle over the desired element. We find it useful so users can see where masked elements are. By default, Diffy masks elements like embedded videos and maps.

Custom Javascript and CSS

BackstopJS has the ability to run javascript files at different stages of execution:

"onReadyScript": "filename.js" // Runs after onReady event on all scenarios -- use for simulating interactions (.js suffix is optional)
"onBeforeScript": "filename.js" // Runs before each scenario -- use for setting cookies or other env state (.js suffix is optional)

These are the files where you can write code to change the User Agent, inject CSS, or perform any other actions on the site.

Diffy on the other hand can also execute javascript before we take the screenshot. The setting of User-Agent is part of UI configuration.

Cookies

Both BackstopJS and Diffy allow setting custom cookies to the browser.

CI setup

BackstopJS comes with a Docker container image that can be used in your CI. The only tricky part is storing the report elsewhere so it can be accessed after the test is completed.

Diffy comes with a CLI tool for triggering comparison jobs. Since reports are in the cloud, you just need to post the link to the report. Diffy also has a nice Github check integration.

Screenshots from the local environment

BackstopJS runs locally. Diffy also has a local worker (a Docker container) that takes screenshots and uploads them to the cloud for comparison.

Night Mode, Retina

You can enforce Night mode and Retina images with single checkboxes in Diffy. You can accomplish similar in BackstopJS with custom javascript. See https://github.com/garris/BackstopJS/issues/1229

Unique features

Actions

BackstopJS has keyPressSelectors, hoverSelectors, and clickSelectors, which allow you to enter keys, hover over certain elements, and click certain elements. To perform these actions in Diffy, you would need to write Javascript.

Comparison of images

BackstopJS uses a pixel-perfect algorithm to compare screenshots, while Diffy uses both a pixel-perfect algorithm and a custom solution. The biggest difference is that Diffy’s algorithm recognizes vertical shifts. So, if you have a change to the header and it shifts the whole page, Diffy will render the changes only in the header, while BackstopJS will light up the whole page.

Authenticated user

Diffy allows you to log in to the site prior to taking a screenshot. There are presets for Drupal and WordPress. Behind the scenes, Diffy opens the Login page, populates the credentials and submits it. Then, it opens the desired page and takes a screenshot.

BackstopJS has some articles on how this can be accomplished with a combination of custom javascript and saving cookies data. Example article.

Mock Content

You can specify a list of selectors and the type of mock. Diffy will replace the element’s content with “lorem ipsum” text. This functionality is widely used when you have dynamic content (new articles are published regularly) and cannot always sync the content between environments.

UI

Diffy has a way more versatile UI for reviewing changes:

Thumbnails preview
Approval per change on the page
Navigation shortcuts and more

Monitoring

Diffy allows you to set up monitoring (daily or weekly) to compare screenshots from different environments or from the same environment (today vs. yesterday).

Some advantages of Diffy as a cloud service

Diffy stores all screenshots/comparisons in the cloud so you can see the history (we retain screenshots for 6 months by default).

Diffy allows you to share results with your teammates easily (shareable links are available even for users without accounts).

BackstopJS is open source

It is a truly great project that got a lot of traction and adoption. In the meantime, it looks like the most active development happened between 2015 and 2022, and now the project gets less attention. Another thing to mention is that BasktopJS is maintained mainly by one person.