Pantheon Quicksilver Quick Start

Setup summary

Pantheon has the ability to automate tasks that run after deployment. In this tutorial, we demonstrate how to set up running visual regression testing after deployment to the Test environment.

Summary of setup:

Create an authentication key and project in Diffy
Set up key and script in your Pantheon’s git repo
Run deployment and ensure visual tests

Set up Diffy keys and project

First of all, you will need to create an account with Diffy.

Create an API key with Diffy (under the main navigation https://app.diffy.website/#/keys)

Create API Key

After creating a key let's create a project.

Create a Project in Diffy by providing Live and Test URLs

Diffy Create Project

If you are using a Trial account that is limited to 10 pages you might see a warning:

“We scanned the site and found 29 internal pages, but you have a limit of 10. So not all pages listed below.” but that is totally OK. We can start by covering only 10 pages. We would be able to change that easily in the future.

Next, notice the ID of the project.

Project ID from URL

After creating a project Diffy will start comparing your Prod environment with Staging. As we haven’t done any deployments yet it should not find any differences. But in my case screenshots were taken from the Staging environment too fast and not all images were loaded. To adjust that you can set a delay in the project’s settings.

Project settings Delay 10 seconds

After setting up a delay I rerun the comparison just to ensure no changes are found as sites are identical at this point.

No changes after setting up delay

Now we can switch to the terminal as we will need to provide credentials to quicksilver so it would be able to authenticate with Diffy to trigger our testing.

For that, we will need to create a secrets.js file and upload it to our site using sftp. So anywhere on your computer run (token is the Diffy's API key we just generated):

echo '{"token": "a9f8a8086c47a52fd000062756f40bc4", "project_id" : "359"}' > secrets.json

And then we need to connect by sftp to Pantheon hosting and upload this file to the live environment.

To find sftp credentials go to Pantheon’s dashboard, switch to Live tab, and open Connection Info.

sftp -o Port=2222 live.56fe4000-47bc-4273-ab23-3456346dccbc@appserver.live.56fe4929-47bc-4273-ab23-3456346dccbc.drush.in

sftp> cd files
sftp> cd private
sftp> put secrets.json
sftp> quit

So now we have all the configurations on the live site. Please also sync the files between your environments so secret file is populated to other environments as well.

The last piece would be to set up Quicksilver.

For that create a folder `/private/scripts` in your web docroot which may be in `web` so that would be `/web/private/scripts`. See Pantheon's documentation for more details. Then copy a file diffyVisualregression.php from https://github.com/pantheon-systems/quicksilver-examples/tree/master/diffy_visualregression.

The last edit is to let Quicksilver know to run that script on deployment. To edit your pantheon.yml file from the code repository’s root folder and add a step:

workflows:
  deploy:
    after:
      - type: webphp
        description: Do a visual regression test with Diffy.website
        script: private/scripts/diffyVisualregression.php

Now we can commit and push the code. It will automatically get deployed to the Dev environment.

Now is the moment of truth -- let's run deployment on the Test (staging) environment. For that go to your dashboard and run deployment.

Deploy to test environment Pantheon

In Diffy you should see screenshots and diffs started.

Diffy jobs started

After all the jobs are completed -- you will receive a notification and would be able to check the differences.

Congratulations!

Next steps

As the next steps, you could set up different notifications. For example to visually compare test and development environments on deployment to development. Also, you can set up a slack notification once visual testing is completed. Let me know if you come up with any other ideas in the comments.

Troubleshooting

If something doesn’t work for you. The best way to debug the deployment script is via Terminus (replace diffy with your sitename)

./terminus workflow:watch diffy

This will show you the execution log of the deployment script diffyVisualregression.php. So you could add different “print_r()” to debug what is wrong.

Alternatively you can debug script execution via terminus

./terminus remote:drush site.environment -- php:script diffyVisualregression.php --script-path=/code/web/private/scripts

Youtube recorded demo

Welcome to watch a video demo.