Regression Testing: Your Step-by-Step Action Plan

Oftentimes you have to maintain or modify a website you did not build. This means understanding someone else’s work and building test cases for a website you know very little about.

Moreover, some of these sites may be massive, with hundreds of changes done to them on a daily basis. You have to put the pieces together, re-compile its history, and, on top of everything, make sure it functions properly.

It’s quite the tall order. So today we’re talking about regression testing:

• How do you create a testing plan for a website you didn’t build yourself? 
• What should you include and what should you exclude? 
• What questions should you ask your client?
• What should your priorities be?

How to Build a Testing Plan for a Website -- a Step by Step Guide

Let’s say you work for a web development agency. Your sales team is proud to hand you over a new project. It’s your job to develop a testing plan for all the changes your development team will be implementing. 

So far so good. 

But, as it often happens, the first problems start to pop up before you’ve even had a chance to enjoy signing a new contract.

Where’s the Documentation? What Happens When There’s No Official Documentation?

Typically, this is your first question. You need the documentation to set things in motion.

But the sales team doesn’t always have the answer you are looking for. After all, their job is to close the deal, not to get lost in the technical nitty-gritty of it all.

What can you do?

Start by looking at the offer that your sales colleagues sent to the client. The contract is also useful here.

Aside from the legalese, you will often find a few lines about the functionalities you are supposed to add. Not enough information there?

No worries; this is very common. Here’s what to do next:

Set a Few Discovery Interviews with the Client

However detailed the contract is, your best source of information remains the client. So set up a few interviews with them and make sure that someone on their technical team is present.

Do keep in mind that the client is more likely to talk about functionalities in general. This is, of course, very important. Make all the necessary notes -- these are the things you will certainly have to cover. But also be sure to stir them back to the issue at hand -- testing --, if they get lost in details.

Before you start the calls, write down a few questions you plan to ask them. They may be accustomed to only testing the functionalities that were changed after the initial development. But when we are talking about regressions, we need to check every single thing because some changes may have unexpected effects on other parts of the website. 

Some of the questions you can ask are:

• What pages do you usually open for spot-checking? This will tell you how they approach spot-checking versus large-scale testing. 
• What functionalities do you test every time?
• What functionalities do you only test occasionally?
• What are the things that usually break? This will give important information about their previous experience with delivery teams. 

The reason to ask all these questions (and especially the last one) is because, as a QA or development engineer, you might get a “feeling” about the website’s quality. But that feeling is never as accurate as asking straightforwardly. 

Get to Know Your New Project Better

The next thing you want to do is get an idea about the most complex issues in your new project. This complexity may be related to the features or to the testing required.

This stage will let you know which things will take the most to test, which are candidates for testing automation, and which aren’t. 

Ask questions about:

• Features complexity
• Unstable features
• Top complex tests to perform

Start Prioritizing Your Tests

OK, so far you’ve got a pretty good lay of the land. You can now start prioritizing the tests you will be performing. 

First of all, you have to cover all the cases that your client will be testing as well. There’s nothing worse than delivering the website and asking them to test it only to have them get back to you with something that broke on the first page they tested. 

Next, cover the tests that are the most critical to your client:

• Business cases: what are the pages that matter most to your client’s bottom line?
• Sales funnels, donation pages, check out pages should always be at the top of your priority list.
• Check Google Analytics (or the analytics solution your client uses) to see which pages have the highest traffic -- the top 10% of pages. These should also be added to your list of priorities.

Running these tests with priority ensures that, even if a bug exists, it will have a lower impact on the website’s performance and your client’s goals.

Ready to begin testing? Not so fast!

Check out some recommendation to help you streamline the testing process even further.

Testing Plan Dos and Don’ts

Don’t Try to Automate Everything Right Away

Since automated tests can make everything easier and faster, why not jump straight to them? It’s a question we get very often.

We run a test automation SaaS, so it would be in our best interest to recommend that you automate as much as you can as early as you can.

But it would not be in your best interest.

Testing large websites takes a lot of time and effort. You definitely also have a limited budget. 

The mere fact that you write test cases will give you better documentation and a better understanding of how the website works. This will be more useful than automating small pieces of tests from the very beginning. Plus, it will help you understand what you can test automatically and what you can’t, so you can use your budget more wisely.

Use Semi-Automated Tools in the Beginning

Whenever possible, instead of writing documentation, try to record your actions and play them later. Tools like Selenium IDE allow you to record typical actions, like submitting a form and verifying that certain features are live on the website. 

For simply spot-checking pages, you can use screenshot-based testing tools for visual regression, like Diffy. The idea behind these tools is fairly simple: you take multiple screenshots of a website and compare them with older versions or simply browse through them when you need to check that everything is alright.

Moving to Full-Scale Testing

Once you are done with the documentation and the semi-automated tools, you can start discussing full-sclae testing with your client. Why so late?

Because automated, full-scale testing needs infrastructure. Plus, if you don’t run the automated tests often, they will not be useful. 

Ready to see how this would look in practice?

Regression Testing in Action: Examples that You Can Use to Model Your Own Action Plan

Nonprofit Website: YMCA of Greater Vancouver

YMCA of Greater Vancouver is a nonprofit site. 

Their mission-critical web pages and features are:

• Marketing pages
• Blog or event pages
• Calendars integrated with MindBody
• Search functionality
• Donations widget

My suggestion is to cover most of these pages with regression testing -- but not all. 

There is no need to go through every single news or blog article, for instance, for your screenshot testing -- two or three pages of each are enough.

The integration with MindBody is a simple widget, so to save you some time, you can simply collect the links and go back to them to see that the widget is there. 

If you do regression testing on pages like this one, you will get a lot of false positives because every time you take screenshots, the date will be different. Still, all you need to do is make sure that the calendar is there, so you can opt for regression testing and ignore the false positives.
 

The donation page is very similar to the one below:

There is a third-party widget embedded in the website, so this is not a native functionality. Again, you can cover it with regression testing just to make sure it’s there. 

The search functionality works as a separate page, so it’s another good candidate for regression testing:

Again, it may generate false positives, but all you need to do is make sure that it works and that there is no error message because of a faulty third-party plugin.

Tourist Attraction Website: Capilano

Capilano is a suspension bridge park in Canada. 

I chose their website because it’s more complex that the nonprofit one above. They have:

• Marketing pages
• Galleries
• Ticket purchase in a separate website -- the most important element of this website for our purposes. 
• Staging environment with a test credit card

The ticket purchase funnel opens up an entirely new website.

This is where I would start designing the testing environment. You will need a testing credit card to complete the transactions, but you can always start by recording a few scenarios where you buy different types of tickets and go through the whole checkout process. 

It depends on your business priorities  -- but I would check with them to see which types of tickets are usually bough more often -- family tickets, day tickets, and so on. You need this information to avoid covering every single type of possible combination and focus on the most important ones.

Again, given how important this is, this page would be my first priority -- starting with Selenium IDE before moving to full-scale testing. This approach could also help you make a solid case for an increase in testing budget. 

With Selenium IDE, testing might take you longer and you won’t be able to do it every time. With an increased budget, you could finish testing much faster!

Once you are done with the basics, you can add more functionalities to your testing scenarios: buying different types of tickets, changing the date, and so on.

Wrapping Things Up

Creating a regression testing plan is entirely up to the tester. However, it’s useful to remember that the website you test doesn’t exist in a void.

Discuss with your client frequently. Make sure you are clear on their business priorities. Oftentimes, even if a functionality exists, you may not need to test it -- at least not in the early stages.

When in doubt, always double-check instead of performing useless, time-consuming tests.