What is Regression Testing and How Can You Perform it Efficiently
“Regression testing” has a solemn sound to it. It conjures images of complicated experiments performed by people wearing pocket protectors. But what it really ought to conjure is images of playing whack-a-mole at the arcade: it’s the software world’s equivalent to standing guard and ensuring that old code remains free of new problems, and that old problems don’t rear their ugly heads again.
What Is Regression Testing?
Regression testing is a technique that consists of testing existing features which has already been tested, after an app or a website has undergone some changes, in order to ensure that said features still work as intended.
In other words, the purpose of regression testing is to ensure that updates or newly-added features have not broken any existing features and that bug fixes only fixed the bugs they were meant to fix, without introducing new ones or causing old issues to appear again.
Regression Testing: Is it Important?
Regression testing sounds a lot like simply “re-testing” a feature, so what’s the point? Hasn’t it been tested already? Why test it again?
Let’s start with a simple analogy.
While something is being built – a car, a bridge, an app, a website – defects can creep in at any time. It’s in the nature of building things.
Most of the time, defects occur in newly-added components. For example, when you’re building or refurbishing a house, most of the problems introduced by painting the walls have to do with, well, the walls. You miss a spot here or chip the wall there.
But sometimes you break other things, like when ceiling paint drips over the doorknobs that you just cleaned or replaced. This is a regression: the doorknobs were dirty, so you cleaned them, but now they’re dirty again. Your doors are just as ugly as they were before.
Regression testing is how you ensure that your website or your app gets steadily better with time. That’s why it’s so important to perform regression testing regularly: it keeps things in line, like the steady marching drum that makes the slackers in a column pick up the pace.
Without it, existing components degrade as defects creep in – the software “rots”, as it were – but instead of figuring that out on your own, you learn about it from your customers.
Is regression testing the same as re-testing an existing component? Not quite – this “marching drum” meaning is what sets regression testing and re-testing apart.
Regression testing is performed against a known good baseline, not a spec or an existing bug report. Its aim is to confirm that existing functionality continues to work as expected after making some changes. Or, to put it another way, that your website has indeed improved after these changes, instead of regressing to a worse state.
The most important benefits of a reliable, regular regression testing process are not limited to customer retention. Regression testing is prominently featured in best practices guides because it enables steady improvement across a steady schedule. Since you don’t have to keep going back to fix issues you never knew about, your development schedule is less likely to be derailed.
Types of Regression Testing
Our more schedule-oriented readers likely have a tough question racing through their minds already. For many websites out there, “existing, already tested functionality” is a lot of functionality. Surely if you were to test all of it, all the time, there would hardly be any time left to do anything else!
Indeed, based on how regression test selection is done, we can break down the regression testing process into two types: complete regression testing and partial regression testing.
Complete Regression Testing
Complete regression testing is exactly what it sounds like: you comprehensively and thoroughly test all the functionality that your app or website exposes. Every button, every page, and every form.
While it sounds wasteful, complete regression testing is the only effective selection strategy when dealing with sweeping changes, like a major WordPress upgrade or a change in the theme code. If a change has the potential to impact every page on a website, then you do need to check every page.
Website regression testing is particularly difficult to perform in this manner. Many defects, especially visual problems like misaligned page elements, only occur on specific combinations of screen size, browser versions, and operating systems. The contact form on your Contact page may look fine in Safari on your iPhone and not work at all in Chrome under Android. In other words, the regression testing effort is multiplied by every device type, form factor, and platform that you need to target.
However, solutions do exist to mitigate the time impact and the effort involved in complete regression testing. Automated regression testing tools can significantly reduce the amount of time required to perform a complete regression testing run.
Partial Regression Testing
Partial regression testing is the logical alternative. Instead of going through the entire regression testing plan again, you perform only a subset of the required tests.
The idea behind partial regression testing is that some changes can be reasonably expected to have a limited impact. For example, if you update the theme code to change some rules about how form elements are positioned, it’s generally safe to assume that only pages that actually have forms will be affected.
Based on an impact assessment, you can therefore do a more radical regression test selection, and perform only a limited range of tests, instead of going through the entire regression test suite.
It sounds like it’s way better but in practice, even with a limited set of test scenarios, regression testing is still a lot of effort due to the platform multiplier effect we mentioned earlier.
Even a short series of tests still have to be replicated across a large series of device types, screen sizes and orientations, browsers, and so on. Consequently, automated regression testing is important even when testing changes that have a limited impact.
How to Perform Regression Testing
So how do you go about planning and implementing a sound regression testing process? Surely, if it were as simple as “just test everything you already have”, software bugs would be history by now – yet clearly, they aren’t.
Visual testing and regression testing. The starting point is figuring out what to test, and the good news is that there’s a simple rule of thumb you can apply here: if your customer can see it, then you should test it.
Web browsing is fundamentally visual, so defects in website interactions are visual in nature. Consequently, visual testing is the fundamental component of website regression testing.
Simply put, visual testing consists of comparing how the website looked in a known good state to how it looks now. You grab screenshots of every page and the interaction feature is compared against known good versions, across all the device types, screen sizes, orientation, platform, etc. that your customers use, and differences are highlighted and assessed.
Visual regression testing is highly effective, as it is inherently tuned for detecting defects that customers immediately notice and therefore have a high customer impact.
However, it is inherently high-effort. Visual regression testing is rarely performed manually, by a human tester – instead, a visual testing tool is used. These are regression testing tools that automatically perform these checks at a high rate, producing a test report that can be used to assess and prioritize defects.
How is regression testing integrated? Regression testing doesn’t happen in a vacuum. It’s part of wider testing progress, so best practices dictate that it has to be integrated not only within this process but also with your development, deployment, and maintenance processes.
Website maintenance: Staging and Production environments. Since we mentioned customer impact, some of you are probably thinking that if you run your regression testing suite on the same website that your customers see, it might be too late.
After all, every defect that you identify on your website is a defect that may have already driven a customer away.
That’s true, and the common solution today is to have separate environments for staging and production.
The staging environment is a private environment, with limited, restricted access, where you can test and verify changes. The production environment (sometimes called the live environment) is where your actual website gets deployed, the one that customers can see.
When you add a new feature, or when you update your CMS – be it WordPress, Drupal, or any other CMS – you perform the changes in the staging environment. This is where you perform all testing – including software regression testing.
After all, bug fixes have been committed and all required configuration changes have been identified, you deploy the changes to the live environment as well. This way, everything your customers get to see has been verified.
Maintaining two different environments can be time-consuming and expensive. In some cases, especially for low-traffic websites, or websites where customer traffic is highly time-concentrated, the risk of deploying straight to production can be acceptable.
In this case, a backup is performed before deploying any changes. The changes are then deployed straight to the production environment, and regression testing is performed on the live copy. If these defects cannot be resolved quickly enough, or if they have a critical and wide-ranging impact, the backup is immediately restored.
Regression Testing and Agile Development: CI integration. While regression testing is always required at the end of a development sprint or a significant update, regression testing can also be performed during development, as part of a continuous integration (CI) pipeline. In fact, integrating regression testing functionality in the CI pipeline is the norm for modern app development environments.
Many visual testing tools, like Diffy, expose a REST API that allows you to trigger an automated regression testing job as part of a regular CI test suite. This way, every pull request can be visually tested, and defects can be identified early in the development process.
Who Should Perform Regression Testing?
Regression testing is essential for any company whose business is strongly intertwined with its online presence. If your website is an essential component of your business identity and processes, then a visual defect on your website has an immediate impact on your business goals. A defective website isn’t just something that could look better, it’s a defect in a core component of your business.
This situation is uniquely challenging because while the need for regression testing is widespread, the ability to seamlessly perform it is limited. Many automation testing tools for web applications are primarily targeted at QA teams, but regression testing, and especially visual regression testing, is not necessarily a QA-only job.
In fact, since visual testing is often performed outside a development process – for example after deploying WordPress updates – it’s important to seamlessly expose visual testing capabilities as widely as possible within an organization.
Regression Testing in Agile Development
It is no coincidence that the rise of automation testing tools for web applications has largely coincided with the rise of Agile development processes. Traditionally, regression testing has been the bane of waterfall methodologies: long development cycles routinely resulted in testing reports that were littered with regression defects, which in turn resulted in endless bug fixing sessions.
Identifying software regressions early in the development process is one of the most important benefits of agile development processes. This requires close regression testing tool integration – for example, as part of the CI pipeline – but, more generally, relies on a comprehensive, fully automated regression testing framework.
There’s a simple and effective test for assessing your agile regression testing framework: can anyone who wants to test their changes run a software regression test on demand? If they can, that’s good news: defects caught early are orders of magnitude easier and faster to fix.
Easy access to regression testing solutions enables efficient visually-centered development. Whenever new features get developed, you can approve/refuse screenshots based on a comparison with baseline images from known good environments. This enables fast, objective decision-making during intermediary stages.
An Effective Regression Testing Strategy
Put simply, an effective regression testing strategy is one that enables you to steadily improve your business and provide effective services 24/7.
Test methodologies are often benchmarked strictly through the defect count, the idea being that an efficient methodology is one that identifies more defects. Unfortunately, this leads to “noisy” testing reports, that frequently include low-impact, and often subjective defect reports in order to make the test process seem more thorough.
This is one of the things that makes visual testing particularly useful for website regression testing. Since it is inherently tuned to defects with visible customer impact, defects identified through visual testing are immediately actionable and easy to prioritize. And, since they rely on a comparison against a known good baseline rather than a subjective assessment, they are inherently aligned to your business goals.
Manual or Automated Testing in 2021
Is manual regression testing still an option in 2021?
We are many decades away from delegating substantial decision-making problems to our machines. In this sense, regression testing will have a human component for the foreseeable future.
However, fully manual regression testing is not sustainable. The repetitive part – comparing images grabbed from various environments – is time-consuming and error-prone. This is the part that machines perform better than people, where software can outmatch even the most dedicated QA team.
Speaking of software that can outmatch human-led teams, do you want to take your regression testing to the next level? Try Diffy, our visual regression tool 100% FREE. No hassle, no obligations; we won’t even ask for your credit card number, so what have you got to lose?