What Kind of Things Break During WordPress or Drupal Updates and How to Fix Them Quickly

Much of the modern Web is powered by open source content management systems (CMS) like WordPress or Drupal. Like all software, though, even the most reliable CMS needs bug fixes, and all world-class software gets new features in time. And sometimes, – we’ve all been there – no matter how good the intentions behind them are, some updates break things.

This is what we’re going to talk about today: 

• What kind of things can break on CMS updates? 
• How can you spot them? 
• What can you do about them? 
• What can you do to prepare?

How and When do Updates Get Released?

This may sound like a boring subject but it’s actually very relevant: if you understand how CMSs like WordPress manage their releases and updates, everything else just falls into place.

Both WordPress and Drupal are open-source software. They are developed by a wide community whose members range from indie developers working in their spare time to large companies. They all work on various pieces of a program, all the time. The code is available all the time, too.

Every once in a while, a snapshot of this code gets through a review and testing process and gets released – and you can update it. Each community has its own rules according to which it decides when a new release is made and what it contains, and how that release is managed.

WordPress, for example, makes three types of releases: 

• major releases 
• minor releases
• maintenance and security releases

Major releases – 1.0, 2.0, and so on – are made once every few years. The community puts a lot of good-faith effort into backward compatibility, but with major releases, some features are deliberately dropped, or radically changed. Themes and plug-ins can and do break – in fact, they usually need to be updated in order to support new major versions.

Minor releases (5.1, 5.2, 5.3, and so on) are not quite like that. They add or improve new features, but backward compatibility is an explicit goal. Breaking changes can occur, but they are a result of bugs more often than a deliberate decision.

Finally, maintenance and security releases, like  5.0.1, 5.0.2, 5.5.1, 5.5.2, and so on, are usually even more subtle. They are usually published in order to fix important bugs or security problems and, since they make no major functional changes, they rarely break things. 

That’s not always the case, though. For example, a while back, the 5.0.1 release had to make some breaking changes in order to fix some security issues.

Drupal’s approach is fairly similar. Drupal also makes a major, all-bets-are-off release once every few years, with minor releases in-between. However, security and maintenance fixes are released monthly, according to a strict calendar, for all supported versions.

What Can Break During Updates – And Why?

Most WordPress and Drupal websites rely on an extensive array of plug-ins and custom extensions. Furthermore, most of them use either a custom, unique theme, or a customized theme. These aren’t a part of the WordPress or Drupal code, so testing them is up to their developers.

This is where most of the breakage occurs. Core CMS features rarely if ever, get broken as part of an update – the release process is designed to guard against that. 3rd-party code that interfaces with core CMS code – such as plug-ins and themes – is more likely to break after updates.

Plug-ins and themes depending on the plug-in and theme APIs of their respective CMS. Sometimes APIs change completely, and then some plug-ins or themes have to be updated before they work at all. 

But even then, in an effort to ensure compatibility with a new version, developers can introduce new bugs. In other cases, the behavior of an API function changes only slightly, which can cause only a specific feature of a plug-in or theme to break. 

These problems are often highly specific – for example, images of a certain width may not be aligned correctly on screens of a certain size – which makes them very hard to spot.

All releases are announced ahead of time, so plug-ins can be tested beforehand. However, not all plug-in developers have the same testing capacity.

Furthermore, there are many possible combinations of plug-ins, themes, and customizations, and many of these are developed specifically for a particular website. Even the most thorough testing process cannot cover them all.

In other words: whenever an update is applied, it’s ultimately up to the users to test their deployment.

How Do You Know If Something Broke?

Many breaking changes are hard to spot because they only occur on some pages. For example, a navbar can look fine on all pages except for those that lack a header image – and even then, only on some browsers, or on screens of certain sizes or orientations.

Furthermore, many changes are confined only to certain types of content, or to certain sections. For example, only the language switcher pop-up, or one of the slides on a homepage, maybe rendered incorrectly – and, again, only on some browsers, or on some screens.

The problem with bugs like these is that their impact is primarily visual, not functional. The data is right -- it’s just not presented right.

The testing technique that’s most effective at spotting them is called visual regression testing. Visual regression testing is very much like those “spot the differences” puzzles you’ve seen in children’s stories. After a CMS update, each page is visually compared against a reference image that shows what the page is supposed to look like -- usually, the way it looked before the update -- and any differences are highlighted. This process is repeated for every target platform, media type, screen size, position, and so on.

Visual regression testing is extremely useful because it can catch a virtually unlimited array of bugs with a visual impact. Many bugs manifest themselves in endless combinations that most of us would generally describe as “it just doesn’t look right”. Visual regression testing is a way to turn that exasperating label into a useful testing technique: it’s a way to catalog all instances of things that “just don’t look right” after an update.

Small enough websites can be checked manually, but it is difficult to systematically ensure that all platforms and device types are covered. Furthermore, manual visual testing is an intense and error-prone process. Many differences are subtle enough that it’s hard to spot them after you’ve looked at pages for a few hours.

For larger websites, thorough manual testing is outright impossible. Testing hundreds or thousands of pages, even across a conservative range of devices and platforms (say, just one or two mobile devices in portrait and landscape orientation, and one, common, 16:9 desktop/laptop monitor resolution) can take prohibitively long. Due to the inherent limits of human perception, the error rate of such a process can be too big for it to yield any useful results.

Automated tools that walk through the entire website, across various platform-specific parameters, and detect potential inadvertencies, can reduce the review times by orders of magnitude. These tools perform the same process, but at a much faster rate, and without running into the physical limits that human biology imposes.

Want to make sure you never miss a visual bug? We’ve got you covered! Check out our features for WordPress and Drupal.

How Do You Deal With Broken Updates?

Avoiding broken updates poses a bit of a chicken-and-egg problem. If plug-ins or themes can break in ways that are specific to your deployment, how do you know they break without deploying them?

The most common strategy is to deploy them in a staging environment – a private environment that only you have access to. Once you’ve tested everything (and made any required changes) you can deploy the update in the production or live version – the one that everybody sees.

Deploying to the staging environment first allows you to test your CMS in a controlled, private setting, where your customers won’t get to see any of the bugs that an update might bring about. Once an update has been deployed to your staging environment, you can employ visual regression testing to spot any differences that the update may have brought about., and make any necessary fixes 

This has two advantages:

• It prevents bugs from creeping into your production website where everyone can see them
• It allows you to test your fixes so that you can make sure all bugs have been fixed before you deploy them to your production environment

Some hosting companies offer separate staging/production environments as part of their more advanced packages, but even if they don’t, it’s fairly easy to roll your own. After all, it essentially boils down to hosting two websites.

Sometimes, though, broken updates do make it to the production website. And many businesses can’t afford the technical or financial burden of maintaining two environments. What can you do?

In these cases, the first response is often in the form of a roll-back. A backup is made right before the update, and if the update fails, you revert to the backup. This is a very broad, but effective sword. 

Most hosting packages include one form of backup or another, and various CMS plug-ins can help you with this, too (WP Downgrade is one of the most popular WordPress plugins, for example).

If the problem is confined to a specific plug-in or theme feature, it is sometimes possible to selectively disable them until the compatibility problem is resolved.

Whatever method you choose, it’s important that you never wait until an update breaks in order to figure out how you’ll deal with it. You want to test back-ups and roll-back features – either offered by your hosting company or through a CMS plugin – on a regular basis, so that you can rely on them when things go south.

However, at the end of the day, the most effective way to ensure that your customers never see a broken page is to never deploy a broken page to your production environment in the first place.

It’s always better to have the tools and the infrastructure to test an update than to deal with a broken update.

Visual regression testing is the ideal tool to help you conduct thorough testing. An automated visual regression testing platform can help you organize and speed up the testing process and take most of the QA work off your plate. Check it out!