A security update that broke every page: why we release website changes in stages

A routine security update can pass every automated check and still return an error on every page of the live site. It happened to ours on 12 October. The lesson is about process: updates that matter for security still need to be released in stages, and the live site needs to be checked by something other than the build.

What happened

A security advisory was published for the framework our sites are built on, and the fix was in a newer version. We upgraded pagecog.com.au to the patched release. The site built without errors and the type checker, which catches a large class of mistakes in the code, reported nothing wrong.

We deployed it, and the live site returned a server error on every page.

The cause turned out to be two copies of the same internal routing library in the project. The upgrade had left our own pinned copy of the router at one version, while the framework brought its own, slightly newer copy along with it. The build and the type checker each looked at one copy and were satisfied. At runtime on the live hosting, the two copies disagreed with each other. We should be plain that this explanation is inferred: it fits the evidence, and aligning the versions fixed the site, but we could not reproduce the broken state again afterwards to prove it.

The fix was to bring our copy of the router and its core library to the versions the framework expected, and bring the matching build plugin along with them, until the project's own dependency listing showed one copy of each. Then we redeployed.

The other nine sites we upgraded the same way had no pinned copy of the router, and all went through without trouble.

Why the checks did not catch it

A build tells you the code can be assembled. A type check tells you the pieces fit together as the code describes. Neither of them runs your site the way a visitor's request does, on the real hosting, with the real settings.

This is the same reason a contact form can look fine and send nothing, which we wrote about in the contact form that says sent but delivers nothing. The thing that fails is the join between the code and its environment, and only a live request exercises that.

What we do not know

We do not have a reliable measure of how long visitors saw errors, so we are not going to put a figure on it. What we can say is that the release was the kind that should have been staged, and was not. Our own site has logins and a database behind it, which is the case where a straight release to everyone is hardest to justify, and the sites with the least to lose are the ones where it is easiest to forget.

What we do now

1. Check for duplicate copies after any upgrade. A single command lists every installed version of the router libraries. If either appears twice, we fix that before anything is released.
2. Release to a small share first. Our hosting lets us upload a new version without sending it live, then give it a small share of real traffic. We use 10 percent to the new version and 90 percent to the old one.
3. Request the live site. We load real pages and check the response before sending the new version to everyone. If they are errors, the old version is still serving 90 percent of visitors.
4. Promote to everyone only after that.
5. Know the way back. A rollback to the previous version takes seconds, so the cost of a bad release is small if you are watching.

For a site with logins or a database, we do not release a dependency upgrade straight to everyone at once. A brochure site that fails loudly is bad enough. A site with accounts that half works is worse.

What a business owner should take from this

You do not need to understand routers. You need to know three things about whoever looks after your site:

• Do they apply security updates? A site left on old software gathers known weaknesses.
• Do they check the live site afterwards? "The deploy succeeded" and "your site works" are different statements.
• Can they roll back quickly? Ask how long a bad release would take to undo.

If the answer to the second is a shrug, the first is risky. Updates are necessary, and an update with no check behind it is a gamble you take with your own customers.

Why this is part of a monthly arrangement

Applying an update, checking it and being ready to reverse it is a small amount of work done at an unpredictable time. Under a one-off build, who does it afterwards is whatever was agreed separately, and often nothing was. On our base plan the upkeep of the site is part of the monthly fee, which is also why we have an incentive to get it right: if the site is down, the customer is the one who feels it, and then we hear about it. Our post on what a website maintenance plan should cover lists what to ask any provider, and our web design service page describes how we build.