As important as it is to highlight the successes of GOV.UK, it's just as important that we discuss our mistakes.
A little over a week ago we began receiving reports of attachments that were incorrectly ordered on the site, or claiming to be one thing but linking to something else, or even missing entirely. This has led to a week of digging around in databases and code repositories to uncover and fix a complex bug which started laying its foundations all the way back in July.
23 July
We release a code change that allowed the manual ordering of attachments. Any new or changed attachments would gain a slowly incrementing ordering number, so attachments would by default be presented in the order of their creation.
Existing attachments were not given an ordering number, which was the source of the eventual issues although no errors would appear until…
7 October
We release a code change which changed the way attachments were associated with their documents to make the system simpler and more flexible.
Whenever a new edition of a document is created, all associated attachments are copied over so we have a history of changes to attachments. Previously any missing ordering numbers on attachments were left empty, but this code change subtly altered the way attachments were copied over, and the copy process started filling in the empty ordering numbers.
Even this wouldn't have caused any problems if they'd all been empty, but any attachments added between 23 July and 7 October already had ordering numbers and these began to clash, causing later attachments to jump earlier in the sequence. Here's a demonstration:
1. Before 23 July, empty ordering numbers:
[nil, nil, nil, nil]
2. Editioned to add attachments between 23 July and 7 October, new attachments are given ordering numbers:
[nil, nil, nil, nil, 0, 1, 2, 3]
3. Editioned again for any reason after 7 October, empty ordering numbers filled in:
[0, 1, 2, 3, 4, 0, 1, 2, 3]
4. When presented in sequence using the ordering numbers, you get this effect:
[0, 0, 1, 1, 2, 2, 3, 3, 4]
To exacerbate this problem, if the document was editioned again and the attachments copied, this incorrect ordering was cemented in our database, making it much harder to find the correct ordering of editions created after 7 October.
The incorrect attachments
So this explains how some attachments shifted order, but what about the ones which said they were one thing but were actually another? Well the interface we've provided our editors allows them to reference attachments in the body of a document using a reference number. This reference number was the very same ordering number which were now shuffled around. If the editor had provided custom text for the link, or perhaps had placed a title above the link for context, the wrong attachment would have been put in place.
The missing attachments
Another side-effect of the 7 October code change was the disappearance of attachments on policy supporting pages when new editions of the policy were created. This was due to the system only copying the relationship between the supporting page and the attachment rather than the attachment itself. This issue was present long before the 7 October change but only appeared when these relationships were changed.
The fix
After a lot of work and a few false starts we've produced a fix which has restored almost all the attachments to their previous state.
- All missing attachments have been reinstated.
- All automatic attachment orderings have been restored.
- All manual orderings which were set before 10am on 7 October have been restored.
- Although we have tried to avoid disturbing documents where editors have already manually fixed the ordering via via the editor, some manual orderings set after 10am on 7 October may have been lost. These attachments will have reverted to their default ordering. Please get in touch with us if we can help you fix any remaining issues.
How it could have been avoided
We've reviewed our processes for catching errors like these, and despite their rigorous nature, it was the combination of two code changes months apart that caused the problems. Issues like this are rare but very difficult to spot even when the bug reports start coming in.
More stringent guides on preventing empty or duplicated ordering numbers would have helped, and we've put those in place for any future tasks that require custom ordering.
We will be shortly altering the way attachments are referenced in documents to prevent ordering issues from causing the more egregious issues we encountered.
Although this would not have prevented this particular issues, we will also be automating more aspects of our deployment to reduce the number of human errors that can creep into the system.
How we can help
Although it should be rare, if you find any remaining documents with incorrect orderings which would be arduous or time consuming to fix through the editor, get in touch with us and we'll help you out.
This issue has been our highest priority these last couple of weeks and we're truly sorry that it happened, both for the impact on the editors and departments, and on the users coming to GOV.UK with the expectation of reliability and clarity of information.
We hope to serve you better in the future.
Featured image by Foomandoomian on Flickr.
