This post is a report about 2 recent incidents on the GOV.UK website. We’re still working through our incident process, but due to the scale of these issues, we felt it was important to publish our report as soon as possible.
This post covers 2 similar and related incidents:
- an intermittent outage of the GOV.UK website and related services for around 3 hours on Friday 21 October 2016
- an intermittent outage of just the GOV.UK website for around 25 minutes on Wednesday 26 October 2016
Background
Both incidents were related to problems with DNS - a mechanism used to convert a website address like 'www.gov.uk' to an IP address like '151.101.60.144'.
Browsers use DNS to identify the IP address of a website so they can send requests back and forth.
Friday 21 October
On Friday, a third-party company was targeted by a large distributed denial of service (DDoS) attack. The company - Dyn - provides DNS services for many large websites.
At around 5pm on Friday, the attack began. It caused problems connecting to Dyn’s servers around the world. This meant that DNS requests received slow or no responses.
This outage affected the GOV.UK website, GOV.UK blogs on 'blog.gov.uk', government services on the 'service.gov.uk' domain name and legacy systems running on the 'direct.gov.uk' and 'businesslink.gov.uk' domains.
The outage also affected some third party systems that GOV.UK uses including ZenDesk and Basecamp.
The GOV.UK on-call technical team worked to restore the service by removing Dyn from the infrastructure for 'www.gov.uk' and moving DNS for 'service.gov.uk' to an alternative provider.
These 2 domains were prioritised over others as they have the highest number of users, and so were most affected.
However, as several internal GDS systems were unavailable, the external organisation that runs our domain names couldn’t be sure that the change requests we were making were authorised. This delayed the work to restore services.
Service for 'www.gov.uk' and 'service.gov.uk' was restored by 8pm. The remaining services including GOV.UK blogs and legacy systems were restored at 11pm when the DDoS attack ended.
Wednesday 26 October
At around 3:15pm, an external organisation made a planned change to the DNS record for the 'www.gov.uk' domain name.
This change was requested by GOV.UK to restore some resilience following the actions we took during Friday’s incident.
An engineer at the external organisation mistakenly changed the DNS record to point at something that didn’t exist.
This meant that requests for 'www.gov.uk' didn’t work.
Once the mistake was identified, the external organisation reacted quickly to fix the record.
The GOV.UK website was restored by 3.40pm and the total downtime was approximately 25 minutes.
What users saw
The nature of DNS means that it is very easily cached. Temporary copies of DNS responses are often kept in various places, including in your browser, your home/office router and at your internet provider.
Only those who had no copy of the relevant DNS records and needed to request them from our DNS provider would have seen problems.
Additionally, during the Friday incident, access to Dyn’s servers was intermittent, so some requests would still have succeeded.
This means that not everyone would have been affected by these outages.
If they were affected, users would not have been able to access the GOV.UK website, GOV.UK blogs, or other government services hosted on the 'service.gov.uk', 'direct.gov.uk' and 'businesslink.gov.uk' domains.
Immediate things we’re doing to prevent this from happening again
Since Friday’s incident, we’ve made a number of changes to our DNS setup and have several more in progress.
We identified that our DNS provision is a single point of failure. So now we use a second DNS service. We’re also looking to see if there are any other single points of failure.
In addition to the DNS changes, we’re looking again at our monitoring and alerting setup. Several of our monitoring services were affected by Friday’s outage, which means we weren’t alerted in the way we should have been (although this wasn’t a problem in this case).
Finally, we’re looking at how we can provide better information to people if we experience an outage in the future. Many of our usual communication methods were unavailable as they all relied on Dyn’s DNS service.
Next steps
As I said at the beginning of the post, we’re still working through our incident process. We’re reviewing the incident, the causes and our handling of it, and expect more actions to come from that.
We’re also working with our colleagues at the National Cyber Security Centre and other parts of government to coordinate our incident management processes and understand the wider government impact.
We’ll post a further update to this incident report if there are any other disruptions or relevant findings.
6 comments
Comment by Bob Long posted on
You mention DNS caching as a mitigating factor, but can't you quantify how much your traffic was reduced, using your Google Analytics stats vs expected stats for that time of day?
Comment by Dafydd Vaughan posted on
Hi Bob.
Yes, we can get a rough idea from our analytics. We think that overall, our traffic was down around 6% compared to similar days.
We think that DNS caching meant that the vast majority of our users weren't affected.
Hope this answers your question.
Comment by Grant Kelly posted on
I still get "server not found". Any clues as to why?
Comment by Keith Emmerson posted on
Hi Grant,
That's strange, are you still having that problem?
Comment by Grant Kelly posted on
Keith
I cleared the cache and the problem disappeared, but it has returned today. Clear cache again but no luck. Same problem with Firefox and Chrome.
Thoughts?
Comment by Keith Emmerson posted on
Hi Grant,
Can you leave your details here: https://www.gov.uk/contact/govuk
That way someone a bit more technically minded than me can try to help you.
Best wishes,
Keith