For some time now we’ve been working on rebuilding GOV.UK’s publishing platform. This post updates on what we’ve done so far, what’s left to do, and goes into more detail about the why - explaining the benefits of this work, both for us and for GOV.UK’s users.
Last time we wrote about this was in Brad Wright’s post. We wanted to provide an update on our progress...
To recap, the purpose of rebuilding the platform is to ensure that GOV.UK is scalable, secure and easy to work on. GOV.UK currently consists of a number of applications (like Whitehall Publisher, Specialist Publisher and Travel Advice Publisher) that we want to consolidate.
These applications were originally built to serve different needs. Travel Advice Publisher, for example, was created solely to provide and manage foreign travel alerts. Each application was built in isolation from the others, but in most instances have many functions (like how they publish to GOV.UK) that are common to all. When we brought government content together for the first time, it was more efficient to build our publishing tools piece by piece.
But now we can see the bits of functionality that are common to all of them and the bits that aren't, we can build a more efficient platform.
The work to rebuild the publishing platform is going well. Since Brad’s post, we’ve come a very long way and are starting to reap the benefits, but we still have a long way to go.
Bringing the teams together
When we started this work, we had a small team working on building the new publishing API, with all other product teams working on updating their applications to use it.
Now that we are further along with the work, the teams are unified under one programme of work. This means that our velocity to deliver the work is increasing as we can more rapidly resolve dependencies between the publishing API and publishing applications.
Technical benefits
Rebuilding our publishing tools is all about reducing technical debt, which in practical terms means we are:
- Migrating all formats to use the publishing API
This means that when a format (like a news story) is published to GOV.UK, it will be via one consistent method - the API. Therefore we need to migrate Whitehall, Mainstream, Specialist and Travel Advice formats. There are 141 formats within the publishing applications in total. - Ensuring that each format uses the right front-end
This means that when publishing certain types of content, like a statistics announcement, the right front-end application is used to render the content on GOV.UK. That allows us to retire unused front-end applications and standardise the components that make up how GOV.UK looks. - Cleaning up overly complex code and removing unused code
As we update formats to using the publishing API there will be bits of the old architecture we can delete, because they are no longer used. Similarly, we are starting to rewrite Specialist Publisher, for example, so that it’s less complex. - Improving system consistency
Doing this allows us to share processes and functionality across the publishing tools. So, for example, we want to be able to clear the cache across the whole of GOV.UK more easily, and ensure that all publishing applications have a single solution for previewing content before it goes live on GOV.UK. - Merge all the databases into one
At the moment each of our applications (like Whitehall Publisher) has its own database. This means that we are maintaining lots of different repositories of data that make up GOV.UK. That can cause duplication, data mismatches and so on. We need one version of the truth.
The technical benefits of this work means that:
- new starters at GOV.UK will be able to begin work more quickly and with greater confidence because the technical detail behind GOV.UK will be simpler
- it will be quicker for us to find and fix problems
- we will reduce the risk of code changes causing unintended consequences
- we’ll be able to iterate components and design elements and share what we’ve learnt across all the applications
- we’ll be able to add new back-end functions or front-end templates faster because they’re built on shared components
Why this work is such a high priority
This work forms the foundation of Neil’s vision for GOV.UK as the rebuild allows us to speed up or start on planned work, such as:
- Improving search, navigation and content tagging
We know that finding things on GOV.UK is one of the biggest frustrations for users. The work is already underway with our ‘Finding Things’ team, but their work will speed up as technical debt is reduced. - Reducing the number of formats
There are currently 141 formats, and we want to reduce this to fewer than 10 templates, which will have in-built flexibility. This is important for publishers because when we add new functionality they will all benefit from it. - Improving workflow for publishers
Part of this will be delivered via the move to the publishing API, but we are also in discovery now to identify the improvements we could make for publishers, like developing the permission levels for publishing. - Building improved features for GOV.UK
We know there is a lot of demand for things like improving print functionality. We want to do this type of work after the publishing tools rebuild so that the work is quicker to deliver, and is consistent across all applications. This will also discourage piecemeal development of features that have multiple user needs. - Querying analytics across GOV.UK
At the moment we can’t follow user journeys across different types of content because of the different applications involved. This change will mean we’ll have a much better understanding of user journeys, and the improvements we need to make. - Enabling others to build on and reuse GOV.UK
This could take the shape of providing an API for content (pulling data from GOV.UK to other websites, such as for devolved and local government sites) or an API for publishing (pushing data to GOV.UK from other government-owned applications). - Develop GOV.UK so it is its own archive
As documents are updated on GOV.UK to keep pace with changes in law and policy, we want to provide a transparent way for users to identify what changes have been made over time.
What we’ve done so far
We’ve made good progress in making the publishing API meet the needs of the different formats so they can start using it - each format has its own special features!
A number of Whitehall Publisher formats have already been successfully migrated, Specialist Publisher is almost there and Travel Advice Publisher has been migrated, albeit with a few unexpected complexities that we’ve needed to attend to in urgency.
We’re just about to start investigating what will be involved in migrating Mainstream Publisher.
We are tracking the analytics to measure the improvements we make in performance, and so we are alerted if we accidentally introduce issues in the process.
What’s left to do
We expect this work will continue to dominate our roadmap for much of the 2016 to 2017 financial year.
We really are conscious of how frustrating this delay is for people, publishers in particular, who want improvements now. But if we were try and deliver those things now we would be doing so in a way that would lead to problems in the future. The need for technical improvements to the architecture would only become more pressing and more complicated as time goes on. So, we’re laying the foundations for the future.
We’ll continue to update you as we make progress.
Jennifer Allum is a GOV.UK Product Manager. You should follow Jen on Twitter.
3 comments
Comment by Gerrit Berkouwer posted on
Interesting stuff! I have one question: what are the reasons that you never chose to implement one of the available open source CMS-systems (e.g. Hippo CMS, Drupal, Typo3 or others) to manage your content from 1 repository?
Comment by Jennifer Allum posted on
Hi Gerrit,
In the early days of GOV.UK using an existing CMS was discounted because there was concern that the team would have to apply lots of workarounds to get it to work as we wanted. In other words, in order to meet the needs of users as we hoped, the short term gain wouldn't be worth the long term impact.
Similarly, it was also felt that taking an existing system could constrain, unhelpfully, the approach taken to the technical architecture too.
It's never an easy decision to get right, whichever route you go!
Thanks,
Jen
Comment by Mark Craddock posted on
It beggars believe, that one of the existing CMS's that power millions of websites, for 100's of millions of users, couldn't meet the needs of our users.