As we transition more organisations and their related content onto GOV.UK we increase the likelihood of returning lower quality results to our users. This blog post outlines one approach we are looking at called scoped search, or automatic filtering, to address this problem.
When an organisation transitions to GOV.UK it brings with it hundreds if not thousands of additional pieces of content, each meeting a new user need. The more content we have on the site, the more competing words and phrases the search engine has to process. So we need ways to differentiate content that uses similar words in different contexts, and differentiate appropriate content from irrelevant content.
As an example, when the Land Registry transitioned in late August our analytics showed a noticeable drop in overall search performance, with more users leaving without clicking on a search result. We hypothesised that this was due to Land Registry users getting poor search results.
Many of their top searches were for generic terms like 'forms', 'fees', 'offices' and 'log in', which depend on the context and return a wide range of results from across GOV.UK, starting with popular content about driving, passports, visas, student finance and so on. Whereas the search function on the previous Land Registry website would only have searched its own content.
To try to address this problem, we decided to see if an automatic scoped search would increase the likelihood of returning relevant search results. Very simply put, this feature turns on the organisation filter by default when searches begin on pages that have Land Registry tagged as the primary publisher.
Regular readers of the GDS blog may remember that we looked at scoped search earlier in the year. The conclusion of our user research was that this approach didn’t work in all situations and was frequently confusing users if it was turned on by default across the site, so we temporarily parked the idea.
This time round, we have so far enabled this feature for Land Registry, Environment Agency and Legal Aid Agency, to see how effective it is and help us understand the appropriate scenarios where we should use scoped search.
Land Registry scoped search
We deployed the scoped search feature for Land Registry related content on 1 September.
Before the automatic filter was implemented, not many users were filtering their results to the Land Registry organisation (between 50 and 150 searches a day). From 1 September onwards, there were up to 2,000 filtered searches a day.
The average rate of ‘refinements’, where users change their search term and try again, dropped from around 25% to 20% for Land Registry related searches. As we anticipated, the most visible improvement was in the generic terms where the user need depends on the topic or organisation. For example:
Refinements before automatic filtering
Refinements after automatic filtering
There was a clear change in the top search refinements, with users no longer having to add 'land registry' to their query to find what they need.
|land registry fees
|land registry forms
|land registry offices
|6. land registry
|contact land registry
|5. land registery
|8. land registry
|10. land registry
Environment Agency scoped search
We deployed the scoped search feature for Environment Agency related content on 8 October.
Many of their search terms are fairly specific and were already returning relevant results, so the automatic filtering had less impact than with the more generic terms we saw in the Land Registry transition. However, for some searches such as 'jobs', the filtered results now return relevant content ('Working for EA' in this case) rather than general results from the whole site.
Overall though, the automatic filtering hasn't made much difference to the refinement rates of around 36% for Environment Agency related searches.
|1. public register
|2. hazardous waste
|3. environment agency
|4. waste carrier
|6. flood map
|8. public registers
|10. waste exemptions
Legal Aid Agency scoped search
We also deployed the scoped search feature for Legal Aid Agency related content on 8 October.
Again many of their search terms are fairly specific, often for a particular form, so the filtering had less impact overall but it does improve results for more generic terms.
The rate of refinements on Legal Aid Agency searches improved a little, from an average of 37% to 35%. As with the Land Registry, some users were searching for generic terms like 'forms' and then refining it to 'legal aid forms', which is no longer necessary with the automatic filter in place.
|1. legal aid
|3. legal aid agency
|6. family mediation
We’re not expecting to deploy further instances of scoped search for a few months. Over the coming sprints we plan to look for further evidence of where scoped search will add most value and where it might not work so well, and we’ll summarise those conclusions on this blog.