https://insidegovuk.blog.gov.uk/2014/10/27/testing-scoped-search-on-gov-uk/

Testing scoped search on GOV.UK

As we transition more organisations and their related content onto GOV.UK we increase the likelihood of returning lower quality results to our users. This blog post outlines one approach we are looking at called scoped search, or automatic filtering, to address this problem.

When an organisation transitions to GOV.UK it brings with it hundreds if not thousands of additional pieces of content, each meeting a new user need. The more content we have on the site, the more competing words and phrases the search engine has to process. So we need ways to differentiate content that uses similar words in different contexts, and differentiate appropriate content from irrelevant content.

As an example, when the Land Registry transitioned in late August our analytics showed a noticeable drop in overall search performance, with more users leaving without clicking on a search result. We hypothesised that this was due to Land Registry users getting poor search results.

Many of their top searches were for generic terms like 'forms', 'fees', 'offices' and 'log in', which depend on the context and return a wide range of results from across GOV.UK, starting with popular content about driving, passports, visas, student finance and so on. Whereas the search function on the previous Land Registry website would only have searched its own content.

To try to address this problem, we decided to see if an automatic scoped search would increase the likelihood of returning relevant search results. Very simply put, this feature turns on the organisation filter by default when searches begin on pages that have Land Registry tagged as the primary publisher.

land-registry-filtered-example
Screen grab of search results page, with Land Registry filter activated

Regular readers of the GDS blog may remember that we looked at scoped search earlier in the year. The conclusion of our user research was that this approach didn’t work in all situations and was frequently confusing users if it was turned on by default across the site, so we temporarily parked the idea.

This time round, we have so far enabled this feature for Land Registry, Environment Agency and Legal Aid Agency, to see how effective it is and help us understand the appropriate scenarios where we should use scoped search.

Land Registry scoped search

We deployed the scoped search feature for Land Registry related content on 1 September.

Before the automatic filter was implemented, not many users were filtering their results to the Land Registry organisation (between 50 and 150 searches a day). From 1 September onwards, there were up to 2,000 filtered searches a day.

The average rate of ‘refinements’, where users change their search term and try again, dropped from around 25% to 20% for Land Registry related searches. As we anticipated, the most visible improvement was in the generic terms where the user need depends on the topic or organisation. For example:

Search term
Refinements before automatic filtering
Refinements after automatic filtering
forms 45% 35%
fees 48% 32%
offices 64% 43%
fee calculator 34% 13%
log in 23% 16%

There was a clear change in the top search refinements, with users no longer having to add 'land registry' to their query to find what they need.

Before:

Search Term
Refined Keyword
1. fees land registry fees
2. forms land registry forms
3. landregistry land registry
4. ap1 tr1
5. offices land registry offices
6. land registry land registry
7. tr1 id1
8. contact contact land registry
9. tr1 ap1
10. ap1 id1

After:

Search Term
Refined Keyword
1. ap1 tr1
2. tr1 ap1
3. tr1 id1
4. ap1 id1
5. land registery land registry
6. landregistry land registry
7. id1 tr1
8. land registry ap1
9. fr1 dl
10. land registry oc1

Environment Agency scoped search

We deployed the scoped search feature for Environment Agency related content on 8 October.

Many of their search terms are fairly specific and were already returning relevant results, so the automatic filtering had less impact than with the more generic terms we saw in the Land Registry transition. However, for some searches such as 'jobs', the filtered results now return relevant content ('Working for EA' in this case) rather than general results from the whole site.

Overall though, the automatic filtering hasn't made much difference to the refinement rates of around 36% for Environment Agency related searches.

Search Term
1. public register
2. hazardous waste
3. environment agency
4. waste carrier
5. esos
6. flood map
7. jobs
8. public registers
9. waste
10. waste exemptions

Legal Aid Agency scoped search

We also deployed the scoped search feature for Legal Aid Agency related content on 8 October.

Again many of their search terms are fairly specific, often for a particular form, so the filtering had less impact overall but it does improve results for more generic terms.

The rate of refinements on Legal Aid Agency searches improved a little, from an average of 37% to 35%. As with the Land Registry, some users were searching for generic terms like 'forms' and then refining it to 'legal aid forms', which is no longer necessary with the automatic filter in place.

Search Term
1. legal aid
2. cw1
3. legal aid agency
4. if1
5. forms
6. family mediation
7. mediation
8. laa
9. if2
10. af1

Next steps

We’re not expecting to deploy further instances of scoped search for a few months. Over the coming sprints we plan to look for further evidence of where scoped search will add most value and where it might not work so well, and we’ll summarise those conclusions on this blog.

2 comments

  1. Lissa Allcock

    I suspect that you will want this also when you get more of the HMRC material onto the site, though it may be beneficial to include (tax specific?) Treasury results in searches begun on the HMRC page also. There is little more furstrating than trying to track down if a particular bit of tax related consultation is from HMRC or the Treasury.

    Link to this comment
  2. Andrew Robertson

    Thanks for blogging about the results. I am surprised refinement rates haven't changed, though we have been working on SEO.

    Is there any way of knowing if people spend less time on search results pages with scoped search? I just wonder if refinement rates aren't much different for Environment Agency content but there are fewer results to look through, so it is quicker for users to get to what they want?

    For example, the top result for ‘public register’ is the same with and without the EA org filter. But without it, you also see ‘register to vote’ (2nd) and the home page of The Charity Commission (3rd), and lots of other registers which may seem less relevant to results for a search from the Environment Agency home page.

    Thanks for letting the Environment Agency pilot scoped search.

    Link to this comment