As we transition more organisations and their related content onto GOV.UK we increase the likelihood of returning lower quality results to our users. This blog post outlines one approach we are looking at called scoped search, or automatic filtering, to address this problem.
When an organisation transitions to GOV.UK it brings with it hundreds if not thousands of additional pieces of content, each meeting a new user need. The more content we have on the site, the more competing words and phrases the search engine has to process. So we need ways to differentiate content that uses similar words in different contexts, and differentiate appropriate content from irrelevant content.
As an example, when the Land Registry transitioned in late August our analytics showed a noticeable drop in overall search performance, with more users leaving without clicking on a search result. We hypothesised that this was due to Land Registry users getting poor search results.
Many of their top searches were for generic terms like 'forms', 'fees', 'offices' and 'log in', which depend on the context and return a wide range of results from across GOV.UK, starting with popular content about driving, passports, visas, student finance and so on. Whereas the search function on the previous Land Registry website would only have searched its own content.
To try to address this problem, we decided to see if an automatic scoped search would increase the likelihood of returning relevant search results. Very simply put, this feature turns on the organisation filter by default when searches begin on pages that have Land Registry tagged as the primary publisher.
Regular readers of the GDS blog may remember that we looked at scoped search earlier in the year. The conclusion of our user research was that this approach didn’t work in all situations and was frequently confusing users if it was turned on by default across the site, so we temporarily parked the idea.
This time round, we have so far enabled this feature for Land Registry, Environment Agency and Legal Aid Agency, to see how effective it is and help us understand the appropriate scenarios where we should use scoped search.
Land Registry scoped search
We deployed the scoped search feature for Land Registry related content on 1 September.
Before the automatic filter was implemented, not many users were filtering their results to the Land Registry organisation (between 50 and 150 searches a day). From 1 September onwards, there were up to 2,000 filtered searches a day.
The average rate of ‘refinements’, where users change their search term and try again, dropped from around 25% to 20% for Land Registry related searches. As we anticipated, the most visible improvement was in the generic terms where the user need depends on the topic or organisation. For example:
|
Search term
|
Refinements before automatic filtering
|
Refinements after automatic filtering
|
|---|---|---|
| forms | 45% | 35% |
| fees | 48% | 32% |
| offices | 64% | 43% |
| fee calculator | 34% | 13% |
| log in | 23% | 16% |
There was a clear change in the top search refinements, with users no longer having to add 'land registry' to their query to find what they need.
Before:
|
Search Term
|
Refined Keyword
|
|---|---|
| 1. fees | land registry fees |
| 2. forms | land registry forms |
| 3. landregistry | land registry |
| 4. ap1 | tr1 |
| 5. offices | land registry offices |
| 6. land registry | land registry |
| 7. tr1 | id1 |
| 8. contact | contact land registry |
| 9. tr1 | ap1 |
| 10. ap1 | id1 |
After:
|
Search Term
|
Refined Keyword
|
|---|---|
| 1. ap1 | tr1 |
| 2. tr1 | ap1 |
| 3. tr1 | id1 |
| 4. ap1 | id1 |
| 5. land registery | land registry |
| 6. landregistry | land registry |
| 7. id1 | tr1 |
| 8. land registry | ap1 |
| 9. fr1 | dl |
| 10. land registry | oc1 |
Environment Agency scoped search
We deployed the scoped search feature for Environment Agency related content on 8 October.
Many of their search terms are fairly specific and were already returning relevant results, so the automatic filtering had less impact than with the more generic terms we saw in the Land Registry transition. However, for some searches such as 'jobs', the filtered results now return relevant content ('Working for EA' in this case) rather than general results from the whole site.
Overall though, the automatic filtering hasn't made much difference to the refinement rates of around 36% for Environment Agency related searches.
|
Search Term
|
|---|
| 1. public register |
| 2. hazardous waste |
| 3. environment agency |
| 4. waste carrier |
| 5. esos |
| 6. flood map |
| 7. jobs |
| 8. public registers |
| 9. waste |
| 10. waste exemptions |
Legal Aid Agency scoped search
We also deployed the scoped search feature for Legal Aid Agency related content on 8 October.
Again many of their search terms are fairly specific, often for a particular form, so the filtering had less impact overall but it does improve results for more generic terms.
The rate of refinements on Legal Aid Agency searches improved a little, from an average of 37% to 35%. As with the Land Registry, some users were searching for generic terms like 'forms' and then refining it to 'legal aid forms', which is no longer necessary with the automatic filter in place.
|
Search Term
|
|---|
| 1. legal aid |
| 2. cw1 |
| 3. legal aid agency |
| 4. if1 |
| 5. forms |
| 6. family mediation |
| 7. mediation |
| 8. laa |
| 9. if2 |
| 10. af1 |
Next steps
We’re not expecting to deploy further instances of scoped search for a few months. Over the coming sprints we plan to look for further evidence of where scoped search will add most value and where it might not work so well, and we’ll summarise those conclusions on this blog.
2 comments
Comment by Lissa Allcock posted on
I suspect that you will want this also when you get more of the HMRC material onto the site, though it may be beneficial to include (tax specific?) Treasury results in searches begun on the HMRC page also. There is little more furstrating than trying to track down if a particular bit of tax related consultation is from HMRC or the Treasury.
Comment by Andrew Robertson posted on
Thanks for blogging about the results. I am surprised refinement rates haven't changed, though we have been working on SEO.
Is there any way of knowing if people spend less time on search results pages with scoped search? I just wonder if refinement rates aren't much different for Environment Agency content but there are fewer results to look through, so it is quicker for users to get to what they want?
For example, the top result for ‘public register’ is the same with and without the EA org filter. But without it, you also see ‘register to vote’ (2nd) and the home page of The Charity Commission (3rd), and lots of other registers which may seem less relevant to results for a search from the Environment Agency home page.
Thanks for letting the Environment Agency pilot scoped search.