John Ploughman is the Content Design Manager at the Driver and Vehicle Standards Agency.
Guerrilla testing your content can help you to challenge the assumptions that subject matter experts make. And it makes you recognise your own assumptions as a content designer.
At the GOV.UK content conference beta, I ran a session to tell the content community about some guerrilla testing I’ve done. So here’s a brief overview of how I used it and what I learned.
What guerrilla testing is
Guerrilla user testing is a low cost method of user testing. The term ‘guerrilla’ refers to its ‘out in the wild’ style, eg at a cafe, library, train station or car park. It’s a quick way of checking that your design actually works with users.
There’s a much more detailed explanation of guerrilla testing in the service manual.
Using evidence to overcome reluctance to change
The Driver and Vehicle Standards Agency (DVSA) runs the vehicle recall scheme. If your vehicle is recalled for a safety reason, you’ll usually be sent a letter by the manufacturer telling you:
- why it’s being recalled
- what you need to do next
- who you should contact
The manufacturer uses a template letter published by DVSA on GOV.UK.
We thought that the letter could be simpler, clearer and faster for users, so we designed a new version.
The subject matter experts were reluctant to change something that’s proven to work, so I decided to do some guerrilla testing to find out what real users thought.
Planning the testing
I joined forces with Sheilah Borsuk from our research team to organise the guerrilla testing. It was the first time either of us had done any.
We decided to A/B test the 2 letters with vehicle owners to find out if:
- the letters were easy to understand
- the vehicle owners knew what to do next
- they know what could happen if they didn’t do what they're asked
As we wanted vehicle owners, we did the testing at a car park in Nottingham city centre. We also filmed it so that we had something to show the subject matter experts.
So we set off, armed with our guerrilla research testing kit.
Testing the 2 letters to find the best one
It was really important for us to:
- get permission from everyone who took part
- make it clear what we were doing and who we were
- reassure people that their information would stay confidential
As we were filming people, we also made sure we got signed consent to do that.
It really helped that we were able to offer a small ‘thank you’ cupcake in return for people’s time (plus holding a tray of cakes helps to get people’s attention!).
When people agreed to take part, we asked them to read both versions of the letter, and then asked them a series of questions. We took notes and filmed the responses.
After just under 2 hours, we’d managed to get 8 people who were happy to be filmed, and a few more that were happy to answer the questions without being filmed.
Show the thing
Back in the office, I edited the video clips together, and prepared a few charts to summarise the results. The brief version: 75% of users preferred letter B, and everyone we asked understood it.
I arranged a meeting with the subject matter experts to show them the videos. Their reaction? Stunned silence.
Nobody wants to make it hard for people to understand something. And as soon as they saw the evidence that the current letter was doing that, they wanted to change it.
It’s fair to say that I could have just showed them charts, but it wouldn’t have been as powerful. It was seeing and hearing the public say they were struggling that made the subject matter experts listen.
What I learnt
I learnt some really important things, both about the content I was working on, and the actual process of guerrilla testing.
While my redesigned letter tested better, users did comment that it made them feel frightened or scared. That’s something we can improve on.
With the testing itself, I learnt that:
- it’s really important to get the questions right before you start, especially making sure you’re not asking leading questions
- video is one of the most powerful ways of challenging assumptions that I’ve ever seen
- it’s important to work with your user researchers, but it’s not something that only they can do - after all, user research is a team sport
Will you give it a go?
At the end of the session in the conference, I asked if people felt more able to go out and do something like this themselves. Most said yes.
So I hope that this inspires you to give it a go. I’d really like to hear about your experiences, so do let me know in the comments if you’ve done any, or will consider giving it a try now.
2 comments
Comment by Huw Pritchard posted on
Fascinating John - can you zoom up the letters a bit and tell us what questions you asked. About how long did it take to get each successful response?
Comment by Michael posted on
Great post John, thank you. Q1 did you film people reading the letters, or expressing their thoughts afterwards? Or both?
q2 you say at a car park, did you do the testing outdoors?
I have done some guerilla testing, sort of. The subjects were staff from my org but not same dept as the content topic. not absolutely ideal I know. Planning more in the wild soon.