User interviews can be a useful method for getting more information about real experiences you can’t observe directly. But it’s a common mistake to simply ask users a series of questions about habitual behaviour and preferences.
A better approach
In order get a more realistic view of what people actually do and why they do it, you need to get more specific. I use cognitive interviews for this.
I’ve taught quite a few people at Skyscanner this approach to user interviews. I thought I’d try to capture the approach in a post, in case it’s useful to anyone else.
In this example, let’s pretend I want to find out about how trip planning takes place for a specific type of trip. Rather than find people who plan these trips and ask them what they usually do, instead I find people who have just planned a trip like that and find out what they actually did.
Daisy went to Italy
Let’s say I’m interviewing Daisy, who has just booked a holiday to Italy and I want to understand how she planned the trip. I’ve quoted Daisy in this post. I made those quotes up just to illustrate. She’s not real.
No script to sign off
Instead of writing an interview script and having that agreed with stakeholders, I will get them to agree on the approach and ask about anything they want to make sure I get answers about. The reason for this is that instead of asking scripted questions, you instead ask the participant to guide you through a past event.
The cognitive interview
I don’t trust the answers people give about general habits because I’ve often uncovered them to be inaccurate. Even when talking about a specific event, some of the initial detail people give you can be inaccurate, or you don’t quite understand it correctly.
Instead I take participants through a simplification of the cognitive interview technique. This is the same technique used in Jobs to be Done interviews. In it you’re trying to understand a story as though you were there when it all happened. The approach helps you to understand, but also helps the participant to recall events more accurately.
You should still accept that some of the understanding you come away with will be inaccurate. There will always be inaccuracies, that’s why direct observation is better. But the important things should form as patterns when you speak to several people and this should give you confidence that the important findings are real.
Begin with reinstatement
I want Daisy to open up her memory of the time when she was planning her trip, so I will ask her to recall when she made the actual booking. For your own interview, ask them about the moment were they made the final commitment, decision or when the event came to an end.
I’ll ask Daisy about details which might seem irrelevant, like where she was sitting, down to the specific room and also where in that room she was. Who was with her? What was the weather like that day? How did she feel after finishing the booking? etc. Most of these details I ask for will be a generated from her initial answers.
Once Daisy has opened up her recollection of the event of booking, she likely already remembers details she wouldn’t otherwise have thought of without having done this. The details I have asked her overlap with other details she might otherwise not have remembered and those details are connected to others she can now recall more easily.
Then go back and start from the beginning
But now I go back to the beginning of the story. I try to find the first awareness she had that she wanted to take this trip.
Daisy: “It had been raining for weeks in London and I was getting really depressed. I’m addicted to Instagram and was on the bus to work. I was looking through Instagram when I saw a picture of this awesome looking restaurant built into a cliff in Italy. I thought, screw this shitty English weather I’m going to go eat in that restaurant”.
From here, it’s a case of directing the conversation so you understand the sequence of events. Try to refrain from jumping straight to the bits you’re most interested in. Instead, if you sweat the details of the story in sequence then you often find the bit you’re interested in has been recalled with greater accuracy than if you’d jumped straight to it.
Me: “So you’re on the bus and you see this amazing restaurant. Did you keep looking through Instagram or did you do something else immediately after seeing it?”
It makes no difference to me whether she spent 5 more minutes on Instagram, but going through the events in sequence with detail will allow her to remember and allow me to understand. You need to use judgement here of course, otherwise you could be talking all day.
All you do from here is walk through the story with her, so you know what she did and what her motivations were for doing those things.
Three techniques to explain and use
I’m going to dispense with walking you through the introduction to an interview and the formalities of the session, such as explaining the format, gaining consent, seeking honesty etc. But there are three specific things I’ll do during our conversation which it’s best to make Daisy aware of before we begin the interview.
I might interrupt you quite a bit and ask you to tell me more about something you’re not talking about
Me: “During our conversation you might find that I’m interrupting you and bringing you back to a different part of the story. The reason I do this is because I want to come away with an understanding of everything that happened and in what order it all took place. I sometimes needs to understand something in more detail when you’ve just mentioned it off-hand”
Daisy is likely to give me one version of events initially, but some of her comments will uncover things, she has forgotten to mention and these details often make the initial recollection untrue or they are simply important to understanding the full picture. For example…
Daisy: “I went to Skyscanner first, I always do that” but later on she mentions “I was looking for the flight that landed at 8pm, the one I saw on the Virgin Holidays website”
I’m going to stop her and ask about that Virgin Holidays bit because she hadn’t mentioned it previously. I need to know where it lives on the timeline and how it came to be.
It might seem like I’m criticising your decisions
Me: “From time-to-time I might offer an alternative way you could have approached something and ask you what made the way you did it suit you better than the method I propose. This might feel like criticism, but it’s not meant to. I just want to understand why you the way you did it was more suitable. Sometimes it might be the case that you hadn’t thought about doing it like that, if so then that’s useful to know also”
Daisy tells me she believes better deals on flights might exist on other websites from time-to-time. She also tells me that Skyscanner was the only site she used to search for flights tickets after she’d initially searched for package holidays. I need to understand her rationale for only using one site for flights rather than checking more.
Me: “Why when you believe better deals on flights might exist, do you not try to find those better deals?”
Daisy: “Sure, it might be possible to save a bit of money, if I spend more time searching for all the different sites out there, but there are hundreds of them and I’ve done this in the past. Yeah I guess there was one site I saw last time I did this and it was showing a price, it was about £20 cheaper on a thousand pound ticket. But I don’t think I even booked it. I’ve kinda learned that Skyscanner is searching lots of the websites I’d think to check anyway. Like Expedia for example. So, much of the time I spend is wasted time and the difference isn’t going to be big anyway.”
You’re essentially asking them why they did things the way they did. By proposing an alternative approach she could have used, I learn the merits of the approach against another. You get to learn what people most value when you ask this and in what order they value them. In this approach it sounds like Daisy is telling me she doesn’t think the payback of doing more research on prices will be worth the effort. But she has taken a lot of words saying it and I need to make sure that what I’m hearing is what she is saying.
I’m going to tell you what I think I’m hearing
Me: “Every so often I will stop the conversation to make sure what I’m hearing is accurate. I’ll sum up what I think I’ve heard and ask if that’s correct. Unless I do this I might go away misunderstanding the conversation we had. I’d really like you to tell me if what I think I’m hearing isn’t what you’re telling me, even if it’s slightly different or there’s more to add”
You’ll no doubt be aware that putting words in participants’ mouths is bad research practice. But you are going to walk away from that conversation with a view of what you thought you heard and you need to be able to check your interpretation. Daisy spent a long time explaining the value of only searching on Skyscanner for her flights. I think I’ve got a good understanding, but I need to know I’m hearing it right.
I’ll pause the conversation and ask her to check my understanding. In this, I will recount what I believe I’m hearing
Me: “This is one of those times when I want to check I’m going away with the right understanding. What I think I’m hearing is that you know it’s possible that a better deal exists on flights, but the difference in price isn’t going to be worth the time it would take you to find it?”
Daisy: “Yes it’s that, but also I often don’t know those other sites like I know Skyscanner”
I now come away more confident that she believes the potential reward is not worth the time and effort, but also there is a secondary element of distrust of sites she hasn’t heard of.
Keep going until you get to the end
Using this approach, you’ll need to keep your eyes on the clock and use judgement to decide how much depth you need to go into for each step of what you heard. You need to get a balance of hearing the story from beginning to end and also getting necessary depth. It might be the case that you’re only interested in a specific part of the story.
Remember to relax
When you’re doing research for the first time there is a tendency to be worried that you’re not doing it right or are going to make mistakes. My advice is to relax and accept that you will indeed make mistakes. Worrying about mistakes will not stop them happening. In fact you’ll be more likely to make them.
Realise that everyone makes mistakes when running research. These mistakes generally don’t nullify your findings. Instead they tend to compromise the reliability of some of the things you hear. If you’re aware of the mistake then you’re aware of what might be unreliable. Just concentrate on allowing the participant to tell their story and only step in for clarification and directing the discussion to the areas you need more understanding of.
The best conversations you have with users are generally the more relaxed ones because your participant is more comfortable, open and honest.