- Pre (Plan)
The most critical part of the whole Usability Testing practice especially when setting it up for the first time.
This is where we initially spent close to 70–75% time of 1 complete cycle (Plan — Conduct — Analysis & Synthesis and Review) of Usability Testing.
The more time you spend initially, the less time you spend on consequent sessions/rounds. So as an example if we spent a month on 1st cycle, subsequent cycles we spent 7–10 days as we had robust processes and framework in place. That’s a 70% improvement right there!
The plan starts with figuring out WHY we are doing Usability Testing and WHAT we wish to achieve out of it. Without this in place, all other things will not matter as mostly everything is depended on it.
Its’ easy and tempting to scrape through this but I highly suggest otherwise. This is the foundation step and if this is shaky one can never really know why the things we build on top is not working. One will make the mistake of blaming other things instead of questioning that may be the core objective itself was not clear.
Depending on what type (exploratory, descriptive, evaluative) of research is required to be done, the Goals of the Research will differ. Usability Testing is more of evaluative research method.
Once the Goal is identified, the Scope of what all needs to be evaluated with the user needs to be defined. However, at times the Goal of Research can also be derived from the Scope.
A scope is nothing but what all needs to be evaluated in the concept/prototype/product. The bigger your scope, the higher the number of diverse users you would probably have to recruit as not every part of the product will be applicable to all users. In this case, even the Screener (more on this below) would have to be more expansive which will allow different types of user to be recruited.
Conversely, the shorter the scope the less diverse set of users you need but your Screener needs to be that narrow to find the right users.
Any product is designed and built to be used to achieve a task or to get a job done. Thinking through what Tasks/Jobs your product is designed for is incredibly rewarding and at times terribly discomforting (when it’s not clear what the product is actually meant for!).
Listing down primary and secondary tasks/jobs which your product accomplishes is key. Keep in mind that the tasks/jobs should be from the user’s view.
It’s also essential to list down all assumptions/hypothesis which was taken so far in the product development journey.
This is also a good exercise in product-market fit. 🙂
1.4 Target audience/users
Once the Tasks/Jobs are clear, finding your Target audience/users becomes comparatively easier.
In our case, we had a set of fairly clear user segments with expected overlaps in some tasks — Person A does Task Y and also Task Z and also Person A and Person B both do Task Y.
Keep in mind that user segment is what we have formed and not what the user may relate to. So while recruiting if you ask the user whether he/she belongs to a particular segment they will not be able to answer reliably and hence based on the tasks they do, bucket them into relevant user segments.
Finding your target audience/user and then finding the right set of users within the audience are linked but in a way a different separate set of activities.
A screener is one of the most critical pieces to get the desired outcome from your Usability Testing session. To take an analogy —
If you are having a fever, will you go to a General Physician or an Orthopedic?
Usability testing is in a way a diagnostic tool and hence who is diagnosing can make all the difference.
Not every user is the right candidate for Usability Testing. A Screener helps you filter the right candidates from the not-so-right ones or even plain wrong ones.
Usability Testing is a fairly intensive activity in terms of effort, time and money involved. Hence finding the right users is absolutely important and should not be rushed. A user who is not right not only results in waste of resources but also can skew the usability test results. This is where a Screener plays such an important role.
Screeners should be as crisp as possible and ideally within a single page (A4). It should clearly contain in which all cases to drop the prospective user and move on to the next one. Only if all conditions are met, as mentioned in the Screener should the user be considered. Sometimes you may need to narrow or broaden based on how the initial set of calls are taking place.
If everyone is willing to come, then may be your Screener needs tightening and on the reverse if no one meets the criteria then it may need some loosening. Doing it a few rounds will give you a hang of it.
In our case, we knew our User segmentation and the kind of tasks they do and mapped those to the Areas of the product we wanted to test. So while calling participants we had a couple of questions to ensure they are the right audience and would be able to help us. From a process perspective, we made the Screener in such a way(simple yet comprehensive) so that anyone can use it and find the right users.
Cant stress enough — the right user profile based on your product and what you want to test is super important. The wrong user profile can give results which can be misleading and change your product in ways which can affect majority of right people, in all the wrong ways.
Some Usability Tests are simply task based. You invite a user, assign certain tasks, watch how they are doing it and measure against your set parameters.
Others are more about starting a conversation and creating a conducive “as close to an actual world” environment for the user, understanding their story and then watching them do the tasks they have in mind.
In our case, we started with a Task-based approach but found that the User’s either couldn’t connect with it or they felt it more of a test and were not natural at doing those tasks in a similar way they would be in their own environment. Both of these circumstances significantly reduce the credibility of the Insights and hence cannot be relied upon.
Thereafter we took a more conversational based approach wherein we first understood what kind of tasks/jobs the user does in their daily lives and how our product fits into it. We did have tasks but those tasks were chosen by the user themselves and not assigned by us explicitly. To give you an example, let’s say the user does Task A, B, C in sequence every morning and using our product. So we allow them to do the same tasks in front of us.
We all use products in a context and it is what drives how we use it. And hence understanding of this helps tremendously. Spend sufficient time to understand user’s story.
We did have a field-guide (covered more in the “During” part) in mind and we gently traversed through it to ensure we cover all the points we wanted to.
One thing to emphasize is that not all tasks/jobs of the scope will be applicable to all users and hence its necessary to first understand what all tasks/jobs are applicable to the user and then evaluative only those parts of the product.
Having said that there may be certain new features which you feel would help the user even though the task associated with the feature is not what the user is currently doing, in such cases no harm pitching in and see how the user reacts. Maybe the user is not doing the task because of the current way of doing it and your new way may make it the user likely to do the task.
1.7 Recruitment & Engagement scripts
To ensure everyone who calls any User says the same thing and more importantly no one should struggle or wonder what and how to engage with prospect user, we made scripts for the following –
Call script — How should we reach out to the customer, how do we introduce, what’s in it for them?
Email script— Once they have agreed to participate in Usability Testing, we send them a formal email/meeting invite to confirm details like Date, Time, Venue, Directions, etc
SMS/WhatsApp — Same purpose as Email, but here it’s more concise. We started with SMS but then later moved to WhatsApp due to features such as Sharing Location, tracking whether the message has been delivered and seen. Another reason was that interestingly not everyone is active on email but everyone is active on their mobile phones — SMS/WhatsApp.
Both Email and WhatsApp message we sent once the appointment is confirmed, a day before the scheduled session and once the session has been completed.
Email and WhatsApp serve the following significant purposes —
- Meeting requests via email help us to better organise our schedule as it reflects on our calendar and helps to avoid overlaps of scheduling 2 users at the same time.
- It lends credibility to the User that we genuinely need their inputs
- It helps to avoid drop-offs and any last minute changes can be handled.
- Post-session we send a Thank You note via Email and WhatsApp to show our gratitude for the user’s inputs and their time spent helping us.
One another significant objective of the above scripts and process is for reducing drop-offs — We had protocols in place once appointments are confirmed, reminders are sent a day before.
Even though users have confirmed their participation, its natural for unforeseen situations to arise or even plain forgetfulness or waning interest. This is all the more important when you are expecting Users to come to a location instead of you visiting them.
Hence it’s always good to follow-up a confirmed appointment (on call) with an email/calendar invite and also an SMS/WhatsApp. This should mention time, date and venue for the session (adding the location is even better). And then a day before, send a reminder showing your interest and enthusiasm for the session tomorrow.
These small things go a long way. If nothing else, it leaves the user with a good experience with you/your brand and even though the user may not turn up for whatever reason, in future the chances of user entertaining and participating definitely increases.
Apart from offering them food and beverages, which we strictly follow, incentives (non-cash) is the least we can do for our users who are helping us improve our product.
In our case, we had given a bag of goodies along with a certificate of participation to all users who came for Usability Testing sessions.
Apart from the product, User is also experiencing the way we design the whole Usability Testing experience for them and hence incentives are also seen as a significant part of adding delight to that experience.
Always keep extra incentives than the schedule no. of participants as at times more than 1 person may come together for a Usability Test. Being over-prepared is always good.
There is one question which I have stumbled upon often, whether to inform the user at the screener and recruitment stage that an incentive exists.
I have two opposing views –
- Incentives may influence an otherwise reluctant user to come for Usability Testing. The risk here is that those users who are not fit for our product may somehow try to game the process only to get the incentives.
- When you do not inform Users of any incentives until the last moment when you actually hand it over, their unexpected happiness of receiving it is priceless. These small things go a long way for a long-term healthy association with the user. The risk here is without mentioning incentives users may not turn up.
Try both and see which one is getting you the participants.
Regardless of whether you inform or not, Incentives is must and keep in mind it forms part of the experience the user has with your company/product. All touchpoints matter.
1.9 Scheduling Templates
While engagement scripts took care of one problem, one other problem still remained that is to track whether the communication has happened and at the right time or not.
We made the following template on Wiki (internal) to help manage user’s schedule and also to ensure reminders are sent at assigned intervals—
Many a time, I heard from participants in advance for rescheduling or cancelling the session. This helps to arrange for other users in time.
Also sending them reminders also show that you genuinely care for their feedback and want them to come. Most of the users felt appreciated by this regular follow-ups and it made them feel important.
1.10 Room Layout & Requirements
When including this (as part of the process), I found this trivial and felt we are overwhelming the various teams doing this.
But in hindsight, details like this matter to ensure a consistent experience for all our teams and for all our users and to ensure operationally all things are taken care of.
This is even more important when you are conducting Usability Testing outside your regular Lab (where even at the last minute things can be taken care of). Say if you are doing at a hotel or another office location or at someone else’s place it’s important to convey beforehand what all you need so that it can be taken care of.
Depending on the nature of your product and your users this can potentially be different.
Room checklist and layout —
Regarding room layout, it may seem like a simple arrangement of a room, but there was a rationale in having this.
Moderator is slightly behind the User and is neither facing the user nor facing the system. This is so that moderator can capture both what’s happening on the screen and also observe the supercritical facial reactions and body language of the user.
The observer is much further behind as two people observing tends to make the participants conscious. At the same time in viewing distance of what’s happening on the screen.
In some locations, we had the option of an extra parallel monitor wherein the observer can observe.
For our in-house Usability Lab, we had a set-up a different Observer room wherein any number of people can join in and view (live — audio and video) these sessions. The software (Morae) we used to record, edit, manage these Usability Testing sessions also provided the option of viewing the live session remotely from any location.
Preparing this was super fun and super gruelling at the same time. I started with 1 page and with a few iterations, it went to 16. Yes, 16 pages of checklists but at different points of time and interval.
While preparing the Checklists, we made a list of everything that can go wrong and then set processes in order to prevent it from happening. however, things may still go wrong statistically speaking but we would be better prepared for it and act accordingly. This governed everything we did. Better to be prepared for any eventuality even if the likelihood of it happening was close to zero.
A lot of things just like the room requirements (here I am laying it separately but it forms a part of the checklist) is painfully obvious and yet its super easy to forget this and it can seriously derail the operations or lead to even cancellation of some sessions.
For me personally, I still use some parts of this checklists to ensure nothing has been missed out and things go as planned. Human memory is not very good at holding a lot of things at the same time.
A TOC of this document —
I would probably upload a template of this which anyone can use. Sharing all the details would make it an extra long post here.
I want to briefly touch upon SPOC — Single Point of Contact.
In the initial 1–2 rounds I was the SPOC managing and coordinating all the various teams at different locations.
SPOC is basically to ensure the whole operations happen smoothly and any contingency is handled for. To get a sense of community and shared purpose we created a WhatsApp group where people shared their Insights and/or if they had any product related doubt which the user has asked, they can immediately get a response from any of the members. When you see other people sharing, one tends to do it themselves. When sessions are happening, this group becomes very active and engaging.
Since I always yearn to go and meet customers, it was critical to set a process so that I can be freed up and at the same time anyone can take over from me. Hence a checklist for SPOC also. 🙂
Even the travel checklist was very crucial as we do not want to get into a situation where for whatever reason one reaches the airport and not able to fly for want of certain ID cards requirement. Not everyone travels often and hence was important to put it down explicitly.
One of the most important deliverables from a process perspective is this document which empowers those on the field to conduct a study as effectively and efficiently as possible. There are different names to this document, some refer to as the Script, some Protocol and some Field-guide.
If this is not well thought through, the outcome can be severely compromised. Field-guide also helps in keeping the conversation on track from our perspective and helps meet the objective in its entirety.
We had divided the Field-guide into 4 sections —
- Introduction — opening script
This section contains Why the user is here and what do we intend to achieve out of it. This is where all legal aspects like NDA and consent for recording come into play.
2. Profile of the user, their environment and tasks/jobs
This is where we wanted to understand the user’s story a little bit better than from the info we already had. A profile can be on a personal level and also from the perspective of their environment/situation and what kind of tasks/jobs they will likely to use/presently using our product for.
If we have done our process well so far, we would have a good idea of what all scenarios, the particular user segment, to which the user belongs, will encounter and we use our field-guide as a reference point to help traverse the user through the areas we want to evaluate or test.
It’s important to mention that here its more of cues than long sentences which is not feasible to read verbatim when you are in the middle of a conversation with a user.
For each sub-section, we had multiple choice answer format to record certain usability aspects we wanted to evaluate. The key thing here was how we can record feedback as quickly as possible. If it takes time then we might miss certain critical aspects of the user’s experience.
We had included questions like —
Where did the user get stuck and how they recovered (if they did)?
Happy, confusing, frustrating moments for the users?
Was the user able to find what they were looking for?
Where all did the user need help to accomplish the task?
There will inevitably be times when the user is stuck and wants to know how they can move forward. In such cases instead of helping immediately, ask questions like “what do you think”, “what should happen now”, “where do you think this option should exist”? Such questions help us uncover and build user’s mental models which are so valuable for the whole product design and development team.
The questions we ask each type of user for a particular type of task was standardised, this helped us to better assess and find patterns across our sample size.
Here we ask the user whether they have any questions and we can ask any follow-up questions we have.
We also included certain NPS questions whether they would recommend this product or would want their peers to experience the product as they did.
We also asked what were their top 1–2 highlights of our product (and which they will share with their peers/friends) and top 1–2 shortcomings. The highlights are usually positive and can form “stories” from a marketing standpoint.
It’s imperative and important to plan a structured onboarding for those members who have not done any Usability Test before and especially for those who are from other departments (non-design).
Things covered in such training —
What is UX Research?
Why it is important?
How is it done?
Apart from this, we run them through all the checklists and also how to conduct a Usability Testing as a Moderator and an Observer.
Time permitting, we do an actual role play and simulate a Usability Testing as if it was happening live with a customer.
For some reason, if the Training cannot be organised, the Checklists and the Field-guide are enough for anyone to conduct end-to-end Usability Testing.
Here I was thankful to get a mandate at the executive level that everyone from PM (to begin with) must go and do Usability Testing, still, we started small and included people in parts.
Once a new set of people have come on board and they have conducted a few sessions, going forward we further tagged another set of new people with these existing people who are well versed with the whole process and the cycle continues. This helps in multiple ways —
- User’s learning gets reinforced and sharpen by practice. Teaching others is the best form of learning.
- When existing users have the ownership of training new users, they become invested in the process.
- Instead of a particular set of people spreading knowledge, influencing and evangelising we have democratised the whole process and creating a kind of network effect.
Adding other criteria for forming teams was individual traits of each person. Some people are more introvert/extrovert than others and hence a balance was necessary to get the desired outcome.
Then language proficiency was another important aspect as our customer base is very diverse and users tend to open up and speak freely in their own language. This was reaffirmed multiple times when the observer (or even moderator) was not proficient and couldn’t grasp much of the conversation which the moderator and the user had. Hence we tried as much as possible to form teams in such a way that an at least one of the team member knew the local language.
Yet another aspect was not to send the same person repeatedly to a fixed place and instead rotate it to get an experience of the diversity of users, diversity of their environment, diversity of their work and therefore diversity of their tasks/jobs.
Since we have offices in almost all Metros, Tier-1 and Tier-2 cities we prefer to do it at our own offices for operational and logistic reasons. For places where we don’t have one, we reached out to our existing Partners or known associates to help us with a place for 2 days.
In both cases, regular communication ensures good coordination. Remember the Room Layout? The individual teams conveyed all their requirements well in advance. Even here the checklist helped.
And invariably when drop-offs happen, its the local people who helped unconditionally to get replacement users at the last moment. So keeping them in the loop is very helpful. When people feel belonged, the end outcome is far greater and better.
One thing which we learnt was that especially in big cities that travel time can actually affect whether the person turns up or not. A lot of people asked where is the venue before confirming their participation. So a good thumb rule is to find a central location from a user’s perspective.
One can always hire a place(hotels, co-working space, etc) and conduct these sessions if need be.
Whether you are doing it at your own office or some one’s else or in a hired public place, remember that for the user they are interacting with 1 company/product and hence the experience you design and provide for should reflect what kind of experience you want the user to leave with.