The user test protocols were (almost) ready, the system was (almost) ready, I arrived (almost) on schedule in China and everything was (almost) perfect (not really ;)). Except the teeny weeny detail that this was the first user test for me in Chinese, i.e., I did not understand anything of what the user was saying.
I have performed user tests and user research in different countries (USA, India, Poland, Netherlands, Germany, etc.) and with users who are from all over the world (South Africa, Italy, Spain, Egypt, Saudi Arabia, Sweden, Denmark,etc….) but they spoke at least one of the languages I understand (English, Hindi, Punjabi or Dutch). So, to my utter joy (served with a good dose of humility and some other things), I learnt a lot from this usability test in China. And here are some reflections:
Know the system like the back of your hand
The user test was conducted in Chinese with a Chinese user interface. So, first off, it was a pleasant surprise that one can get quite far without understanding the language. What really helped here was that I trained in detail on the English interface and also, was familiar with the interface and the domain well from previous user tests for this product. My role in this test was to train the local team for user testing and provide guidance and I could easily recognize the use errors follow-up in almost all of the cases. There were some instances where I did not recognize the user message or dialog that came up. I would mark these and at the end of the follow-up interview would check with the facilitator regarding these.
Definitely need to work as a team
Working with an internal local team made so many things go easier, including arranging the logistics and recruitment which can be challenging in for international testing. See this article by Phil Dahnke and Toni Allen for recruitment in international user tests.
We worked as a team of one facilitator (Chinese speaking) and two note-takers (a Chinese speaker and I). Either the facilitator or the note-taker acted as my interpreter during the test. A professional interpreter could be an option too and this article by Jakob Nielsen covers the different options for international usability testing. But given the complexity of the domain and the product, hiring an interpreter would increase the training overhead further and the project did not allow for that time. Having said this, it took us some time, patience, and a few sessions to establish a rhythm and work as a team, building on each other’s strengths. So, I would advise to plan at least a couple of pilot sessions. As an example, the process of following up on use errors could be a bit of multiple-player ping-pong requiring patience from all of us (including the user!), e.g.,
Me To facilitator: What happened in task A in situation XYZ? (English)
Facilitator to the user: Translates question to Chinese (sometimes iterative phrasing of the question till the user understood)
User to the facilitator: Replies in Chinese
Facilitator/Note-taker to me: Translates in English
Me to the facilitator: Follow-up to follow-up — Okay, and why? …..
(Repeat iteratively till satisfied or tired)
Let go and Breathe
As a usability engineer, you repeatedly learn to ‘pick your battles’ and prioritize. But sitting in a room where you are the ‘expert’ on usability testing but also the only person who doesn’t understand the language at all took ‘letting go’ to a whole new level. So, I would prioritize on things that were absolutely important and discuss with the team after the session, e.g., at what point should we assist the user; and let go of the things that were a matter of taste, e.g. making ‘live’ notes vs. reviewing video later and taking notes (hint: I usually prefer to take notes ‘live’ as far as possible even if there’s video recording).
Some practical matters
So, make sure to set aside time for yourself to relax and recharge for the successive days of testing.
Also, in case you have dietary restrictions, e.g., are a vegetarian like me — ask a colleague to write down your requirement in the local language. Make sure to read about the sites and apps that do not work in China, think Google family and Whatsapp, etc. , to avoid getting literally lost in translation.
- Learn the system and the context of use well before the test.
- As far as possible, work with a local team and plan more than one pilot and for changes to the protocol after the pilot sessions.
- Debrief the team (including yourself) and the users in advance for the need for patience with the translations.
- Practice humility and the art of letting go and reserve time to relax and recharge for yourself.
- Prepare well for the ‘trip’ to the foreign (or exotic) land you are heading towards.
Have you had fun experiences while user testing locally or internationally, would love to hear from you!