How to design for mobile and head-mounted displays using a language learning application as an example
I failed to learn Dutch
I remember the last time I tried to learn to speak Dutch. To learn that language I could choose between books, articles and videos with the best practices and apps. They promised me to learn Dutch in a short amount of time. Of course, I wanted quick results, like we all want, so I tried them all. I was very motivated at the beginning and successful, but after a certain time I lost my motivation. My expectations were too high and I could not meet them. I was so focused on the big picture and the final result instead of focusing on the learning itself.
I am not the only one who faces this issue. We have so many possibilities of mastering a new skill like a language but maybe we should decide on how comfortable we are with the process of learning rather than on the end-result.
Life is a journey, not a destination
Learning is hard and costs a lot of cognitive energy. Technology should support learning and make it as pleasurable as possible. As a designer I wanted to create something which allows us to enjoy the process of learning. We should stop thinking of learning and start playing instead. I wanted to create a language learning experience that uses recent technology which allows us to learn with. This article is about Lalinga. A perfect use case for immersive learning in Mixed Reality. This is how I would like to learn a language, considering all the circumstances we face in a world of information overload. All examples are shown for learning German, dedicated to my friends who moved from Iran to Germany.
Lalinga focuses purely on your learning process. It will not track your results and never remind you with push notifications. It focuses on learning vocabulary, not grammar. If you enjoy learning itself, you will progress in your language skill. It is a valid part of your language learning toolbox.
How Mixed Reality can enhance language learning
Currently, one of our society’s biggest problem is the gap between the digital and the physical world. Whenever we want to receive information, we bow our heads down looking into the square screen of our smartphones.
Mixed Reality using head-mounted displays could change this by placing digital content in our real world environment. It merges the real with the digital world and it has the potential to help us learn new things.
It enhances immersion, which is supportive for language learning. Immersion helps the learner to think in the new language rather than doing translations. And to add something on top, it is fun to play with.
Unfortunately, this technology is currently not available for the masses, so a lot of the concepts of Lalinga have to be tested by using existing mobile devices. However, it will allow a smooth transition from mobile to head-mounted displays using Mixed Reality. The use of head-mounted displays is necessary to guarantee the most convenient user experience which would not be possible with mobile Augmented Reality.
Psychology of learning
Conceptualizing a language learning application has to be based on psychological foundations. During the process of learning our brain is basically connecting newly acquired knowledge with existing one. Learning requires cognitive load and therefore needs motivation. Lalinga is based on exploring and testing. It balances challenges and positive emotions. Lalinga entertains the learner but challenges him in the right moment. It focuses on four main concepts.
Learning by itself is not a pleasant task and requires a bunch of brain power. Furthermore, language learning in schools is based on preparing for tests. Researchers found that taking tests results in better learning than studying for a test. It’s basically the trail and error method that is common in games. People are trying to complete a quest to move to the next level. Their engagement is much higher, they overcome failures and have positive emotions.
There is no linear stack of words or tasks which have to be completed. The learner can choose the word he wants to learn and explores the language. This is based on free will and results in a higher motivation.
Immersive learning happens when the learner lives in the country where the language is spoken. He is surrounded by the language which forces him to think in the language he wants to learn. He is not translating into his mother tongue and that’s what Lalinga wants to achieve as well. The connection to the word is made with illustrations and sounds.
Testing the knowledge
Connecting newly acquired information with existing knowledge requires continuous repetitions. If this information is continuously repeated, it won’t get lost so quickly. A reason for this is deeper encoding of the information. This effect can be enhanced if encoding happens with multiple senses. It is important that encoding is meaningful. It doesn’t make sense just to repeat a word multiple times, but it is better to put it in a new context, e.g. building a sentence with it or illustrating it. If more of these contexts exist, more connections are made in the brain. Ebbinghaus explains this in his forgetting curve which displays that repetition intervals should be short at the beginning, but can be longer later on. In this way more information will stay in the long-term memory of our brain.
Learning by exploration
Tapping on objects and diving deeper into them is explorative interaction. It is similar to a video game where players have to explore the world to finish a quest. Almost every object in Lalinga can be tapped and some of them can be explored more in detail. In a bathroom the learner can choose all kinds of furniture. In this case a cabinet can be selected and opened. It will expose it’s content like toothbrush, perfume and soap.
And what about untouchable words?
There are certain words which can not be selected visually. We are talking here about verbs and adjectives which can not be expressed as a single object but it’s possible to explain them visually. Any kind of verb can be explained using an animation. Many adjectives can be explained by using comparisons. To connect them to the explorative world, learners can access use cases for the object they selected. These use cases are shown after pressing the chain button. If “bed“ would be the displayed word, pressing on the chain button the learner will get words like “sleeping“ and “tired“. All these words are connected to “bed“ and easier to remember. After showing them, they are listed in the stack.
Testing the knowledge
Successful learning is based on memorizing the knowledge and recalling it. In other words, the learner need to do vocabulary tests. If tests are passed successfully, the learner can unlock further sessions. There are three types of tests.
The most obvious difference between a mobile and a head mounted display application is the interface and interaction. Luckily, 3D mobile applications like games can be redesigned for a head mounted display, because interaction happens with the 3D world. The basic concept of Lalinga is the interaction with the 3D illustrations.
What needs to adapt is the interface, which displays the word and allows navigating. These are the key differences between mobile and head-mounted displays:
Placement of content
Sequentially: Content is placed as a “sequence of slides” on a rectangle screen. Due to limited space only one illustration is displayed.
Spatially: Content is placed around the user, taking into account the comfortable ranges. This allows displaying multiple illustrations.
The learner navigates by using swipe gestures and buttons.
There are no standard gestures yet. The best way to navigate in the application is using buttons.
Haptic feedback through touching a surface and using vibration.
Haptic feeling can be created when placing interface elements on real surfaces.
In most cases every task (except pinch to zoom) can be accomplished with one finger.
Tasks can be accomplished with one hand. Rotating and grabbing the illustration however requires both hands.
When translation is needed
Sometimes the learner wants to be sure that the word he has in mind is the right one. To check that, the learner can tab on the flag button which will translate the word into his mother tongue.
Further ideas for the real world — virtual world connection
One of the benefits of Lalinga is to combine the strengths of the digital and the real world. New technologies like Machine Learning help to capture content from the real world. Furthermore, Mixed Reality helps to place digital content there.
Capturing real objects
When learning languages, the learner wants to know the names of the objects in his environment. In Lalinga he can capture them by using image recognition and link them to an illustrative object. In the next step this object can be linked to corresponding verbs or adjectives.
Vocabulary test using handwriting
Especially for me it is easier to remember words when writing them by hand instead of typing them. Again image recognition is able to recognize handwriting, which can be very handy when doing vocabulary tests.
With Lalinga I am trying to use modern technology to create a new language learning experience. The focus stays on the process rather than the result. This does not mean that the result is not less important. It’s like writing a thesis. Sometimes it helps just to start writing instead of worrying how much work is left when looking at the big picture.
Up to now this project is only a concept. Building this for head-mounted displays requires a lot of challenges due to the current hardware. While the Magic Leap One or Microsoft Hololens are great devices they still have issues like a narrow field of view, inaccurate occlusion between hands and 3D objects and tracking issues. All these technical limitations have a big impact on user experience. Head-mounted displays are not yet available for consumers. There are no UX standards yet, and we still don’t know if people will use these devices like they use desktop computers or smartphones. And, I still don’t know if people would like to learn languages this way.
To validate this assumption, my team and I created a landing page which describes the Mixed Reality version of Lalinga. At the end, visitors can sign up for a beta and communicate ideas. This will help us to evaluate if it is worth building this project. Let’s see what will happen.