For Hollywood, AI is a somewhat nuanced boogie-man. Movies like Terminator and She propose a generally intelligent agent capable of being more human than human, at least in certain circumstances. The reality of AI thankfully falls far short. At least so far. The mundanity of current human/AI interactions doesn’t diminish the need for the engineers and designers of these systems to give some thought to human interactions. (Note: I’m going to lump “machine learning” and “AI” together under the general umbrella of “algorithms” for the purpose of this essay. I recognize this obscures a lot of complexity but it’s not really relevant here).
What does AI and machine learning mean for people holding software design titles like “UX Designer”, “Product Designer”, or “Interaction Designer”? With the exception of service design, most of these designers have likely spent the majority of their careers working on GUIs for phones and computers. Maybe a few have done kiosks (stand-alone UI), large screen displays (really big UI). Maybe an even smaller subset have worked with wearables or conversational AI agents (i.e. chatbots). The point I’m trying to make is that experience with these types of technologies likely doesn’t equip designers to work with AI applications. The argument could be made that many AI products currently on the market are essentially a kind of middleware or service(s) which operate below the level of user interface. I think that’s a valid argument; designers who don’t care about middleware or how system architecture impacts user experience can continue their careers as before. There will likely be a need for UI designers who are tasked with designing attractive UIs for a long time to come. They don’t need to trouble themselves with the technologies that enable the products they are building. My view is that this attitude will relegate designers to being the “make it pretty” people. They will lay out text, choose color palettes, create style guides and pattern libraries and the like. Again, I think there’s a place for this in the software world. However, for UX architects and more senior designers (and those who aren’t explicitly “creative directors”), especially those working on complicated enterprise applications, “making it pretty” will be necessary but not sufficient.
Given this, I believe that designers of the future will need to evolve some skills around AI if they are to maintain relevance. There is currently a huge unmet need in this area. One of the great stumbling blocks for AI-enabled products is adoption. Regulators and lawmakers are (rightly) skeptical of autonomous vehicles. Companies are skeptical of adopting business-critical technologies that they can’t understand, maintain, debug, upgrade from, etc. Consumers will grow increasingly skeptical of adopting AI if it is viewed as skewed (think discriminatory algorithms) or prone to exploitation (Cambridge Analytica). Some of this is a matter of data-governance and corporate ethics. One could argue that governance and ethics fall into a kind of “meta” user experience but that’s outside my scope here. A more “designerly” area for investigation is how humans assess the validity of data and actions produced by such systems. For those who gravitate to learning and blurring boundaries, the good news is that this work will be interdisciplinary. It will draw primarily from computer science but also from social science and psychology. The goal being to understand how humans work with, adopt, debug and validate the actions of agent/human interactions. Further, as multi-agent systems come online, designers will need to understand how these systems interact not only with users but with each other (machine to machine interaction).
My naive personal view is that well-designed predictive or ML algorithms should include some kind of confidence or scenario-based data coupled with predictive data. The challenge for HCI designers of such applications falls broadly into two areas. First, failures must not be silent while the system returns plausible results or recommendations. Designing interactions with products, services and experiences built on machine learning or other AI technology with predictive/prescriptive capabilities presupposes human trust in the system. This was demonstrated by Springer et. al. showing that users place an inordinate amount of trust in applications deemed “intelligent” even when results from such systems may be fundamentally at odds with the users own experience. (Dice in the Black Box: User Experiences with an Inscrutable Algorithm). This gives rise to a second design challenge; designing effective means for users to evaluate the validity of results or actions produced by AI/ML systems. Of course this varies by context. Evaluation may be moot in the case of autonomous vehicles because catastrophic failures would occur too quickly for users to realistically avert. In other cases, products calling other AI/ML systems as dependencies (i.e. micro-services, APIs, etc.) would need to be fault-tolerant and fail gracefully in the absence of their dependencies. A third challenge arises specifically from predictive systems. Prediction puts users in the impossible position of evaluating hypotheticals that have not yet occurred and at a scale far beyond what human cognition is capable of. In many cases, users will not be able to rely on empirical perceptions to evaluate the performance or validity of such applications. In such circumstances a single failure will result in loss of trust in blackbox algorithms.
Broadly stated, the challenge for designers is to remove the opacity from the blackbox and find ways to render the complexity of “machine cognition” comprehensible to humans. From an HCI perspective, this might be understood as how humans interact with, verify and adopt/adapt to agent-based systems. In the case of human users, this could include human in the loop, human over the loop and eventually human under the loop interactions. From an architectural perspective it could be seen as how networked systems interact with data derived from other agents. In multi-agent systems I assume this to mean AI/ML agents interfacing with each other via some kind of service broker or middleware architecture.
A key tenet of my previous work has been to reduce opportunities for introducing human error in enterprise IoT applications. But as agent-based systems evolve, strategies to reduce human error are necessary but not sufficient. From an engineering perspective, algorithm quality is generally judged on performance (i.e. speed) and memory usage. Optimality, that is, how well an algorithm solves a particular kind of problem, can be determined mathematically. It’s as if the question of designing a better mousetrap is moot because we can quantitatively prove that no better mousetrap can be invented. I contend that we need to augment these quantitative aspects of algorithm performance with “meta” measures evaluating things like confidence, data quality and fault-tolerance. At a minimum, end users, customers and HCI professionals need heuristics to assess the quality of human/agent interactions. Without this, failures will be opaque and risk being siloed into traditional HCI or engineering perspectives. It may be possible to mitigate this using the standard approaches of HCI and algorithm design best practices, but widespread adoption of AI/ML systems will suffer if AI/ML remain “black boxes”. I’ll summarize this with a few interesting questions:
- What constitutes failure in an agent-based system and how should failures manifest for a user?
- How should failures due to data (ie. sparsity) be treated as opposed to flaws caused by algorithm design, connectivity, dependencies, etc.
- How should users interpret, ‘debug’, maintain and recover from such failures? Should humans second-guess the agent/model? If so, when?
- How might irrational or embodied human behavior impact agent results/actions? For example, skewing training data.
- Based on this, what are some best practices for designers, developers, architects, building human/agent interactions?