Photo under a CC0 licence, by startupstockphotos.com
Modern computing devices allow users to enter information using keyboards, mice, or touch screens. Mobile devices have additional sensors like gyrometers and accelerometers that orient the screen. All of these input devices collect data at millisecond precision.
Recent research has focused on a person’s emotional state and how those emotions affect their fine motor movements. Your emotions, like happiness, sadness, anger or frustration, cause immediate, uncontrolled changes in how your hand moves a computer mouse, navigates a touch pad screen or holds a smartphone. Below are the results of a series of controlled laboratory studies that we conducted to determine the effects of negative emotions on a person’s mouse cursor movements.
Participants were told they were going to take a timed, scored intelligence test, then given a monetary reward depending on how many questions they answered correctly. They were divided into two groups.
While the participants in the first group read the instructions, a timer in the upper right-hand corner of the screen began to count down. The questions took a long time to load, while the timer continued to count down. By the time each question loaded, there was only fifteen seconds left to answer a difficult question.
The screen repeatedly advanced to the next question before the participants could answer, telling them their time had run out because they took too long; creating frustration. Then these participants were told that because of their slow reaction time and incorrect answers, their scores indicated a lower intelligence level than most of the people who had taken the test; increasing frustration.
The second group was not timed, had easy questions, and was congratulated for answering every question correctly at the end of the test.
Theory predicts that frustration impacts the mind, increasing a person’s mental effort. As people experience increased mental effort, they become more indecisive, and their mouse movements get longer and slower (Figure 1).
Figure 1: Example Mouse Trajectory under the Influence of Negative Emotion
The participants in both groups were asked to drag six 4-digit numbers from a box on the left side of the screen to a box on the right side of the screen and to order them from the lowest number to the highest (Figure 2). Their mouse movements were captured.
Now that we had one “frustrated” group and one that wasn’t, we were ready to test how their emotional states affected their mouse movements.
Figure 2: Number sorting task where mouse movements were tracked.
At the end of the task, each participant was also asked to rate their current emotional state.
First, the group we were trying to frustrate reported significant negative emotions. Our attempts to frustrate them appeared successful. Second, the participants in the “frustrated” group showed significantly longer and slower mouse cursor movements when completing the number-ordering task than the participants in the “non-frustrated” group.
Studies 2 and 3
In Study 2, we asked two different groups to navigate a fictitious e-commerce website, pretend to buy the product and then fill out a survey rating their emotional level.
The first group had a website that loaded slowly, along with many other errors, in an attempt to frustrate them. The second group experienced no problems with the website. Mouse cursor movements were captured during the website navigation for each group and the results were similar to Study 1—frustrated participants had longer, slower mouse movements.
We were also able to predict which participants had the “frustrating” website, and which ones didn’t, at an accuracy rate of about 82 per cent.
In Study 3, participants were asked to use real, web-based, product configuration systems to build either a Dell computer or a Volkswagen automobile. Each participant navigated through five different configurations with random degrees of difficulty, and was asked to rate his or her emotions after each task. Without knowing which participants had which configuration, we were able to identify who was experiencing frustration, whose frustration was building over time, and ultimately who was working on an easy, medium or hard configuration task, just by the changes in their mouse movements.
The studies above show that unfairness and challenging content leads to frustration. Prior research shows that increased frustration leads to increased mental effort. Increased mental effort causes predictable changes in mouse movements, which can be captured by all modern computer input devices. This insight has led to the development of patented algorithms commercialised by Neuro-ID.
Neuro-ID’s technology is used to capture a person’s response to various types of questions. Those experiencing frustration and associated indecision when answering a particular question can be immediately identified. Clients can then decide if they wish to look into that question further. To clarify, Neuro-ID’s technology is not a lie detector; it’s an indecision detector.
Figure 3 provides a real world example of this approach. Here, a customer applying for a subprime loan is asked a series of yes/no questions while the mouse movements are tracked. In the potentially frustration-inducing-question, inside the red rounded rectangle, they are asked about their ability to repay the loan. Their movements slowed, circled and made a wide arc before selecting a final answer.
Figure 3: Mouse trails of subprime loan customer who ultimately defaulted (screenshot)
Neuro-ID’s ValitectTM product flagged this response as “highly suspicious”; the customer defaulted within a few weeks. The customer seemed to know they weren’t going to be able to pay the loan back.
Neuro-ID believes their technology can be used in a broad range of contexts:
- Airports: Are you carrying any banned items?
- Insurance claims: Was there any pre-existing damage to your automobile?
- Financial Services: Do you expect any reductions in your future income?
- Insider threat: Are you accessing data that violates organisational IP agreements?
Opportunities for this technology extend across many industries, with the potential for far-reaching impacts.
Credit: The authors thank Jackie Valacich for writing this blog post on their behalf.
- This blog post is based on the authors’ paper How is Your User Feeling? Inferring Emotion Through Human-Computer Interaction Devices, MIS Quarterly (in press).
- The post gives the views of the author, not the position of LSE Business Review or the London School of Economics.
- Before commenting, please read our Comment Policy.
Martin Hibbeln is an assistant professor of Finance at the Mercator School of Management, University of Duisburg-Essen, Germany. His research interests include financial risk management, empirical banking, and individual decision making.
Jeffrey Jenkins is an assistant professor of Information Systems at the Marriott School of Management, Brigham Young University and Chief Technology Officer at Neuro-ID, Inc. His research examines how to infer human cognitive states using computer input devices such as the computer mouse, keyboard, or touchscreen.
Christoph Schneider is an assistant professor in the Department of Information Systems at City University of Hong Kong. His primary research interests include human–computer interaction, electronic commerce, and computer-mediated collaboration.
Joe Valacich is an Eller Professor of MIS at the University of Arizona and the Chief Science Officer at Neuro-ID, Inc. His research focuses on human-computer interaction, fraud and deception detection, cyber security and emerging technologies.
Markus Weinmann is assistant professor of information systems at the University of Liechtenstein. His research focuses on human–computer interaction, behavior change and tracking, applied behavioral economics, and individual decision making in online contexts.