Researchers at the Chair for Human-Centered Artificial Intelligence at the University of Augsburg have developed an interactive assistance system for special use on mobile devices. It offers a new diary-like questionnaire and individual coaching tool for stress management that can be used as a workplace risk assessment tool as well as for reintegration into the workplace. In the federally funded EmmA project (short for Emotional Mobile Avatar as Coaching Assistant), a multimodal real-time sensor analysis developed by the University of Augsburg was coupled with a virtual avatar. Based on a socio-emotional behavioural model, the avatar reacts to user sensory cues.

Headed by Professor Elisabeth André, the Chair for Human-Centered Artificial Intelligence has specialised in human-machine interactions in Artificial Intelligence research for over 15 years and is one of the world’s leaders in the field of machine multimodal behaviour analysis. “The software we developed enables the synchronised recording and analysis of affective behavioural features.  Together with colleagues from clinical psychology, we are investigating what information such signals provide about a person’s state of mental health in order to initiate suitable therapeutic measures in good time,” says André. Through the EmmA project, the recognition software was further developed for mobile devices and coupled with an avatar. This allows people at risk to keep a kind of personal diary with the avatar outside of a therapy session. It thus serves as a personalised assistance system and replaces the classic questionnaire as a way of providing the therapist with information about a patient’s state of health.

The biggest challenge is human dialogue

The messages conveyed by the avatar to the user are based on behavioural therapy models that help to make dialogue goal and evidence oriented. The multimodal signals sent by the user are incorporated into the dialogue to make the avatar react appropriately in a human-like manner. Yet implementation is complex because facial features, gestures, and the user’s way of speaking must first be correctly understood via the mobile device’s sensors (camera, microphone), with background noise simultaneously filtered out. As an example, a user might report experiencing stressful problems at work, and the programme might then discern from their tone of voice and facial expressions signs of sadness and exhaustion. The avatar would then ask specific questions in order to offer a relaxation exercise.

For all of this to work, a complex interplay of efficient model architectures for data processing, flexible interfaces, end device computations, access to pre-trained models, and neuronally structured online learning methods and applications for communication transfer are required. “The approach used in the project used the currently available technical possibilities to their full extent,” explains Michael Dietz, research associate at the Chair of Human-Centered Artificial Intelligence. “Our goal was to achieve a good balance between the accuracy of the recognition and the resources needed to run the model. To protect privacy, sensitive data should not be transferred to a cloud-based service for processing but rather remain on the respective end device. However, mobile devices have only limited computational resources, which is challenge that we have overcome,” says Dietz.

Easy access to support services

The underlying reason for the EmmA project is the alarming increase in the number of reported absences from work due to mental stress, which has been steadily increasing for years. For this reason, lawmakers expanded the Occupational Health and Safety Act (Arbeitsschutzgesetz) to include a mental risk assessment back in 2013. Although it is obligatory, less than a quarter of all companies have implemented it to date. The mobile assistance system developed in Augsburg can contribute to providing employees with individual and easy to access support in times of mental stress. It is explicitly not intended to replace psychotherapy, but rather to provide support in everyday life. People who are reserved and fear negative reactions may particularly benefit because they may have less inhibition when it comes to interacting with an avatar.

Future applications on the horizon

During the development of the project, user and acceptance studies were also conducted, which were fed into the learning system. “We don't just have to work on technical solutions, but also keep an eye on the ethical, legal, and social implications of using such assistants,” says André. The results of the EmmA project will be used in the follow-up federally funded project UBIDENZ (Ubiquitous Digital Empathic Therapy Assistance), which aims to develop an assistance system that accompanies depression sufferers after inpatient treatment.

The software for multimodal behavioural analysis developed in Augsburg is also a central component of the Tōku Hoa project, which is being led by the University of Auckland in New Zealand on behalf of several renowned institutions worldwide. Tōku Hoa is taken from the Māori language and means “my best friend.” The intention is to develop “a digital friend” that could serve as a personal companion for people at risk of mental illness.