Many of the AI dialogue systems currently in use are primarily text-based and therefore often come across as impersonal. As a realistic conversation partner, the innovative AI avatar is designed to incorporate nonverbal cues such as gestures, facial expressions, and tone of voice, thereby going beyond mere text analysis. This enhances the informative value of interviews and makes the conversations more pleasant and easier to understand for the interviewees. Project leader Dr. Tobiasz Kaduk explains: “The system will combine large language models (LLMs) capable of structured thinking and dialogue validation with an emotionally expressive avatar that communicates through facial expressions, gestures, body language, and context-specific environmental cues.” Thus, the avatar responds not only through text or speech but also through nonverbal expressions.
A key aspect of this research is the gamification of human-AI interaction through immersive audio-visual communication and AI integration to enhance user experience and accessibility for users of different ages and backgrounds. “The development of our technology focuses on naturalistic animations, emotional expressions, and adaptive responses designed to create more engaging and authentic interactions”, adds Tobiasz Kaduk. With funding from the prototyping grant, these findings are to be verified for the use case and a functional prototype of the AI-based avatar is to be developed.
Various use cases from therapy to games
The fields of application for such an AI system are diverse and range from therapies in the healthcare sector to training purposes or gamified avatars. In psychotherapy, for example, the avatar could handle initial questionnaires or automated dialogues. This would help compensate a growing shortage of staff in the clinical sector, reduce waiting times for patients, and improve the comfort of treatment. In the field of video games, avatars could help create a more realistic gaming environment. For smaller game studios that lack the resources to develop customized, naturalistic animated avatars, this technology could provide an accessible solution for creating more immersive and emotionally engaging virtual environments.
Successful collaboration between medicine and computer science
The “FeelForge” project combines research findings and expertise from the fields of medicine and computer science. It builds on preliminary work from the InVirtuo 4.0 joint project of the University of Bonn, University Hospital Bonn, the German Center for Degenerative Diseases (DZNE) and TU Dortmund, where Dr. Tobiasz Kaduk developed several interactive 3D virtual environments and gained practical experience in working with avatar systems, animation pipelines, and the integration of LLM-based conversational AI tools.
Project members of “FeelForge” are Dr. Tobiasz Kaduk, Prof. Dr. Lucie Flek, head of the “Data Science & Language Technologies” research group at the Bonn-Aachen International Center for Information Technology (b-it), Prof. Dr. Alexandra Philipsen, director of the Department of Psychiatry and Psychotherapy at UKB, Dr. Niclas Braun, head of the “Virtual Reality Therapy and Medical Technology” research group at UKB, and Dr. Tatiana Goregliad Fjaellingsdal (UKB).
Funding from the Transfer Center enaCom
The prototyping grant supports scientific ideas that are to be prepared for practical application. In addition to financial support of up to 20,000 euros, the researchers receive a comprehensive support program featuring workshops on various topics as well as close consultation with start-up experts from the Transfer Center enaCom. The goal is to create an application-oriented prototype that serves as a viable foundation for a future spin-off.