21. October 2021

Convincing immersion in distant worlds Convincing immersion in distant worlds

NimbRo team from University of Bonn ranks first in ANA Avatar XPRIZE Semifinals. 133,333 US dollars prize money

A leap into the future: the goal of the ANA Avatar XPRIZE competition is to develop robotic systems that can transmit a human's senses, actions and presence to a remote location in real time. Through these avatars they can perform a variety of handling tasks and can interact with humans – from audio and video transmission to direct physical contact. Team NimbRo from the University of Bonn qualified in Miami (USA) for the finals and is to receive 133,333 US dollars. The prize purse for the finals is eight million US dollars.

Team NimbRo of the University of Bonn
Team NimbRo of the University of Bonn - at the ANA Avatar XPRIZE semifinals in Miami (USA). © Photo: University of Bonn/Autonomous Intelligent Systems
Download all images in original size The impression in connection with the service is free, while the image specified author is mentioned.

38 teams from 16 countries had qualified for the semifinals. Among them was NimbRo from the University of Bonn, which was ranked first with 99 out of 100 points. "I am proud of my team, which developed an avatar system well suited for the complex competition tasks and made it fit to the point," said Prof. Dr. Sven Behnke, head of the Institute for Computer Science VI – Intelligent Systems and Robotics at the University of Bonn, after the semifinals. The finals are scheduled to take place next year in the USA. The 133,333 US dollars in prize money that NimbRo will receive for qualifying will be used to prepare for the finals. In the final round, a further eight million U.S. dollars will then be distributed among the top three teams.

The Bonn-based team NimbRo has been developing the Avatar system, which consists of an operator station and a mobile avatar robot, for two years. The avatar has a human-like upper body with two arms and five-finger hands. Attached to the head is a wide-angle stereo camera, a stereo microphone, and a display on which the operator's face is animated. The remote scene is visualized to the operator through a 3D head-mounted display, with the operator's head movements captured and transferred to the avatar head. This allows the operator to look around freely, see past obstacles, and view objects from different angles.

Exoskeletons mediate the hand movements

The three-dimensional perception of the scene is created in the operator's head by the different viewing angles and camera movements. Stereo audio also contributes to immersion. The operator feels the interaction forces of the avatar hands via force-torque sensors in the wrist. Robotic arms sense the operator's hand movements and relay them to the avatar arms. Hand exoskeletons mediate the finger movements. Gripping forces are sensed by finger motor currents and transmitted to the operator's fingers. The avatar's base drive can be controlled by a foot controller, with cameras creating a bird’s eye all-around view for orientation. The head-mounted display is equipped with three cameras that capture the operator's eye movements and facial expressions for facial animation.

In the run-up to the semifinals, the Bonn team used its avatar system to perform numerous tasks in everyday environments. These included, for example, making coffee, playing chess, but also measuring blood pressure, temperature and oxygen saturation like a nurse. Like a gentleman, the avatar helped a person in need of assistance into a jacket. In the semifinal tasks in Miami, the operator played jigsaw puzzles together with a test person through the avatar system, shone at the celebration of a business deal and explored an object with his senses. The system was not operated by the development team, but by members of the international jury. Therefore, intuitive operation was one of the requirements. The Bonn system worked perfectly in the semifinals and impressed the jury members.

That such avatars have a practical significance became clear not least by the contact restrictions of the Corona pandemic. With such systems, people can overcome distances and be present in another location, for example, when working in dangerous or inaccessible environments, when supporting emergency forces to cope with disasters, or in space. "Avatar systems have the potential to overcome spatial separation between people and, through mass production – and the associated reduction in costs – may be as widespread in a few years as video conferencing systems are today," says Prof. Behnke.

Video of semifinals run by Team NimbRo from the University of Bonn: https://youtu.be/UA7vpQO5oZQ

Information about the avatar system: https://www.ais.uni-bonn.de/nimbro/AVATAR

Information about the competition: https://www.xprize.org/prizes/avatar


Prof. Dr. Sven Behnke
University of Bonn
Institute of Computer Science VI - Intelligent Systems and Robotics
Tel. +49-228-73 4116
E-mail: behnke@cs.uni-bonn.de

Cooperative solving of a puzzle
Cooperative solving of a puzzle - by two jury members using the NimbRo Avatar system. © Photo: University of Bonn/Autonomous Intelligent Systems
Jury member
Jury member - exploring a vase through the NimbRo Avatar system. © Photo: University of Bonn/Autonomous Intelligent Systems
Jury member
Jury member - in the operator station of the NimbRo Avatar system. © Photo: University of Bonn/Autonomous Intelligent Systems
Wird geladen