A partner project of the PhenoRob Cluster of Excellence, AgriLoop is based in the Sustainable Futures Transdisciplinary Research Area. The Bonn-based element of its research has secured €463,000 in funding from the German government, specifically the Federal Ministry of Research, Technology and Space (Bundesministerium für Forschung, Technologie und Raumfahrt, BMFTR).
Growing crops in greenhouses is currently facing some major challenges, with a shortage of skilled labor, increasingly complex operations and the need for more sustainable cultivation methods putting increasing pressure on producers. Although robotic systems are already able to aid monitoring and intervention efforts, to date they have rarely shown themselves capable of taking domain-specific information from plant experts on board in a structured way and translating it into tangible action.
Talking to robot colleagues as one would with a human
This is where AgriLoop comes in. The project is developing an integrated human-in-the-loop system that combines AI models such as language models, vision-language models and vision-language-action models with physically embedded robotics and the 3D perception of plant stocks. The aim is for experts to be able to use natural language to ask questions about plant condition, give instructions and control the behavior of their greenhouse robots in a targeted way via an intuitive interface. The system links these interactions directly with 3D reconstructions of greenhouse scenes and translates them into actions that the robots can perform safely and reliably.
The project is designed as a close partnership between the German and Taiwanese sides, with the former contributing robot platforms, 3D perception and reconstruction techniques, real-life greenhouse environments and crop cultivation expertise and the latter building key skills for multimodal language processing, dialogue systems, command grounding and learning from the experts’ feedback. The funding from the BMFTR is thus supporting efforts to forge international research partnerships and apply innovative AI and robotics methods to fields of significant practical relevance for the future, such as smart agriculture.
Lofty ambitions
The project is geared toward producing a demonstrator that will clearly show how a greenhouse robot can be given comprehensible instructions and targeted control commands. To this end, the researchers are devising new ways in which the robot can map out its surroundings inside the greenhouse and link them to spoken or written instructions from experts. Thus the aim is for orders like “Investigate this plant more closely” or “Remove this leaf” to be able to be translated into reliable robotic movements in the future.
The project is also producing a number of ultra-efficient AI models that can be used on robot platforms without constantly having to rely on vast data centers. The data that it is garnering on human-robot communication is intended to help researchers to fine-tune agricultural robotics systems. The project’s results may therefore furnish new scientific findings as well as facilitating practical applications for smarter, more sustainable agriculture in Germany and Taiwan over the long term.
Bennewitz and her doctoral student Rohit Menon will be working on the project in Bonn, while an additional doctoral student post will also be created out of the project’s funding.
Media contact:
Professor Maren Bennewitz
Email: maren@cs.uni-bonn.de
Four questions for Maren Bennewitz
Why have robots so far failed to understand these kinds of instructions?
Professor Bennewitz: For us humans, it’s second nature to connect language with our surroundings and the knowledge we’ve gained from experience. If an expert says “Remove this leaf,” then you’re going to know straight away which plant and which leaf are meant and how you should go about it without damaging the plant. That’s much harder for a robot. It needs to interpret the language, identify the right object in its immediate vicinity, gauge its exact position and use all this information to determine a suitable movement. Up until now, these abilities—multimodal AI models, spatial 3D perception and the physical control of the robot—have largely worked in isolation. This is precisely where AgriLoop comes in, aiming to meld these individual steps into a single system.
How is AgriLoop planning to solve the problem?
The robot uses cameras and sensors to map its surroundings in order to create a spatial model of the plants in the greenhouse. At the same time, the system processes the verbal instructions from the experts via an intuitive interface. State-of-the-art AI models, including language and vision-language models, then combine these information sources together, synchronizing what has been said and what specific detail inside the 3D greenhouse the instruction relates to. This linking is ultimately designed to produce a concrete, safe and reliable action on the part of the robot. For example, the order “Investigate this plant more closely” could be translated into a targeted movement by the robot toward a specific plant.
Why do we still need people?
When you’re working with plants in particular, not every decision can be programmed in advance. Experts possess empirical knowledge that will often hinge on the situation at hand: how healthy does a plant seem? Which leaf should be removed? Where is there a particular need for caution? This is why AgriLoop is keeping the human in the loop, who is there to issue instructions, correct the robot and evaluate its decisions. The robot is expected to use this feedback to adapt its actions more closely to what’s actually needed in the greenhouse. In other words, the aim isn’t to replace human expertise but rather to enable robotic systems to make use of it.
What specifically could that change?
In the long term, systems like this could support experts doing time-consuming or repetitive tasks, such as monitoring plant stocks or undertaking targeted plant care. This would be particularly appealing where there’s a labor shortage or large plant stocks require regular checks. At the same time, more precise interventions could help to make more targeted use of resources. We still need to do some more research before we get there, however: AgriLoop is starting by laying the foundations and is aiming to use a demonstrator to show that human-robot collaboration can work in this way as a basic principle.
