This paper presents an innovative large language model (LLM)-based robotic system for enhancing multi-modal human-robot interaction (HRI). Traditional HRI systems relied on complex designs for intent estimation, reasoning, and behavior generation, which were resource-intensive. In contrast, our system empowers researchers and practitioners to regulate robot behavior through three key aspects: providing high-level linguistic guidance, creating "atomics" for actions and expressions the robot can use, and offering a set of examples. Implemented on a physical robot, it demonstrates proficiency in adapting to multi-modal inputs and determining the appropriate manner of action to assist humans with its arms, following researchers' defined guidelines. Simultaneously, it coordinates the robot's lid, neck, and ear movements with speech output to produce dynamic, multi-modal expressions. This showcases the system's potential to revolutionize HRI by shifting from conventional, manual state-and-flow design methods to an intuitive, guidance-based, and example-driven approach.
Use Cases of Large Language Models driven Multi-Modal Human-Robot Interaction
The system's architecture includes three key modules: "Scene Narrator", "Planner", and "Expresser". The Scene Simulator mirrors the states of objects and humans as detected by sensors. The Planner module processes multi-modal inputs as event messages, encompassing the positions of individuals within the scene. Inter-module communication is facilitated using ROS.
The system structure
The interaction typically begins with a person's speech. For instance, "Scene Narrator" detects "Felix speaks to Daniel: 'Please hand me the red glass'." This event is then translated into natural language and relayed to the "Planner" module, initiating a GPT query. Simultaneously, the "Planner" informs the "Expresser" for an immediate rule-based response, leading the robot to look at Felix while its ears and lid roll back, simulating a listening gesture. Approximately 2 seconds later, GPT responds by invoking theget_persons() and get_objects() functions to identify people and objects present. The resulting data, including "Felix," "Daniel," and object details, are sent back to GPT for further analysis. During the wait for GPT's next response, the robot exhibits a 'thinking' gesture, looking from side to side with blinking lid movements. Shortly after, the LLM calls check_hindering_reasons() to assess if Daniel can see and reach the red glass and whether he is busy. Concurrently, facial_expression() is activated for the robot to look towards Daniel. The outcome indicates Daniel can hand over the glass, and the robot, following pre-defined guidance, opts not to intervene, silently displaying the reasoning on the GUI. Subsequently, Felix asks Daniel to pour cola into the glass. The robot, attentive to their conversation, deduces through check_hindering_reasons that Daniel is occupied with a phone call and learns from is_person_busy_or_idle that Felix is holding the cup. The robot then opts to pour cola from the bottle into Felix's glass. Should Felix not be holding the glass, or if it's beyond the robot's reach, the robot will instead place the bottle near Felix. Directed by LLM, the robot's head tracks the bottle during pickup and shifts to the glass while pouring. Upon completion, the robot nods towards Felix and announces, "I've poured Coca-Cola into your glass as Daniel is currently busy.".
The interaction flow. The blue square are the action generated by the LLM; the grey ones are rule-based function.
"You are in control of a robot called 'the_robot' and observe persons talking in the form '<sender> said to <receiver>: <instruction>'. "
"You can call given functions to gather information or act, or response with text only for reasoning. "
"Your task is: "
"You should check the reasons that could hinder the <receiver> from performing the <instruction>. "
"If there is NO hindering reason for the <receiver>, then you MUST never help and also not speak. "
"If there is a hindering reason for the <receiver>, then you MUST solve the <instruction> yourself by always targeting the <sender>. "
"If you like to speak, you must use the speak function and be concise. "
"If 'the_robot' is the <receiver>, you should always help. "
"You must make sure to always use correct and precise object and person names that are available in the scene, always start with getting this information. "
"Try to infer which objects are meant when the name is different, but if unsure, ask for clarification. "
"IMPORTANT: Following list of rules you must obey: "
"1. Before calling the STOP function you MUST respond with a brief explanation of your behavior. "
"2. Always call 'is_person_busy_or_idle' to check if <sender> is busy or idle before helping. "
"3. If <sender> is idle, use the 'hand_object_over_to_person' function. "
"4. If <sender> is busy, you must use the 'move_object_to_person' function. "
"When calling each function, call robot_facial_expression() at the same time to communicate you intent."
"When calling can_person_see_object(), the robot need to look at the person."
def can_person_see_object(self, person_name: str, object_name: str) -> str:
"""
Check if the person can see the object. If the person cannot see the object, it would be hindered from helping with the object.
:param person_name: The name of the person to check. The person must be available in the scene.
:param object_name: The name of the object to check. The object must be available in the scene.
:return: Result message.
"""
...
if result is None or len(result) != 1:
return f"It could not be determined if {person_name} can see {object_name}. There were technical problems."
if result[0]["is_visible"]:
return f"{person_name} can see {object_name}."
return f"{person_name} cannot see {object_name}, it is occluded by {self.id_to_utterance_mapping[result[0]['occluding_objects'][0]]}"
"For example, when call move\_object\_to\_person(), can\_person\_see\_object(), can\_person\_reach\_object(), speak(), you also need to call robot\_facial\_expression(), such as:"
'"tool_calls="["ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"head_motion": null, "ears_lid_motion": "observe", "gazed_target": "the_cola_bottle" }","name=""robot_facial_expression"")","type=""function"")", "ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"person_name": "Daniel", "object_name": "the_cola_bottle"}","name=""can_person_see_object"")","type=""function"")"]"'
'"tool_calls="["ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"head_motion": null, "ears_lid_motion": "focus", "gazed_target": "the_cola_bottle"}","name=""robot_facial_expression"")","type=""function"")", "ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"person_name": "Felix", "object_name": "the_cola_bottle"}","name=""move_object_to_person"")", "type=""function"")"]"'
'"tool_calls="["ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"head_motion": null, "ears_lid_motion": "focus", "gazed_target": "the_cola_bottle"}","name=""robot_facial_expression"")","type=""function"")", "ChatCompletionMessageToolCall(id=""...","function=Function(arguments=""{"person_name": "Felix", "text": "Here is the coke, you can now pass it to Felix."}","name=""speak"")", "type=""function"")"]"'