Technology Introduction:
This system integrates LLMs, VLMs, object and skeleton recognition to achieve environmental perception, dynamic task planning, change monitoring, object rearrangement, and human–robot interaction. It enables accurate execution of natural language commands with continuous iteration to enhance the adaptability and reliability of home-service robots in complex environments. Real-time experiments validate generative AI's potential and practical value for semantic understanding, interaction, and planning.
Industry Applicability:
With mature generative AI and multimodal technologies, our system’s Dynamic Task Planner adapts workflows in changing environments for greater autonomy in human–robot collaboration and unstructured settings. The Object Rearrangement System automatically arranges tableware to international standards, and the Multimodal HRI System makes interactions more natural and diverse. Applicable to manufacturing, warehousing, home service, and medical care.