Multimodal Smart Home Care: Lightweight Vision-Language Models for Real-Time Event Awareness and Anomaly Response
Technology Introduction:
"Our system is built around lightweight Vision-Language Models (VLMs) designed for real-time understanding of continuous visual streams and multimodal information in home environments. It enables continuous awareness of daily activities, environmental changes, and potentially abnormal events, while maintaining computational efficiency and supporting privacy-conscious deployment for long-term smart care applications.1. Lightweight and Efficient Multimodal Models: Through model compression and efficient inference techniques, MASU reduces computational and hardware requirements, enabling multimodal AI deployment in homes, on edge devices, and in resource-constrained environments.
2. Continuous Understanding of Streaming Visual Events: Rather than focusing only on individual image recognition, MASU continuously analyzes streaming visual data and temporal context to understand daily activities and event transitions, including activity patterns, routines, and unusual behaviors.
3. Anomaly Awareness and Real-Time Response: By integrating visual, language, and other in-home sensor information, MASU can identify potential risks such as falls, prolonged inactivity, or unusual movement, and provide event summaries, alerts, and relevant information to support timely responses."
Industrial Applications:
"Our technology can be applied to smart homes, long-term care, elderly care, healthcare, and safety monitoring. Its lightweight multimodal models reduce the computational cost of continuous audiovisual monitoring while transforming raw sensor data into semantically meaningful descriptions of daily activities and abnormal events, helping caregivers quickly understand situations that require attention.The technology can also be extended to care facilities, smart hospitals, telecare, public safety, and other domains requiring continuous event monitoring, serving as a core technology for real-time environmental understanding, anomaly detection, early warning, and AI-assisted decision support."
Academia Sinica、National University of Tainan、National Yang Ming Chiao Tung University、Chung Yuan Christian University、National Institutes of Applied Research
MASU (Multimodal AI SUite) focuses on the development of lightweight and highly efficient multimodal large language models. The project aims to enable AI systems to continuously understand visual, linguistic, and other sensory information under limited computational resources, supporting real-time, long-term, and practically deployable multimodal applications. In particular, MASU will develop compact Vision-Language Models (VLMs) for streaming visual event understanding, enhancing their ability to capture continuous events, temporal context, and abnormal situations while maintaining high inference efficiency, low deployment cost, and privacy awareness.
MASU targets a wide range of applications, including smart home care, long-term care, smart healthcare, and environmental and safety monitoring. In home-care scenarios, for example, the system can integrate continuous visual streams with various in-home sensing modalities to understand daily activities and living conditions, detect potentially abnormal events such as falls or prolonged inactivity, and provide real-time alerts, event summaries, and relevant information to support timely responses.
The MASU core team brings together researchers from several leading AI and information technology institutions in Taiwan, including the Institute of Information Science, Academia Sinica; the Research Center for Information Technology Innovation (CITI), Academia Sinica; National Yang Ming Chiao Tung University; Chung Yuan Christian University; and National University of Tainan. The team combines expertise in large language models, multimodal learning, computer vision, natural language processing, and efficient AI to advance a new generation of multimodal AI technologies that are efficient, deployable, and highly relevant to real-world industrial applications.
MASU targets a wide range of applications, including smart home care, long-term care, smart healthcare, and environmental and safety monitoring. In home-care scenarios, for example, the system can integrate continuous visual streams with various in-home sensing modalities to understand daily activities and living conditions, detect potentially abnormal events such as falls or prolonged inactivity, and provide real-time alerts, event summaries, and relevant information to support timely responses.
The MASU core team brings together researchers from several leading AI and information technology institutions in Taiwan, including the Institute of Information Science, Academia Sinica; the Research Center for Information Technology Innovation (CITI), Academia Sinica; National Yang Ming Chiao Tung University; Chung Yuan Christian University; and National University of Tainan. The team combines expertise in large language models, multimodal learning, computer vision, natural language processing, and efficient AI to advance a new generation of multimodal AI technologies that are efficient, deployable, and highly relevant to real-world industrial applications.
