Research Focus
Living Support System Using Multimodal Situation Understanding
This study aims to develop an intelligent living support system that understands users' states and behaviors in a smart environment. The system collects information from indoor cameras, environmental sensors, and wearable devices, including user location and actions, room temperature, illuminance, heart rate, and body temperature. By integrating heterogeneous data, the system estimates the user’s current situation. It provides appropriate support, such as controlling home appliances accordingly. For example, when the room temperature is high, and changes are observed in the user’s body temperature or heart rate, the system can operate a fan by considering both environmental information and biological information. Furthermore, by combining these data with camera-captured human motion information, the system can infer internal states, such as fatigue or heat discomfort, that are difficult to determine from images alone. Through this approach, the study aims to create a natural and user-friendly living support environment.
Situation Understanding Using Temporal Scene Graphs with Multi-Camera
This study aims to develop a method for situation understanding in intelligent spaces using multiple cameras and temporal scene graphs. In an intelligent space, machines need to understand what is happening in a room by recognizing relationships among people, objects, and actions. However, a single camera may fail to observe people or objects hidden behind obstacles, making robust recognition difficult. In addition, directly feeding video data into an AI model makes it difficult to handle temporal changes and event flow. To address these issues, the proposed method observes a room from multiple viewpoints. It represents relationships among people, objects, and actions as scene graphs. These graphs are then connected along the time axis to construct temporal scene graphs. By inputting them into a large language model, the study examines applications such as situation description, question answering, and action support based on a structured understanding of indoor events.
Spatial Log System for Recording Relationships Between People and Objects
This study focuses on a spatial log system for realizing an intelligent space that provides appropriate services to users. The proposed system records human actions and environmental changes, enabling the space to understand what happened and support users based on their activity history. Multiple sensors are installed in the space, and people are detected using RGB-D cameras and YOLO. The system also recognizes objects that have been moved through human actions and identifies who moved which object. In addition, to acquire more detailed information about target objects, the system uses the mobile robot Kachaka equipped with a camera mounted on an arm. The robot moves to the target object's position and observes it at close range, complementing information that is difficult to obtain with fixed sensors alone. Through this approach, the study aims to understand in detail the relationships between people and objects and to apply spatial logs to intelligent support services in daily environments.
Multi-Tracking System with Face Recognition Using Multiple Cameras
This study aims to develop a person identification system for intelligent spaces that is location-independent. In conventional methods, face recognition is performed at fixed points, such as entrances, and a person’s name is associated with their position. Therefore, users must be aware of the camera and behave in a way that allows it to be recognized. In addition, face recognition and person tracking, or ReID, operate independently, which can cause identity switching when a person is occluded or tracked for a long time. To address these issues, this study re-executes face recognition at appropriate times during the tracking process. It integrates information from multiple DINDs (distributed intelligent network devices) equipped with cameras and sensors. By combining these data, the system maintains consistent personal identification across the entire space. Through this approach, the study aims to realize an intelligent space that users can access naturally without performing any special actions.