Research Focus
3D Avatar Reconstruction for Robotic system
This research advances the development of high-fidelity digital twins for robotic systems, bridging the critical gap between human facial appearance and behavioral synthesis. Utilizing multi-view acquisition and state-of-the-art computer graphics, the study reconstructs realistic 3D avatars and behavioral clones designed to disentangle universal facial morphology from personalized, non-verbal cues. The technical framework employs dual deep learning pipelines to achieve this synthesis. To model non-verbal facial behavior, a neural network is trained on extensive dyadic interaction datasets, capturing the intricate social nuances of human engagement. Simultaneously, appearance reconstruction is facilitated by a model trained on large-scale datasets of facial profiles performing diverse expressions. This allows the system to accurately map facial geometry and the underlying mechanical nuances of human expression. Ultimately, these digital twins function as the foundational architecture for next-generation robotic facial interfaces. By integrating these high-fidelity models into virtual and mixed reality (VR/MR) environments, this research significantly enhances the realism and clinical efficacy of medical training simulations, providing practitioners with more responsive and lifelike patient-interaction experiences.
Natural Pain Facial Expression Generation for a Robot Patient Simulator
This study focuses on facial expression generation for a robot patient simulator, aiming to better represent patient pain. In medical education, students need to observe a patient’s facial expressions and reactions as they learn to assess the situation and respond appropriately. If a robot patient can display expressions that reflect the degree and condition of pain, it can provide a more practical training environment for clinical practice. This study focuses on facial Action Units (AUs), which describe movements of facial parts. It analyzes how movements around the brow, eyes, and mouth affect the naturalness of pain expressions. In addition, impression evaluations are conducted to investigate which types of facial changes lead observers to perceive the expression as painful and natural. Through these analyses, the study aims to develop a clear and natural pain expression generation system that can be applied to medical education and nursing training.