Leveraging Computer Vision-Based Hand Tracking in Smartphone XR Exposure Scenarios to enable Controller-Free Interaction
College
College of Engineering
Mentor Information
Byeol Kim
Description
Extended reality (XR) has emerged as an effective and convenient solution for broadening accessibility to exposure therapy. Smartphone-based XR, though cost-effective, is often constrained by limited interactivity, engendering a passive experience that can compromise the user-immersion and treatment efficacy. To address this limitation, this study presents a novel computer vision–based hand-tracking approach using Google MediaPipe, integrated within a stereoscopic smartphone-based XR system to enable real-time, controller-free interaction in a dog-fetch scenario. Hand landmark coordinates derived from MediaPipe are algorithmically adjusted based on changes in inter-knuckle distances during anterior/posterior movement to approximate depth, at which point interactive spheres are overlayed to support object interactions with hand transformations. Grabbing was implemented by creating a custom BoxCollider encapsulating the palm and by moving the object synchronously with the BoxCollider when the object abuts both the BoxCollider and the spheres corresponding to the tips of the fingers. When no longer in contact with the tips of the fingers, the object is released. To address latency issues with throwing, an algorithm was developed to adjust the velocity of thrown objects to their velocity a short time before release. Stereoscopic perception was integrated using two Unity virtual cameras, situated 0.065m from each other to emulate inter-pupillary distance, that render left and right viewpoints on the smartphone screen in split-screen. This work verifies the technical viability of leveraging computer vision to enable controller-free interaction within a stereoscopic smartphone-based XR system in an exposure therapy scenario, justifying future investigations examining outcomes of the system with human subjects.
Leveraging Computer Vision-Based Hand Tracking in Smartphone XR Exposure Scenarios to enable Controller-Free Interaction
Extended reality (XR) has emerged as an effective and convenient solution for broadening accessibility to exposure therapy. Smartphone-based XR, though cost-effective, is often constrained by limited interactivity, engendering a passive experience that can compromise the user-immersion and treatment efficacy. To address this limitation, this study presents a novel computer vision–based hand-tracking approach using Google MediaPipe, integrated within a stereoscopic smartphone-based XR system to enable real-time, controller-free interaction in a dog-fetch scenario. Hand landmark coordinates derived from MediaPipe are algorithmically adjusted based on changes in inter-knuckle distances during anterior/posterior movement to approximate depth, at which point interactive spheres are overlayed to support object interactions with hand transformations. Grabbing was implemented by creating a custom BoxCollider encapsulating the palm and by moving the object synchronously with the BoxCollider when the object abuts both the BoxCollider and the spheres corresponding to the tips of the fingers. When no longer in contact with the tips of the fingers, the object is released. To address latency issues with throwing, an algorithm was developed to adjust the velocity of thrown objects to their velocity a short time before release. Stereoscopic perception was integrated using two Unity virtual cameras, situated 0.065m from each other to emulate inter-pupillary distance, that render left and right viewpoints on the smartphone screen in split-screen. This work verifies the technical viability of leveraging computer vision to enable controller-free interaction within a stereoscopic smartphone-based XR system in an exposure therapy scenario, justifying future investigations examining outcomes of the system with human subjects.
