NRT Trainee Team Presentations
Fall 2024 Presentation Abstracts
XR Co-signing: an Innovative Tool for Learning American Sign Language
Project team: James Spann, Yamin Zheng, and Caitlin Sharp
Faculty advisor: Dr. Zhen Bai
Abstract
American Sign Language (ASL) education presents unique challenges, particularly in integrating non-manual markers such as facial expressions, head movements, and eye gaze, which are essential for linguistic fluency. Traditional teaching methods often separate manual and non-manual components, limiting learners' ability to grasp the holistic nature of ASL. Here, we introduce a novel Virtual Reality (VR) co-signing game that leverages collaborative learning and cognitive load theories to address these challenges. In the system, two players collaboratively control an avatar: one manages manual signs through hand tracking, while the other controls non-manual facial expressions. The technical framework employs the Meta Quest Pro, utilizing its advanced tracking capabilities, and a web socket server to synchronize user inputs across devices for seamless real-time avatar rendering. This integration of manual and non-manual components provides dynamic feedback, fostering interdependence, reducing cognitive load, and enhancing ASL learning outcomes. We expect our system to enhance ASL proficiency, engagement, and teamwork, while offering a scalable solution for immersive language education.
Augmented Reality (AR) Navigation System for Wheelchair Users
Project team: Anis Idrizović, Md Mamunur Rashid, and Huiran Yu in collaboration with LBS Tech, Korea
Project Supervisor: Daniel Nikolov
Mentors: Kristen Love, JeiriFlores
Abstract
This project presents an app featuring an Augmented Reality (AR) Navigation System designed to assist wheelchair users in navigating the University of Rochester’s campus. There are no commercial or university-supported tools for accessible wheelchair navigation on the University of Rochester campus and beyond, posing significant accessibility challenges. To address this gap, we developed an AR application that suggests wheelchair-accessible routes and provides essential navigation features. Using the Road Scanner app provided by our collaborator, LBS Tech from Korea, we collected detailed information and photographs of accessibility features such as doors, stairs, elevators, and restrooms across key buildings in the University of Rochester's Eastman Quadrangle. This data was synchronized with LBS Tech’s API and integrated into the AR environment using Unity, resulting in a user-friendly app interface. The app includes truly wheelchair-accessible navigation, estimated travel time to destinations, AR-enhanced visualization of building entrances and navigation paths, and voice feedback for hands-free operation. Future work would include extending the scope beyond the Eastman Quadrangle, adding voice prompting and tactile feedback, integrating weather and surface condition tracking, conducting more user testing with diverse representation, integrating adaptive technologies, andincorporating additional facility features.
VR-Enabled Foundational Vision Language Models for Medical Training
Project team: Akhil Kasturi and Adma Gama-Krummel
Project Supervisor: Daniel Nikolov
Advisor: Mudjat Cetin
Abstract
In this work, we have developed a virtual reality (VR) platform aiming to enhance medical training and patient education. The platform transforms traditional MRI brain scans into interactive 3D models, with a focus on brain tumorvisualization. Utilizing a UNETR-based segmentation network, our model achieves 76% accuracy in segmenting complete brain tumors and 81% accuracy in segmenting separate tumor (ROIs) on the Brain Tumor Segmentation(BraTS) 2020 dataset.
The segmented outputs are rendered in 3D using Blender and 3D Slicer and subsequently integrated into a VRenvironment via Unity. This setup enables a volumetric visualization, and an interactive exploration of complexneurological structures present in this data. Additionally, in our system we included a conversational avatar based on fine-tuned large language agentic models (LLM-Agents), where we finetuned a GPT-4O LLM-Agent onn our MRI data and contextual medical knowledge base using a knowledge-augmented transformer (KAT) approach. These avatars provideresponses to questions about brain tumors and specific queries related to the segmentation results and condition of the patient.
Our work aims to fill the gap between theoretical knowledge and clinical application, providing a practical tool formedical education. The initial results suggest that BrainScale VR has significant potential to improve anatomical understanding, procedural training, and patient communication.
