A virtual reality app (VR) uses advanced display and immersive audio technologies to create an interactive, three-dimensional image or environment. An augmented reality app (AR), meanwhile, uses digital technology to overlay video and audio onto the physical world to provide information and embellish our experiences. At the University of Rochester, we’re crossing disciplines to collaborate on VR/AR innovations that will revolutionize how we learn, discover, heal, and create as we work to make the world ever better.
What are new, innovative ways to increase community access to much-needed mental and behavioral health services? That’s the big question Michael Hasselberg’s department chair asked him to ponder. Hasselberg, an assistant professor of psychiatry and clinical nursing, is acutely aware of the high demand, regionally and nationally, for behavioral health services and specialists to treat conditions like depression, anxiety, substance abuse, and eating disorders. In this line of work, the gold standard treatment is cognitive-behavioral therapy. CBT is a form of psychotherapy that usually involves a therapist and patient working together to recognize and change the latter’s thinking patterns while also developing coping skills, according to the American Psychological Association. “It’s our first-line treatment for several of the diagnostic clusters we treat, specifically the depressive and anxiety disorders, and there’s a lot of evidence to show its effectiveness,” Hasselberg explains. Given that a majority of Americans today own a smartphone—77 percent, according to the Pew Research Center—a web or mobile app seems like an obvious answer. Indeed, such apps abound. “The idea of putting cognitive-behavioral therapy on a computerized app isn’t new,” Hasselberg says. Typically, such computerized cognitive-behavioral therapy (cCBT) features a voiceover with animation or perhaps a video of a therapist talking—but no meaningful interaction. “Patients download the app, but after using it once or twice, they disengage from it.”
Virtual reality app offer cognitive-behavioral therapy
Could a more immersive, customized experience, one harnessing today’s advances in virtual reality and augmented reality (VR/AR), help keep patients engaged with an app? Hasselberg pitched the idea a year ago during a meeting of the UR Health Lab, a multidisciplinary campus collaboration that leverages computing, analytics, and technology to solve challenges in medicine. Afterward, two faculty members approached him: Jiebo Luo, professor of computer science in the Hajim School of Engineering and Applied Sciences, and Matthew Brown, professor of music theory at the Eastman School of Music. Luo, whose research interests include machine learning, data mining, and personal analytics, has previously worked on a computer program that could monitor mental health by analyzing “selfie” videos and social media posts. Brown, meanwhile, directs TableTopOpera, a musical multimedia ensemble at Eastman, and has always “been very interested in the social impact and therapeutic functions of music.” Both were keen to bring their expertise to bear on such a project. Galvanized, Hasselberg tapped his colleague Wendi Cross, a clinical psychologist and associate professor in the Department of Psychiatry, to help develop evidence-based scripts and storyboards for a working prototype of the virtual reality app. Cross immediately thought ahead to how patients would use such an app. “In a typical psychotherapy situation, you’re interacting in the office for 45 minutes,” she explains. “Well, nobody is going to stay online in an app for 45 minutes—people like to come and go in ways that suit them. With this virtual reality app, the psychotherapy session is literally in the hands of the patient.” According to Christopher Winders, a laboratory technician in the Department of Electrical and Computer Engineering, “These videos are being embedded into the mobile app and have our doctor [Lauren DeCaporale-Ryan, a clinical psychologist at URMC] laying out tenants of the CBT process. We filmed Lauren in front of a green screen and then I composited that footage into a 360-photograph of the office setting.”A multidisciplinary collaboration yields a working prototype
Here’s how the virtual reality app works:- Patients use their smartphones to access the app. They can see and hear the therapist, but can also look around the therapist’s room, which was filmed using a virtual reality camera.
- During a session, the therapist might assign the patient a task, such as completing a questionnaire about anxiety levels.
- The patient completes the in-app questionnaire. The answers dictate which video of the therapist plays next. Later, the therapist might suggest a mindfulness exercise in which the patient is asked to imagine being in a relaxing environment, such as a beach.
- The patient is then shown environmental photographs and asked to select a preference. Earlier in the module, the patient was able to identify personal preferences for calming music.
- The app’s algorithms then serve up the patient’s preferred 3D environment (a beach, complete with the sound of waves) and accompanying music. Meanwhile, the virtual therapist talks the patient through the mindfulness exercise. The next time the patient uses the app for an on-demand therapy session, the previous answers along with visual and aural preferences will be taken into account.