A virtual reality app (VR) uses advanced display and immersive audio technologies to create an interactive, three-dimensional image or environment. An augmented reality app (AR), meanwhile, uses digital technology to overlay video and audio onto the physical world to provide information and embellish our experiences.
At the University of Rochester, we’re crossing disciplines to collaborate on VR/AR innovations that will revolutionize how we learn, discover, heal, and create as we work to make the world ever better.
What are new, innovative ways to increase community access to much-needed mental and behavioral health services?
That’s the big question Michael Hasselberg’s department chair asked him to ponder. Hasselberg, an assistant professor of psychiatry and clinical nursing, is acutely aware of the high demand, regionally and nationally, for behavioral health services and specialists to treat conditions like depression, anxiety, substance abuse, and eating disorders.
In this line of work, the gold standard treatment is cognitive-behavioral therapy. CBT is a form of psychotherapy that usually involves a therapist and patient working together to recognize and change the latter’s thinking patterns while also developing coping skills, according to the American Psychological Association. “It’s our first-line treatment for several of the diagnostic clusters we treat, specifically the depressive and anxiety disorders, and there’s a lot of evidence to show its effectiveness,” Hasselberg explains.
Given that a majority of Americans today own a smartphone—77 percent, according to the Pew Research Center—a web or mobile app seems like an obvious answer. Indeed, such apps abound.
“The idea of putting cognitive-behavioral therapy on a computerized app isn’t new,” Hasselberg says. Typically, such computerized cognitive-behavioral therapy (cCBT) features a voiceover with animation or perhaps a video of a therapist talking—but no meaningful interaction.
“Patients download the app, but after using it once or twice, they disengage from it.”
Virtual reality app offer cognitive-behavioral therapy
Could a more immersive, customized experience, one harnessing today’s advances in virtual reality and augmented reality (VR/AR), help keep patients engaged with an app?
Hasselberg pitched the idea a year ago during a meeting of the UR Health Lab, a multidisciplinary campus collaboration that leverages computing, analytics, and technology to solve challenges in medicine. Afterward, two faculty members approached him: Jiebo Luo, professor of computer science in the Hajim School of Engineering and Applied Sciences, and Matthew Brown, professor of music theory at the Eastman School of Music.
Luo, whose research interests include machine learning, data mining, and personal analytics, has previously worked on a computer program that could monitor mental health by analyzing “selfie” videos and social media posts. Brown, meanwhile, directs TableTopOpera, a musical multimedia ensemble at Eastman, and has always “been very interested in the social impact and therapeutic functions of music.”
Both were keen to bring their expertise to bear on such a project. Galvanized, Hasselberg tapped his colleague Wendi Cross, a clinical psychologist and associate professor in the Department of Psychiatry, to help develop evidence-based scripts and storyboards for a working prototype of the virtual reality app.
Cross immediately thought ahead to how patients would use such an app. “In a typical psychotherapy situation, you’re interacting in the office for 45 minutes,” she explains. “Well, nobody is going to stay online in an app for 45 minutes—people like to come and go in ways that suit them. With this virtual reality app, the psychotherapy session is literally in the hands of the patient.”
According to Christopher Winders, a laboratory technician in the Department of Electrical and Computer Engineering, “These videos are being embedded into the mobile app and have our doctor [Lauren DeCaporale-Ryan, a clinical psychologist at URMC] laying out tenants of the CBT process. We filmed Lauren in front of a green screen and then I composited that footage into a 360-photograph of the office setting.”
A multidisciplinary collaboration yields a working prototype
Here’s how the virtual reality app works:
- Patients use their smartphones to access the app. They can see and hear the therapist, but can also look around the therapist’s room, which was filmed using a virtual reality camera.
- During a session, the therapist might assign the patient a task, such as completing a questionnaire about anxiety levels.
- The patient completes the in-app questionnaire. The answers dictate which video of the therapist plays next.
Later, the therapist might suggest a mindfulness exercise in which the patient is asked to imagine being in a relaxing environment, such as a beach. - The patient is then shown environmental photographs and asked to select a preference. Earlier in the module, the patient was able to identify personal preferences for calming music.
- The app’s algorithms then serve up the patient’s preferred 3D environment (a beach, complete with the sound of waves) and accompanying music. Meanwhile, the virtual therapist talks the patient through the mindfulness exercise.
The next time the patient uses the app for an on-demand therapy session, the previous answers along with visual and aural preferences will be taken into account.
Evidence-based psychotherapeutic practices? Check.
Aesthetically pleasing and immersive video and music? Check.
Built-in algorithms that deliver personalized content based on your needs and preferences? Check.
And no special headsets required. “If people have to use headsets like Oculus or even the cardboard headsets for mobile phones, that really limits the patient population we could reach with this technology,” says Hasselberg. “And our goal is to touch as many patient lives as possible.”
With a working prototype in hand, the team’s next step is conducting usability testing to fine-tune the technology. If this intervention proves useful with patient populations, the big-picture goal is having the in-app data synch with a patient’s electronic health records.
‘A common purpose is so important’
With funding from Arts, Sciences & Engineering and the Medical Center, “we formed a multidisciplinary team to create something that is otherwise not possible,” says Luo.
“I can’t tell you how much I’ve enjoyed working with colleagues toward a common purpose, one that will have a social impact,” says Brown. Cross agrees: “A common purpose is so important. Frankly, this project has been an adventure. It’s not what I do every day, but it’s taking what I do every day and thinking about it in a brand new way with people I don’t usually work with.”
And Hasselberg contends that collaborating with researchers and professionals outside of medicine—musicians, computer scientists, data scientists—has made him into a better clinician and health care innovator. “In the end, I think bringing these multiple perspectives to a problem will allow us to help more people.”
