{"id":297772,"date":"2018-02-13T14:24:22","date_gmt":"2018-02-13T19:24:22","guid":{"rendered":"http:\/\/www.rochester.edu\/newscenter\/?p=297772"},"modified":"2018-06-01T12:52:07","modified_gmt":"2018-06-01T16:52:07","slug":"intersections-better-audio-vr-videos","status":"publish","type":"post","link":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/","title":{"rendered":"Giving virtual reality a \u2018visceral\u2019 sound"},"content":{"rendered":"
\n

Virtual reality (VR) uses advanced display and immersive audio technologies to create an interactive, three-dimensional image or environment. Augmented reality (AR), meanwhile, uses digital technology to overlay video and audio onto the physical world to provide information and embellish our experiences.<\/em><\/p>\n

At the University of Rochester, we\u2019re crossing disciplines to collaborate on VR\/AR innovations that will revolutionize how we learn<\/a>, discover<\/a>, heal<\/a>, and create as we work to make the world ever better.<\/em><\/p>\n

\n

\"logo<\/a>Imagine what it would be like to sit, not in the audience, but in the midst of an orchestra during a rehearsal or concert. To experience firsthand the sounds of string, brass, woodwind, and percussion sections washing over you from every direction. To see the conductor, not from behind, but pointing the baton directly at you.<\/p>\n

Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University of Rochester researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.<\/p>\n

\u201cInstead of just watching a movie of the concert, you can put headphones on and walk around among the performers,\u201d says Matthew Brown, a professor of music theory at the University\u2019s Eastman School of Music<\/a>, and one of the researchers on the project. \u201cThe more you get inside the performance, the more visceral and exciting the music becomes. This is very different from the normal video you see on YouTube.\u201d<\/p>\n

The quality of the sound, of course, is at least as important as the quality of the images. And that\u2019s where this collaboration hopes to make a major contribution.<\/p>\n

\u201cPrevious AR\/VR research has focused mainly on the visual side,\u201d says Zhiyao Duan, an assistant professor of electrical and computer engineering<\/a>, who is working on the project with Brown and Ming-Lun Lee, an assistant professor of audio and music engineering.<\/a> \u201cYou see the 3-D, but you don\u2019t really hear the audio change when you move your head or look at different objects. The purpose of this proposal is adding 3-D sound. That\u2019s what\u2019s novel here.\u201d<\/p>\n

Duan\u2019s expertise is in designing intelligent algorithms and systems that can understand sounds (including music) and be applied, for example, to audio-visual analysis of music performances. Meanwhile, Lee, who has degrees in both electrical engineering and musicology, is an expert in audio software programming and sound recording techniques and technology. He is also an active choral director and baritone.<\/p>\n

\u201cThat\u2019s the strength of this project, because in addition to having access to Eastman and its students, we\u2019ve got the expertise in electrical and computer engineering, and audio and music engineering,\u201d says Brown. Also part of the team is Christopher Winders, who has a PhD in composition from Eastman and helped Brown pioneer TableTopOpera<\/a>, a music ensemble that produces innovative multimedia performances, including \u201ccomic book operas.\u201d<\/p>\n

\"three
From left, Ming-Lun Lee, Zhiyao Duan, and Matthew Brown look at sound level readouts from a microphone in a sound studio they’ve set up in the Computer Studies Building. (University of Rochester photo \/ Bob Marcotte)<\/figcaption><\/figure>\n

With funding from the University\u2019s AR\/VR Initiative, the team has obtained cutting-edge recording equipment, including an Eigenmike 32-channel spherical microphone array. This past semester, they were able to make 11 recordings of Eastman ensembles in live concerts, as well as in rehearsals, where there is greater freedom to experiment.<\/p>\n

Much of the effort so far has focused on perfecting what Lee refers to as the \u201cworkflow.\u201d For example, \u201cWhere do we want to place a microphone? At the best seat in the concert hall? Or do you want to put it over the conductor\u2019s head?\u201d asks Lee. The answer may vary depending on the acoustics of the particular hall, and the size and type of ensemble.<\/p>\n

Eventually Duan will develop algorithms to best integrate the audio and visual files.<\/p>\n

The researchers envision a host of possible applications, from enhanced listener enjoyment to feedback for performers, live streaming of University concerts, and music therapy for patients.<\/p>\n

\u201cWe read stories that classical music is losing its relevance,\u201d Brown says. \u201cThis is a way to make the experience of classical music more exciting than just sitting and watching a bunch of people in penguin suits performing. The idea that you could interact with the environment, and move around in it, strikes me as being a really powerful way for us to make the music more relevant to younger audiences.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.<\/p>\n","protected":false},"author":286,"featured_media":298252,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[116],"tags":[31972,34362,19382,3286,18632,34272,34062,30532],"class_list":["post-297772","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sci-tech","tag-audio-and-music-engineering","tag-augmented-reality","tag-department-of-electrical-and-computer-engineering","tag-eastman-school-of-music","tag-hajim-school-of-engineering-and-applied-sciences","tag-interdisciplinary","tag-virtual-reality","tag-zhiyao-duan"],"acf":[],"yoast_head":"\nGiving virtual reality a \u2018visceral\u2019 sound<\/title>\n<meta name=\"description\" content=\"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Giving virtual reality a \u2018visceral\u2019 sound\" \/>\n<meta property=\"og:description\" content=\"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/\" \/>\n<meta property=\"og:site_name\" content=\"News Center\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-13T19:24:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-06-01T16:52:07+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Bob Marcotte\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bob Marcotte\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/\"},\"author\":{\"name\":\"Bob Marcotte\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e0d8d271cd290d592461fa9cefca013b\"},\"headline\":\"Giving virtual reality a \u2018visceral\u2019 sound\",\"datePublished\":\"2018-02-13T19:24:22+00:00\",\"dateModified\":\"2018-06-01T16:52:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/\"},\"wordCount\":752,\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2018\\\/02\\\/fea-augmented-reality-sound.jpg\",\"keywords\":[\"Audio and Music Engineering\",\"augmented reality\",\"Department of Electrical and Computer Engineering\",\"Eastman School of Music\",\"Hajim School of Engineering and Applied Sciences\",\"interdisciplinary\",\"virtual reality\",\"Zhiyao Duan\"],\"articleSection\":[\"Science & Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/\",\"name\":\"Giving virtual reality a \u2018visceral\u2019 sound\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2018\\\/02\\\/fea-augmented-reality-sound.jpg\",\"datePublished\":\"2018-02-13T19:24:22+00:00\",\"dateModified\":\"2018-06-01T16:52:07+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e0d8d271cd290d592461fa9cefca013b\"},\"description\":\"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2018\\\/02\\\/fea-augmented-reality-sound.jpg\",\"contentUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2018\\\/02\\\/fea-augmented-reality-sound.jpg\",\"width\":1000,\"height\":600,\"caption\":\"Most work to date on VR\\\/AR technologies has focused on the 3-D visuals, says Zhiyao Duan, an assistant professor of electrical and computer engineering. University of Rochester researchers and Eastman School of Music musicians are working together to make the quality of the audio experience as high as the visuals. (Getty Images photo)\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/intersections-better-audio-vr-videos\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Giving virtual reality a \u2018visceral\u2019 sound\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\",\"name\":\"News Center\",\"description\":\"University of Rochester\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e0d8d271cd290d592461fa9cefca013b\",\"name\":\"Bob Marcotte\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/author\\\/bmarcotte\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Giving virtual reality a \u2018visceral\u2019 sound","description":"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/","og_locale":"en_US","og_type":"article","og_title":"Giving virtual reality a \u2018visceral\u2019 sound","og_description":"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.","og_url":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/","og_site_name":"News Center","article_published_time":"2018-02-13T19:24:22+00:00","article_modified_time":"2018-06-01T16:52:07+00:00","og_image":[{"url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg","width":1000,"height":600,"type":"image\/jpeg"}],"author":"Bob Marcotte","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bob Marcotte","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#article","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/"},"author":{"name":"Bob Marcotte","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e0d8d271cd290d592461fa9cefca013b"},"headline":"Giving virtual reality a \u2018visceral\u2019 sound","datePublished":"2018-02-13T19:24:22+00:00","dateModified":"2018-06-01T16:52:07+00:00","mainEntityOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/"},"wordCount":752,"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg","keywords":["Audio and Music Engineering","augmented reality","Department of Electrical and Computer Engineering","Eastman School of Music","Hajim School of Engineering and Applied Sciences","interdisciplinary","virtual reality","Zhiyao Duan"],"articleSection":["Science & Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/","url":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/","name":"Giving virtual reality a \u2018visceral\u2019 sound","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#primaryimage"},"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg","datePublished":"2018-02-13T19:24:22+00:00","dateModified":"2018-06-01T16:52:07+00:00","author":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e0d8d271cd290d592461fa9cefca013b"},"description":"Using recital halls as their \u201clabs,\u201d and recording some of the best music students in the world, University researchers are creating virtual reality videos of concerts that literally immerse viewers \u201cwithin\u201d the performance onstage.","breadcrumb":{"@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#primaryimage","url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg","contentUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2018\/02\/fea-augmented-reality-sound.jpg","width":1000,"height":600,"caption":"Most work to date on VR\/AR technologies has focused on the 3-D visuals, says Zhiyao Duan, an assistant professor of electrical and computer engineering. University of Rochester researchers and Eastman School of Music musicians are working together to make the quality of the audio experience as high as the visuals. (Getty Images photo)"},{"@type":"BreadcrumbList","@id":"https:\/\/www.rochester.edu\/newscenter\/intersections-better-audio-vr-videos\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.rochester.edu\/newscenter\/"},{"@type":"ListItem","position":2,"name":"Giving virtual reality a \u2018visceral\u2019 sound"}]},{"@type":"WebSite","@id":"https:\/\/www.rochester.edu\/newscenter\/#website","url":"https:\/\/www.rochester.edu\/newscenter\/","name":"News Center","description":"University of Rochester","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.rochester.edu\/newscenter\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e0d8d271cd290d592461fa9cefca013b","name":"Bob Marcotte","url":"https:\/\/www.rochester.edu\/newscenter\/author\/bmarcotte\/"}]}},"_links":{"self":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/297772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/users\/286"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/comments?post=297772"}],"version-history":[{"count":7,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/297772\/revisions"}],"predecessor-version":[{"id":300092,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/297772\/revisions\/300092"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media\/298252"}],"wp:attachment":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media?parent=297772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/categories?post=297772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/tags?post=297772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}