Please consider downloading the latest version of Internet Explorer
to experience this site as intended.
Tools Search Main Menu

With automatic transcription, musicians can save themselves the treble

August 3, 2017
retro travel postcard says SUMMER OF RESEARCH AT THE UNIVERSITY OF ROCHESTER

Summer of Research: Our latest Newscenter series showcases a cross-section of summer research conducted by undergraduates at Rochester and beyond.

Imagine listening to a live jazz improvisation. Moved by what you hear, you decide to try to reproduce the solo on your piano at home. To do so, you would have had to record the piece. Then you would have to listen to it over and over again, and transcribe the notes yourself.

It’s a painstaking process. That’s one reason why Zhiyao Duan, an assistant professor of electrical and computer engineering at the Hajim School, his PhD student Andrea Cogliati, and David Temperley, a professor of music theory at Eastman, have been developing a computer program that could receive audio, extract data from it, and produce an accurate printed musical score.

This summer they’ve have two additional researchers working on the project. Two undergraduate students—Arlen Fan ’18, an electrical and computer engineering major, and rising senior Andrew Smith, a computer science major and music minor at the University of Central Florida—joined the project as part of the NSF-funded Research Experience for Undergraduates (REU) program “Computational Methods for Understanding Music, Media, and Minds.”

The interface would allow users to fix any incorrect portions—such as the red note in this organ music score—of the automatic transcription generated by the computer. Using machine learning techniques, the computer would “learn” these mistakes and avoid similar errors in the future. (University of Rochester image / Duan lab)

 

According to Duan, “Machine transcription is fast, but it isn’t as accurate as a person. Human transcription is accurate, but not very fast. We want to combine these to have both an accurate and fast system.”

Fan and Smith have spent the summer creating an interface that would allow human users to fix portions of the transcription that the computer notated incorrectly and then input these changes into the computer. Using machine learning techniques, the computer would then “learn” from these mistakes in order to avoid similar errors in the future.

Such an automatic music transcription program would help advance music education, retrieval, and research. For example, it would allow users to notate music for which a score is not readily available or to search for music with similar melodies and chord progressions.

 

four people standing around a computer

From left, Arlen Fan ’18, professor Zhiyao Duan, Andrea Cogliati (PhD ’17), and Andrew Smith (University of Central Florida), demonstrating their computer-generated music transcription interface. “Machine transcription is fast, but it isn’t as accurate as a person. Human transcription is accurate, but not very fast,” says Duan. “We want to combine these to have both an accurate and fast system.” (University photo / Lindsey Valich)

“There are musical traditions—not just jazz, but types of Eastern or Indian music—where there is not a written score,” Cogliati says. “The program can bridge the gap between oral tradition and written notation.”

Smith says he is already interested in using the program to produce sheet music for improvisational musicians such as 20th-century comedian and pianist Victor Borge because “you can’t find his sheet music anywhere.”

Reflecting on his decision to work at Rochester this summer, Smith says, “I’m interested in music theory and performance, as well as the technical aspects of computers, so this opportunity fit like a glove.”

Now he’s considering pursuing a graduate degree at Rochester.“Now that I’ve worked in this lab and got to experience how interesting this project is, I might try to apply here next year.”

Tags: , , , ,

Category: The Arts