{"id":573482,"date":"2023-11-09T09:36:11","date_gmt":"2023-11-09T14:36:11","guid":{"rendered":"https:\/\/www.rochester.edu\/newscenter\/?p=573482"},"modified":"2024-04-08T20:42:08","modified_gmt":"2024-04-09T00:42:08","slug":"audio-deepfake-detective-developing-new-sleuthing-techniques-573482","status":"publish","type":"post","link":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/","title":{"rendered":"Audio deepfake detective developing new sleuthing techniques"},"content":{"rendered":"<h2>A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.<\/h2>\n<p>With artificial intelligence-powered audio generation making it increasingly hard to distinguish between real and fake audio, an <a href=\"https:\/\/www.hajim.rochester.edu\/ece\/graduate\/phd.html\">electrical and computer engineering PhD student<\/a> is working to develop tools to protect against scammers. You \u201cNeil\u201d Zhang of the <a href=\"https:\/\/labsites.rochester.edu\/air\/\">Audio Information Research (AIR) Lab<\/a> at the <a href=\"https:\/\/www.rochester.edu\/\">University of Rochester<\/a> received a competitive <a href=\"https:\/\/nij.ojp.gov\/funding\/awards\/15pnij-23-gg-01933-ress\">National Institute of Justice graduate research fellowship<\/a> to develop new audio deepfake detection systems.<\/p>\n<p>Audio deepfakes have <a href=\"https:\/\/www.nytimes.com\/2023\/08\/30\/business\/voice-deepfakes-bank-scams.html\">grabbed headlines<\/a> and caught the attention of cybersecurity experts as scammers increasingly use voice-related AI schemes to target individuals\u2019 bank accounts or spread political disinformation. Zhang expects these attacks to show an increase in scale and complexity.<\/p>\n<p>\u201cAudio deepfake techniques are getting more and more advanced these days,\u201d he says. \u201cPeople are developing new text-to-speech and voice-conversion techniques that criminals might take advantage of to threaten the security of more of our systems. We need to develop more advanced detection systems to keep up.\u201d<\/p>\n<p>Under the guidance of <a href=\"https:\/\/www.hajim.rochester.edu\/ece\/people\/faculty\/duan_zhiyao\/index.html\">Zhiyao Duan<\/a>, an associate professor of electrical and computer engineering and of computer science, Zhang will target the problem from three different angles. First, he intends to create robust algorithms that can spot new and unknown deepfake techniques using a novel training strategy called multicenter one-class learning. The technique would account for different widely used audio recording and compression techniques.<\/p>\n<p>Zhang is also developing watermarking techniques for the audio-generation process that identify the origins of deepfakes. Akin to exploding dye packs that help identify money stolen from banks, these watermarks could be presented as evidence in cases where fraudulent activity took place, according to Zhang.<\/p>\n<p>\u201cCriminals who create these deepfakes typically do not have a very high technology background,\u201d says Zhang. \u201cThey are often using open-source code or an API [application programming interface] provided by a company. So, if the developers who create these algorithms can add a watermark to their systems, experts can identify where deepfakes occur.\u201d<\/p>\n<p>Lastly, Zhang aims to create systems that also use visual information when available to spot telltale signs of deepfakes, such as mismatches on the synchronization of audio-visual cues.<\/p>\n<p>Zhang was part of the first cohort of an <a href=\"https:\/\/www.rochester.edu\/newscenter\/new-training-in-ar-vr-tech-gives-rochester-doctoral-students-an-edge-397062\/\">augmented and virtual reality training program for PhD students<\/a> at the University of Rochester funded through the <a href=\"https:\/\/www.nsf.gov\/funding\/pgm_summ.jsp?pims_id=505015\">National Science Foundation (NSF) Research Traineeship (NRT) program<\/a>. The program was launched to give doctoral students the skills needed to advance AR\/VR technologies and help them gain an appreciation for the broader cultural and societal implications of the technologies.<\/p>\n<p>Beyond the new initiative funded by the NIJ, Zhang and his fellow researchers at the AIR Lab are studying other ways AI voice generation is disrupting established sectors, including the music industry. Through their <a href=\"https:\/\/singfake.org\/\">SingFake project<\/a>, for example, they are attempting to develop detection systems to identify when deepfake techniques are used to create authentic-sounding cover versions of songs sung by other artists.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A National Institute of Justice fellowship allows Rochester graduate student You \u201cNeil\u201d Zhang to develop novel defenses against deepfake scams.<\/p>\n","protected":false},"author":1242,"featured_media":573502,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[116],"tags":[24292,34362,19382,18632,9186,34062],"class_list":["post-573482","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sci-tech","tag-artificial-intelligence","tag-augmented-reality","tag-department-of-electrical-and-computer-engineering","tag-hajim-school-of-engineering-and-applied-sciences","tag-research-funding","tag-virtual-reality"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Audio deepfake detective developing new sleuthing techniques<\/title>\n<meta name=\"description\" content=\"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Audio deepfake detective developing new sleuthing techniques\" \/>\n<meta property=\"og:description\" content=\"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\" \/>\n<meta property=\"og:site_name\" content=\"News Center\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-09T14:36:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-09T00:42:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1050\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Luke Auburn\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Luke Auburn\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\"},\"author\":{\"name\":\"Luke Auburn\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1\"},\"headline\":\"Audio deepfake detective developing new sleuthing techniques\",\"datePublished\":\"2023-11-09T14:36:11+00:00\",\"dateModified\":\"2024-04-09T00:42:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\"},\"wordCount\":495,\"image\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg\",\"keywords\":[\"artificial intelligence\",\"augmented reality\",\"Department of Electrical and Computer Engineering\",\"Hajim School of Engineering and Applied Sciences\",\"research funding\",\"virtual reality\"],\"articleSection\":[\"Science &amp; Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\",\"url\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\",\"name\":\"Audio deepfake detective developing new sleuthing techniques\",\"isPartOf\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg\",\"datePublished\":\"2023-11-09T14:36:11+00:00\",\"dateModified\":\"2024-04-09T00:42:08+00:00\",\"author\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1\"},\"description\":\"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage\",\"url\":\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg\",\"contentUrl\":\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg\",\"width\":2000,\"height\":1200,\"caption\":\"Electrical and computer engineering PhD student You \u201cNeil\u201d Zhang has received a National Institute of Justice fellowship to develop audio deepfake detection systems. (University of Rochester photo \/ J. Adam Fenster)\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.rochester.edu\/newscenter\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Audio deepfake detective developing new sleuthing techniques\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/#website\",\"url\":\"https:\/\/www.rochester.edu\/newscenter\/\",\"name\":\"News Center\",\"description\":\"University of Rochester\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.rochester.edu\/newscenter\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1\",\"name\":\"Luke Auburn\",\"url\":\"https:\/\/www.rochester.edu\/newscenter\/author\/lauburn\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Audio deepfake detective developing new sleuthing techniques","description":"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/","og_locale":"en_US","og_type":"article","og_title":"Audio deepfake detective developing new sleuthing techniques","og_description":"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.","og_url":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/","og_site_name":"News Center","article_published_time":"2023-11-09T14:36:11+00:00","article_modified_time":"2024-04-09T00:42:08+00:00","og_image":[{"url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg","width":1050,"height":630,"type":"image\/jpeg"}],"author":"Luke Auburn","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Luke Auburn","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#article","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/"},"author":{"name":"Luke Auburn","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1"},"headline":"Audio deepfake detective developing new sleuthing techniques","datePublished":"2023-11-09T14:36:11+00:00","dateModified":"2024-04-09T00:42:08+00:00","mainEntityOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/"},"wordCount":495,"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg","keywords":["artificial intelligence","augmented reality","Department of Electrical and Computer Engineering","Hajim School of Engineering and Applied Sciences","research funding","virtual reality"],"articleSection":["Science &amp; Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/","url":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/","name":"Audio deepfake detective developing new sleuthing techniques","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage"},"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg","datePublished":"2023-11-09T14:36:11+00:00","dateModified":"2024-04-09T00:42:08+00:00","author":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1"},"description":"A National Institute of Justice fellowship allows a Rochester graduate student to develop novel defenses against deepfake scams.","breadcrumb":{"@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#primaryimage","url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg","contentUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2023\/11\/fea-Neil-Zhang.jpg","width":2000,"height":1200,"caption":"Electrical and computer engineering PhD student You \u201cNeil\u201d Zhang has received a National Institute of Justice fellowship to develop audio deepfake detection systems. (University of Rochester photo \/ J. Adam Fenster)"},{"@type":"BreadcrumbList","@id":"https:\/\/www.rochester.edu\/newscenter\/audio-deepfake-detective-developing-new-sleuthing-techniques-573482\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.rochester.edu\/newscenter\/"},{"@type":"ListItem","position":2,"name":"Audio deepfake detective developing new sleuthing techniques"}]},{"@type":"WebSite","@id":"https:\/\/www.rochester.edu\/newscenter\/#website","url":"https:\/\/www.rochester.edu\/newscenter\/","name":"News Center","description":"University of Rochester","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.rochester.edu\/newscenter\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1","name":"Luke Auburn","url":"https:\/\/www.rochester.edu\/newscenter\/author\/lauburn\/"}]}},"_links":{"self":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/573482","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/users\/1242"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/comments?post=573482"}],"version-history":[{"count":7,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/573482\/revisions"}],"predecessor-version":[{"id":600852,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/573482\/revisions\/600852"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media\/573502"}],"wp:attachment":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media?parent=573482"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/categories?post=573482"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/tags?post=573482"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}