{"id":87632,"date":"2015-01-29T10:13:02","date_gmt":"2015-01-29T15:13:02","guid":{"rendered":"http:\/\/www.rochester.edu\/newscenter\/?p=87632"},"modified":"2015-08-03T11:24:38","modified_gmt":"2015-08-03T15:24:38","slug":"mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632","status":"publish","type":"post","link":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/","title":{"rendered":"New app would monitor mental health through \u201cselfie\u201d videos, social media"},"content":{"rendered":"

Researchers at the University of Rochester have developed an innovative approach to turn any computer or smartphone with a camera into a personal mental health monitoring device.<\/p>\n

In a paper<\/a> to be presented this week at the American Association for Artificial Intelligence conference in Austin, Texas, Professor of Computer Science Jiebo Luo and his colleagues describe a computer program that can analyze \u201cselfie\u201d videos recorded by a webcam as the person engages with social media.<\/p>\n

Apps to monitor people\u2019s health are widely used, from monitoring the spread of the flu to providing guidance on nutrition and managing mental health issues. Luo explains that his team\u2019s approach is to \u201cquietly observe your behavior\u201d while you use the computer or phone as usual. He adds that their program is \u201cunobtrusive; it does not require the user to explicitly state what he or she is feeling, input any extra information, or wear any special gear.\u201d For example, the team was able to measure a user\u2019s heart rate simply by monitoring very small, subtle changes in the user\u2019s forehead color. The system does not grab other data that might be available through the phone \u2013 such as the user\u2019s location.<\/p>\n

The researchers were able to analyze the video data to extract a number of \u201cclues,\u201d such as heart rate, blinking rate, eye pupil radius, and head movement rate. At the same time, the program also analyzed both what the users posted on Twitter, what they read, how fast they scrolled, their keystroke rate and their mouse click rate. Not every input is treated equally though: what a user tweets, for example, is given more weight than what the user reads because it is a direct expression of what that user is thinking and feeling.<\/p>\n

To calibrate the system and generate a reaction they can measure, Luo explained, he and his colleagues enrolled 27 participants in a test group and \u201csent them messages, real tweets, with sentiment to induce their emotion.\u201d This allowed them to gauge how subjects reacted after seeing or reading material considered to be positive or negative.<\/p>\n

They compared the outcome from all their combined monitoring with the users\u2019 self reports about their feelings to find out how well the program actually performs, and whether it can indeed tell how the user feels. The combination of the data gathered by the program with the users\u2019 self-reported state of mind (called the ground truth) allows the researchers to train the system. The program then begins to understand from just the data gathered whether the user is feeling positive, neutral or negative.<\/p>\n

Their program currently only considers emotions as positive, neutral or negative. Luo says that he hopes to add extra sensitivity to the program by teaching it to further define a negative emotion as, for example, sadness or anger. Right now, this is a demo program they have created and no \u201capp\u201d exists, but they have plans to create an app that would let users be more aware of their emotional fluctuations and make adjustments themselves.<\/p>\n

Luo understands that this program and others that aim to monitor an individual\u2019s mental health or well-being raise ethical concerns that need to be considered. He adds that using this system means \u201ceffectively giving this app permission to observe you constantly,\u201d but adds that the program is designed for the use of the user only and does not share data with anyone else unless otherwise designated by the user.<\/p>\n

Luo\u2019s co-authors on the paper, \u201cTackling Mental Health by Integrating Unobtrusive Multimodal Sensing,\u201d are Dawei Zhou, Jiebo Luo, Vincent Silenzio, Yun Zhou, Glenn Currier, and Henry Kautz. The 29th<\/sup> AAAI Conference on Artificial Intelligence (AAAI) takes place in Austin, Texas, from Jan. 25-30, 2015. The paper can be downloaded here:\u00a0https:\/\/drive.google.com\/file\/d\/0Byh2_bxvelahUzJCZDZNYUlNMXc\/view.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

In a paper to be presented this week at the American Association for Artificial Intelligence conference in Austin, Texas, computer science professor Jiebo Luo and his colleagues describe a computer program that can analyze \u201cselfie\u201d videos recorded by a webcam as the person engages with social media.<\/p>\n","protected":false},"author":6,"featured_media":87722,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[116],"tags":[11716,18802,4626,24202,3696,18572,16072],"class_list":["post-87632","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sci-tech","tag-data-science","tag-department-of-computer-science","tag-featured-post","tag-jiebo-luo","tag-mobile-app","tag-research-finding","tag-school-of-arts-and-sciences"],"acf":[],"yoast_head":"\nNew app would monitor mental health through \u201cselfie\u201d videos, social media<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"New app would monitor mental health through \u201cselfie\u201d videos, social media\" \/>\n<meta property=\"og:description\" content=\"In a paper to be presented this week at the American Association for Artificial Intelligence conference in Austin, Texas, computer science professor Jiebo Luo and his colleagues describe a computer program that can analyze \u201cselfie\u201d videos recorded by a webcam as the person engages with social media.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/\" \/>\n<meta property=\"og:site_name\" content=\"News Center\" \/>\n<meta property=\"article:published_time\" content=\"2015-01-29T15:13:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-03T15:24:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Leonor Sierra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@leonor_sierra\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Leonor Sierra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/\"},\"author\":{\"name\":\"Leonor Sierra\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/b7147819f5697bc51d79e734e5a9efcf\"},\"headline\":\"New app would monitor mental health through \u201cselfie\u201d videos, social media\",\"datePublished\":\"2015-01-29T15:13:02+00:00\",\"dateModified\":\"2015-08-03T15:24:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/\"},\"wordCount\":646,\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2015\\\/01\\\/fea-selfie-study1.jpg\",\"keywords\":[\"data science\",\"Department of Computer Science\",\"featured-post\",\"Jiebo Luo\",\"mobile app\",\"research finding\",\"School of Arts and Sciences\"],\"articleSection\":[\"Science & Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/\",\"name\":\"New app would monitor mental health through \u201cselfie\u201d videos, social media\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2015\\\/01\\\/fea-selfie-study1.jpg\",\"datePublished\":\"2015-01-29T15:13:02+00:00\",\"dateModified\":\"2015-08-03T15:24:38+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/b7147819f5697bc51d79e734e5a9efcf\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2015\\\/01\\\/fea-selfie-study1.jpg\",\"contentUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2015\\\/01\\\/fea-selfie-study1.jpg\",\"width\":1000,\"height\":600,\"caption\":\"multiple screenshots of man making faces into webcam\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"New app would monitor mental health through \u201cselfie\u201d videos, social media\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\",\"name\":\"News Center\",\"description\":\"University of Rochester\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/b7147819f5697bc51d79e734e5a9efcf\",\"name\":\"Leonor Sierra\",\"description\":\"Leonor Sierra is press officer for science and engineering. She covers computer science, electrical and computer engineering, laboratory for laser energetics, optics, mechanical engineering, physics and astronomy, and the Office of the Dean of Engineering and Applied Sciences.\",\"sameAs\":[\"https:\\\/\\\/x.com\\\/leonor_sierra\"],\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/author\\\/lsierra\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"New app would monitor mental health through \u201cselfie\u201d videos, social media","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/","og_locale":"en_US","og_type":"article","og_title":"New app would monitor mental health through \u201cselfie\u201d videos, social media","og_description":"In a paper to be presented this week at the American Association for Artificial Intelligence conference in Austin, Texas, computer science professor Jiebo Luo and his colleagues describe a computer program that can analyze \u201cselfie\u201d videos recorded by a webcam as the person engages with social media.","og_url":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/","og_site_name":"News Center","article_published_time":"2015-01-29T15:13:02+00:00","article_modified_time":"2015-08-03T15:24:38+00:00","og_image":[{"url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg","width":1000,"height":600,"type":"image\/jpeg"}],"author":"Leonor Sierra","twitter_card":"summary_large_image","twitter_creator":"@leonor_sierra","twitter_misc":{"Written by":"Leonor Sierra","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#article","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/"},"author":{"name":"Leonor Sierra","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/b7147819f5697bc51d79e734e5a9efcf"},"headline":"New app would monitor mental health through \u201cselfie\u201d videos, social media","datePublished":"2015-01-29T15:13:02+00:00","dateModified":"2015-08-03T15:24:38+00:00","mainEntityOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/"},"wordCount":646,"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg","keywords":["data science","Department of Computer Science","featured-post","Jiebo Luo","mobile app","research finding","School of Arts and Sciences"],"articleSection":["Science & Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/","url":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/","name":"New app would monitor mental health through \u201cselfie\u201d videos, social media","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#primaryimage"},"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg","datePublished":"2015-01-29T15:13:02+00:00","dateModified":"2015-08-03T15:24:38+00:00","author":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/b7147819f5697bc51d79e734e5a9efcf"},"breadcrumb":{"@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#primaryimage","url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg","contentUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2015\/01\/fea-selfie-study1.jpg","width":1000,"height":600,"caption":"multiple screenshots of man making faces into webcam"},{"@type":"BreadcrumbList","@id":"https:\/\/www.rochester.edu\/newscenter\/mental-health-monitoring-through-selfie-videos-and-social-media-tracking-87632\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.rochester.edu\/newscenter\/"},{"@type":"ListItem","position":2,"name":"New app would monitor mental health through \u201cselfie\u201d videos, social media"}]},{"@type":"WebSite","@id":"https:\/\/www.rochester.edu\/newscenter\/#website","url":"https:\/\/www.rochester.edu\/newscenter\/","name":"News Center","description":"University of Rochester","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.rochester.edu\/newscenter\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/b7147819f5697bc51d79e734e5a9efcf","name":"Leonor Sierra","description":"Leonor Sierra is press officer for science and engineering. She covers computer science, electrical and computer engineering, laboratory for laser energetics, optics, mechanical engineering, physics and astronomy, and the Office of the Dean of Engineering and Applied Sciences.","sameAs":["https:\/\/x.com\/leonor_sierra"],"url":"https:\/\/www.rochester.edu\/newscenter\/author\/lsierra\/"}]}},"_links":{"self":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/87632","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/comments?post=87632"}],"version-history":[{"count":2,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/87632\/revisions"}],"predecessor-version":[{"id":113522,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/87632\/revisions\/113522"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media\/87722"}],"wp:attachment":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media?parent=87632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/categories?post=87632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/tags?post=87632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}