{"id":649992,"date":"2025-05-03T17:40:22","date_gmt":"2025-05-03T21:40:22","guid":{"rendered":"https:\/\/www.rochester.edu\/newscenter\/?p=649992"},"modified":"2025-05-06T09:15:58","modified_gmt":"2025-05-06T13:15:58","slug":"ai-text-to-video-ai-metamorphic-capabilities-649992","status":"publish","type":"post","link":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/","title":{"rendered":"Text-to-video AI blossoms with new metamorphic video capabilities"},"content":{"rendered":"

Using time-lapse videos as training data, computer scientists have developed video generators that simulate the physical world more accurately.<\/strong><\/h2>\n

While text-to-video artificial intelligence models like OpenAI\u2019s Sora are rapidly metamorphosing in front of our eyes, they have struggled to produce metamorphic videos. Simulating a tree sprouting or a flower blooming is harder for AI systems than generating other types of videos because it requires the knowledge of the physical world and can vary widely.<\/p>\n

But now, these models have taken an evolutionary step.<\/p>\n

Computer scientists at the University of Rochester<\/a>, Peking University, University of California, Santa Cruz, and National University of Singapore developed a new AI text-to-video model that learns real-world physics knowledge from time-lapse videos. The team outlines their model, MagicTime, in a paper<\/a> published in IEEE Transactions on Pattern Analysis and Machine Intelligence<\/em>.<\/p>\n

\u201cArtificial intelligence has been developed to try to understand the real world and to simulate the activities and events that take place,\u201d says Jinfa Huang<\/a>, a PhD student supervised by Professor Jiebo\u00a0Luo<\/a>\u00a0from Rochester\u2019s\u00a0Department of Computer Science<\/a>, both of whom are among the paper\u2019s authors. \u201cMagicTime is a step toward AI that can better simulate the physical, chemical, biological, or social properties of the world around us.\u201d<\/p>\n

Previous models generated videos that typically have limited motion and poor variations. To train AI models to more effectively mimic metamorphic processes, the researchers developed a high-quality dataset of more than 2,000 time-lapse videos with detailed captions.<\/p>\n

Currently, the open-source U-Net version of MagicTime<\/a> generates two-second, 512\u2009-by-\u2009512-pixel clips (at 8 frames per second), and an accompanying diffusion-transformer architecture extends this to ten-second clips. The model can be used to simulate not only biological metamorphosis but also buildings undergoing construction or bread baking in the oven.<\/p>\n

But while the videos generated are visually interesting and the demo can be fun to play with, the researchers view this as an important step toward more sophisticated models that could provide important tools for scientists.<\/p>\n

\u201cOur hope is that someday, for example, biologists could use generative video to speed up preliminary exploration of ideas,\u201d says Huang. \u201cWhile physical experiments remain indispensable for final verification, accurate simulations can shorten iteration cycles and reduce the number of live trials needed.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Using time-lapse videos as training data, computer scientists have developed video generators that simulate the physical world more accurately.<\/p>\n","protected":false},"author":1242,"featured_media":650892,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[116],"tags":[24292,18802,18632,24202,18572],"class_list":["post-649992","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sci-tech","tag-artificial-intelligence","tag-department-of-computer-science","tag-hajim-school-of-engineering-and-applied-sciences","tag-jiebo-luo","tag-research-finding"],"acf":[],"yoast_head":"\nText-to-video AI blossoms with new metamorphic video capabilities<\/title>\n<meta name=\"description\" content=\"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Text-to-video AI blossoms with new metamorphic video capabilities\" \/>\n<meta property=\"og:description\" content=\"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/\" \/>\n<meta property=\"og:site_name\" content=\"News Center\" \/>\n<meta property=\"article:published_time\" content=\"2025-05-03T21:40:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-06T13:15:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions-1200x630.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"author\" content=\"Luke Auburn\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Luke Auburn\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/\"},\"author\":{\"name\":\"Luke Auburn\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e928dc2863b53a89ece6d40c7992a4e1\"},\"headline\":\"Text-to-video AI blossoms with new metamorphic video capabilities\",\"datePublished\":\"2025-05-03T21:40:22+00:00\",\"dateModified\":\"2025-05-06T13:15:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/\"},\"wordCount\":378,\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/fea-text-to-video-ai-dandelions.gif\",\"keywords\":[\"artificial intelligence\",\"Department of Computer Science\",\"Hajim School of Engineering and Applied Sciences\",\"Jiebo Luo\",\"research finding\"],\"articleSection\":[\"Science & Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/\",\"name\":\"Text-to-video AI blossoms with new metamorphic video capabilities\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/fea-text-to-video-ai-dandelions.gif\",\"datePublished\":\"2025-05-03T21:40:22+00:00\",\"dateModified\":\"2025-05-06T13:15:58+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e928dc2863b53a89ece6d40c7992a4e1\"},\"description\":\"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/fea-text-to-video-ai-dandelions.gif\",\"contentUrl\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/fea-text-to-video-ai-dandelions.gif\",\"width\":1869,\"height\":1119,\"caption\":\"MIGHTY MORPHING: \u201cMagicTime is a step toward AI that can better simulate the physical, chemical, biological, or social properties of the world around us,\u201d says computer science PhD student Jinfa Huang. (University of Rochester GIF created using MagicTime)\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/ai-text-to-video-ai-metamorphic-capabilities-649992\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Text-to-video AI blossoms with new metamorphic video capabilities\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#website\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/\",\"name\":\"News Center\",\"description\":\"University of Rochester\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/#\\\/schema\\\/person\\\/e928dc2863b53a89ece6d40c7992a4e1\",\"name\":\"Luke Auburn\",\"url\":\"https:\\\/\\\/www.rochester.edu\\\/newscenter\\\/author\\\/lauburn\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Text-to-video AI blossoms with new metamorphic video capabilities","description":"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/","og_locale":"en_US","og_type":"article","og_title":"Text-to-video AI blossoms with new metamorphic video capabilities","og_description":"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.","og_url":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/","og_site_name":"News Center","article_published_time":"2025-05-03T21:40:22+00:00","article_modified_time":"2025-05-06T13:15:58+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions-1200x630.gif","type":"image\/gif"}],"author":"Luke Auburn","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Luke Auburn","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#article","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/"},"author":{"name":"Luke Auburn","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1"},"headline":"Text-to-video AI blossoms with new metamorphic video capabilities","datePublished":"2025-05-03T21:40:22+00:00","dateModified":"2025-05-06T13:15:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/"},"wordCount":378,"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions.gif","keywords":["artificial intelligence","Department of Computer Science","Hajim School of Engineering and Applied Sciences","Jiebo Luo","research finding"],"articleSection":["Science & Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/","url":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/","name":"Text-to-video AI blossoms with new metamorphic video capabilities","isPartOf":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#primaryimage"},"image":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#primaryimage"},"thumbnailUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions.gif","datePublished":"2025-05-03T21:40:22+00:00","dateModified":"2025-05-06T13:15:58+00:00","author":{"@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1"},"description":"Using time-lapse videos as training data, computer scientists have developed text-to-video generators that simulate the physical world more accurately.","breadcrumb":{"@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#primaryimage","url":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions.gif","contentUrl":"https:\/\/www.rochester.edu\/newscenter\/wp-content\/uploads\/2025\/04\/fea-text-to-video-ai-dandelions.gif","width":1869,"height":1119,"caption":"MIGHTY MORPHING: \u201cMagicTime is a step toward AI that can better simulate the physical, chemical, biological, or social properties of the world around us,\u201d says computer science PhD student Jinfa Huang. (University of Rochester GIF created using MagicTime)"},{"@type":"BreadcrumbList","@id":"https:\/\/www.rochester.edu\/newscenter\/ai-text-to-video-ai-metamorphic-capabilities-649992\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.rochester.edu\/newscenter\/"},{"@type":"ListItem","position":2,"name":"Text-to-video AI blossoms with new metamorphic video capabilities"}]},{"@type":"WebSite","@id":"https:\/\/www.rochester.edu\/newscenter\/#website","url":"https:\/\/www.rochester.edu\/newscenter\/","name":"News Center","description":"University of Rochester","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.rochester.edu\/newscenter\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.rochester.edu\/newscenter\/#\/schema\/person\/e928dc2863b53a89ece6d40c7992a4e1","name":"Luke Auburn","url":"https:\/\/www.rochester.edu\/newscenter\/author\/lauburn\/"}]}},"_links":{"self":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/649992","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/users\/1242"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/comments?post=649992"}],"version-history":[{"count":5,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/649992\/revisions"}],"predecessor-version":[{"id":651512,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/posts\/649992\/revisions\/651512"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media\/650892"}],"wp:attachment":[{"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/media?parent=649992"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/categories?post=649992"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rochester.edu\/newscenter\/wp-json\/wp\/v2\/tags?post=649992"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}