Rochester engineers have discovered a way to obtain the clearest still pictures yet from video. By using information from surrounding video frames to boost a single frame's resolution dramatically, the team of University of Rochester engineers and scientists at Eastman Kodak have taken the fuzz out of single frames distilled from video.
"The one image you want is related to all the images before and after it," says graduate student Andrew Patti. "We extract that information and use it to help clarify our image." Patti and his Kodak colleague and co-adviser, Ibrahim Sezan, have filed four patents on it and similar processes.
The work will have the most impact in video filled with quick motion, such as sporting events or action movies, says Patti's adviser, Murat Tekalp. "Let's say you want to print a football scene from the TV. It might be impossible to tell whether the receiver's foot was in bounds because his foot is blurry and the line is jagged. This would clear that up, almost like magic," he says.
To increase resolution, Patti and colleagues compensate for motion in a new way: They take into account not only the motion between frames (a widespread practice), but also the motion that takes place while an image is captured. For instance, if a video camera snags one frame by opening and closing in 1/100th of a second, the subject probably moved a tiny bit during that time. Such tiny motions might not seem like much, but Patti and colleagues have shown that taking such motion into account boosts picture quality greatly when enlarging an image or zooming in on one feature.
The work promises to further erode the barriers between TV, computers, and video. For instance, it's now possible to tape a television show on the VCR, play it back on computer, grab a few special images and print them out on the color printer. Or one can use a camcorder to videotape the family, scan the images into the computer, crop and retouch them, then print them out as holiday cards for family and friends.
But often the images lack clarity because the software instructions that make it so easy to click on and manipulate images don't have the information they need to reproduce them with high resolution. For example, on a television an entire image never really appears. Instead, images are interlaced -- half of an image appears as a set of lines, and then 1/60th of a second later the other half appears. This happens so quickly that we don't notice, until we try to see or print a single frame. Then, straight lines appear crooked, colors can be mixed, and detail is lost.
Most of today's printers, VCRs and other devices take the two sets of interlaced lines and average them, filling in the missing lines. But those images are usually fuzzy. Some devices compare the desired frame to those immediately before and after, and compensate for the motion. But parts of the image, especially straight lines, are still fuzzy and jagged. The work by the Rochester team removes that fuzziness.
Patti's work also has applications in the future, when high- definition TVs become a fixture in the home. Current TV programs in the U.S. and Japan flash by at 60 frames per second, while in Europe it's 50. A motion picture shows you a new scene only 24 times a second. To show an American TV show in Europe, or a motion picture on any TV, requires elaborate ways to convert the rate at which the images speed by. These conversions often produce some fuzziness and jagged edges which current TVs don't pick up but which are visible on HDTV screens. Patti's formula prevents this fuzziness from occurring, assuring that HDTV customers could see domestic and foreign films and TV shows with the same clarity. The formula can also be used in other applications such as forensics or satellite imaging -- anywhere someone wants to see one clear frame from a video.
Patti will graduate this summer and has accepted a position with Hewlett Packard. He is part of a digital video processing group headed by Tekalp, associate professor of electrical engineering, who has just completed the first book on Digital Video Processing, to be published later this year by Prentice- Hall. Tekalp's group is funded by Kodak, New York State, and the National Science Foundation at the University's Center for Electronic Imaging Systems. tr