Improving AI-generated images, pixel by pixel
Luke Auburn
|
Director of Communications, Hajim School of Engineering & Applied Sciences Researchers from URochester and NVIDIA develop new models that provide pixel-level control over AI image generation.
Why this matters:
- The model advances AI image generation by allowing users to make precise changes without unintentionally altering other details.
- Users gain greater control by working directly with individual pixels.
- The pixels-space diffusion model achieves nearly the same performance as latent space models that underpin state-of-the-art image generation tools.
- The pixel-space diffusion model could enable greater creative control, shaping the future of AI video generation.
If you have ever used an artificial intelligence text-to-image generator like ChatGPT or Google Gemini, you may have encountered a common but maddening problem: the image looks nearly perfect, but when you ask the AI to fix one small detail, it messes up something unrelated.
The issue is caused by the way the AI models are built. Most state-of-the-art AI image generators use a diffusion model in latent space, meaning they compress concepts such as a cat or a dog into their essential details and store them so they can be retrieved quickly and efficiently when it’s time to create an image. This is part of what gives the models their seemingly magical ability to generate the images you describe in seconds. But the method has its drawbacks.
“Using the latent space helps cut down the computation, but the problem is that you lose controllability because you are not adjusting individual pixels,” says Jiebo Luo, the Albert Arendt Hopeman Professor of Engineering at the University of Rochester. “As a result, you can’t easily control specific parts of the images you generate.”
Pix(els) or it didn’t happen
Developed by researchers at URochester and NVIDIA, PixelDiT2 efficiently implements pixel space diffusion to achieve almost the same performance as the state-of-the-art models—but provides controllability over individual pixels.
Reimagining image generation at the pixel level
Luo’s lab and collaborators at the AI infrastructure company NVIDIA were one of the first teams to develop a model that trades the latent space for the pixel space, dubbed PixelDiT. They presented their first iteration at the 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), where it was nominated as a best paper finalist, given to just 15 of over 4,000 accepted papers at the conference. The model allows users to zero in on certain regions of an image, like fixing just the pixels used to illustrate a pair of glasses on a person’s face or adjusting the color of a car.
The team has since created an even more powerful model, PixelDiT2, that implements pixel space diffusion in an efficient way to achieve almost the same performance as the state-of-the-art latent space models but providing controllability over individual pixels. The team will present a paper showcasing PixelDiT2 at the Neural Information Processing Systems conference (NeurIPS) this December.
Luo and his collaborators, who include computer science PhD student and NVIDIA intern Yongsheng Yu as well as alumnus Wei Xiong ’22 PhD (computer science), a senior researcher at NVIDIA, now have turned their attention to an even more difficult problem: video generation.
“Video is a big challenge because you have to worry about smooth transitions from frame to frame; otherwise it looks funny to the human eye,” says Luo. “You also have a lot more data to work with.”
But Luo says the progress the team has made with PixelDiT2 leaves them in a strong position to pursue AI video generation in the pixel space, which would provide users powerful new levels of control over the videos they generate.
