Audrey Woods, MIT CSAIL Alliances | September 8, 2026
Have you ever tried to generate an image or video with AI but just can’t get it right? The person wasn’t exactly how you described, or the scene was a little bit off, and asking the model to fix the problem only makes it worse. Today’s tools are undeniably impressive, but getting a highly specific, tailored output can be frustrating. MIT CSAIL Professor Frédo Durand says, “It's one thing to be able to generate every picture that's ever been created. You also want to generate the right picture at the right moment. We're not quite there.”
As a leader in the Computer Graphics Group and a well-known figure in computational photography, famous for co-creating open-source contributions like Halide, Durand has seen firsthand how machine learning and generative AI have transformed the field. Now, as the Amar Bose Professor of Computing at MIT, an ACM Fellow, and winner of the 2016 SIGGRAPH Computer Graphics Achievement Award, he is pushing the limits of AI and image generation to see how much better it can get.
CLOSING THE LOOP: FROM TEXT TO IMAGE AND BACK AGAIN
Photography, Durand explains, was once an area focused on measurements “and on making those measurements as good as possible.” The computation being done was largely to “reveal information from the measurement,” or to add priors, pre-existing knowledge about what images of the world are supposed to look like. “If I have a measurement that's ambiguous—it's blurry, or it's very noisy—I know how to make it better by bringing in that a priori knowledge about what good images look like. We were hand-designing algorithms to do that, and we were using very simple priors.” Generative AI has changed the game because these incredibly powerful models are “in a sense the ultimate prior.” They know a lot about the world, which has flipped the problem on its head. “The priors are incredible, but we’ve lost the ability to connect them as rigorously to measurements.” In other words, it’s easy to produce high-quality images, but hard to make them faithful depictions of the world. “That's the big tension in the field and the industry.”
One way Durand and his colleagues are solving this is to try and “close the loop” between text and images. “When you go from text to image and then image back to text, or vice versa, you should get the same thing. You should get a cycle.” Working with Associate Professor Phillip Isola and graduate students Hyojin Bahng and Caroline Chan, Durand helped create a system which is rewarded for prioritizing this kind of cycle consistency, offering a better alternative for model quality control than ratings from human reviewers. “This simple idea turns out to be a really good way to make sure these models do a good job.” Such work has opened exciting avenues for “asking more philosophical questions to understand how images and text relate: the good old question of how many words a picture is worth.”
But what about models that follow a user’s intent, giving people the video or images that are not just correct but aligned with what they want? Beyond empowering models with a stronger text-to-image foundation, the way to provide true personalization is robust editability. One recent project in that space was work that shifted a video generation model from a pretrained bidirectional diffusion transformer to an autoregressive transformer. Traditionally, Video diffusion models denoise an entire clip at once, with every frame attending to every other frame—including future ones—so you have to wait for the whole sequence before you can watch anything. Durand and his colleagues solved this by training the model to generate one frame at a time, faster and more accurately than previous attempts. A hurdle in this approach is that any imperfection in one frame can quickly degrade over the course of multiple generations—one error leads to another leads to another until the video is unusable. But Durand’s team trained their model with imperfect inputs, so the model is “not surprised when it has to generate new video from frames that aren't 100% perfect.” Also, they used a method called distillation where, using a student model trained on a teacher model, the student model can learn to skip steps. “The big trick our former student Tianwei [Yin] introduced is that the student model doesn't need to learn exactly what the teacher is doing. The student doesn't need to map the same noise to the same image, it just needs to map noise to one of the images the teacher would create.”
The shift toward more personalization in image and video generative models is already being reflected in industry. One startup he works with, Reve—co-founded by a CSAIL alumnus—offers “more editability: you can drag things around and move them where you want, and you have finer-grained control over individual objects without changing the rest.” Among other signals, this makes Durand hopeful. “It's getting there. I'm very optimistic, because this is just the beginning of adding that kind of controllability, and I think a lot of progress will happen in the near future.”
OTHER WORK: RIS, VACCINES, AND A NEW TEXTBOOK
Another startup Durand is advising is CSAIL Alliances Startup Connect Plus member Remote Intelligence Solutions, a company aiming to modernize the electrical grid. Durand is “trying to guide them through these crazy times. They recognize that AI is taking over the world and presents tremendous opportunities for people who need to analyze a lot of data,” like utilities companies with massive, aging infrastructure. He offers RIS a direct, front-seat view of how machine learning and AI are changing as well as insight into the opportunities available for organizing and processing visual data. On his end, “one thing I find very exciting about what they do is that the visual data they have is among the most challenging I've seen for the field.” Poles and wires and backgrounds full of dense foliage present a situation that modern vision algorithms weren’t designed for, which offers a “really challenging situation that can push the field to make things more robust.”
Durand’s work also extends into healthcare. In 2025, he was part of a project using microneedle arrays to tattoo invisible markers in animal studies—with readout also demonstrated on donated human skin—for mRNA vaccines in an effort to offer physical medical records in poor parts of the world that don’t have them. The idea is that, instead of relying on faulty or nonexistent medical records while conducting vaccination campaigns, the “needles carry the vaccine or therapeutic plus an invisible near-infrared dye that eventually biodegrades over time. When you come back three months later, you can use a special camera and see how the person was vaccinated on that day.” Durand contributed to this project by helping create the computer vision algorithm which could “decode the tattoo” as well as determining how to make the marking resilient against error. “If some of the dots are missing, you want the information to be resilient, so we essentially used similar technology used in CDs.”
Durand’s current focus is writing a new textbook about digital and computational photography, covering both the standard material and the many ways the field is changing. This is a project he’s wanted to tackle for years but has never had the time. “Now, with coding agents, I can really magnify my productivity. It's allowed me to do things I've wanted to do for a while but didn't have the personal time or resources for.”
FUTURE THOUGHTS: UTOPIAN OPTIMISM, REALISTIC CONCERNS
Overall, Durand says he’s “turning into a techno-utopian. I think new tools are great.” Personally, AI has made it possible to dedicate time to his textbook, and he’s excited by all the creators and innovators who are able to do things that wouldn’t have been possible before. “AI is a creative tool that will let people with low budgets do incredible things.” He’s conscious of the issues—propaganda, false data, slop—but hopes that people with good stories to tell will be empowered and “the good content will bubble up, and good things will happen.”
That will, in part, boil down to AI’s ever-improving ability to create personalized, editable, and faithful outputs. With researchers like Durand working to make everything from 3D scene reconstruction to the individual frames of video generation more adaptable, the problem of generating the specific content you want might soon be an issue of the past. “I think generative AI is the biggest revolution in humans' ability to make images since the invention of photography, almost exactly 200 years ago. Things are changing so much, and we have such exciting new capabilities that I could not have dreamed of 10 years ago. It's an incredible time.”
Learn more about Professor Durand on his website or CSAIL page.
