WRITTEN BY: Audrey Woods | VIDEO BY: Karl Sims
On the first floor of the MIT Stata Center, right beside the CSAIL Alliances entrance, there’s a large display that’s always in motion. This interactive piece reflects the scene in front of it—including the people walking by—but augmented with vibrant colors, swirling textures, and entertaining effects. Standing in front of the aptly titled Flow exhibit can be an enjoyably dizzying experience, with every motion, step, and wave producing a bloom of ink, a strobe of colored light, or whorls of paint strokes inspired by Van Gogh’s The Starry Night.
But Flow is more than just a way to entertain CSAIL visitors. It’s also a testament to an often-overlooked subset of computer science, an industry that few would put in the same sentence as things like artificial intelligence and deep learning: the art world.
There are plenty of jokes about the gap between artists and engineers, but the truth is that much of the visual media we have today wouldn’t be possible without computer science. CGI, photography apps, video games, movie streaming, and so many other pieces of entertainment and media depend on technology that didn't exist 50 years ago. Every part of the movie and music industries, from idea to finished product, is affected by computer programs and hardware that has come out of research centers like MIT CSAIL. Even the visual art world is beginning to change with the advent of algorithms like AI image-generator DALLE.
As an example, look at the career of the man behind Flow, digital media artist Karl Sims. After graduating from MIT in 1987 with a master’s in computer graphics, he became the artist in residence at Thinking Machines, a supercomputer manufacturer and artificial intelligence company. As Sims describes it, “my job was to test out all this new software and make animation that was a visible way to show off the computer power of the machine.” This led to the creation of the Virtual Creatures Project, which simulated Darwinian evolution in a virtual setting and led to a series of landmark papers about the surprising creativity of this artificial process. Soon after his work at Thinking Machines, Karl Sims started his own company called GenArts, which developed a package of special effects plugins that helped create the effects of major blockbuster movies like The Matrix, X-Men, Star Wars, and Lord of the Rings. Sims says, “when there are Orcs nearby and Frodo pulls his sword out and it glows blue, that was our glow.”
Sims is far from alone at this juncture of computer science and art. Many other computer scientists are supporting and inspiring the artistic world, especially here at MIT CSAIL. Professor Fredo Durand’s Halide compiler was a critical tool in trimming down the code needed for massive photo processing software like Adobe Photoshop. Recently, MIT researchers developed a way to make DALLE-2 more creative, which could lead to more interesting and diverse AI-assisted artwork. A group of professors, including CSAIL Professor Wojciech Matusik, created a way to use machine learning to automate parts of the notoriously slow and tedious CGI process. Professor Matusik is currently working on a differentiable cloth simulator that will improve and speed up methods for fabric animation.
Speaking of fabric, CSAIL PhD student Yunyi Zhu—working in Professor Stefanie Mueller’s lab—is designing dynamic cloth and materials that can change color and shape, adding new possibilities to the field of fashion design. Another innovation that came out of Professor Mueller’s lab was the creation of spray-on paint interfaces, which can transform any room into an augmented reality experience. Researchers in both Professor Mueller’s and Professor William Freeman’s groups worked together to create a system that can print 3-D statues of a figure in motion from a 2-D video. And, with CSAIL researchers hard at work on the technology, holograms could be the next big thing in home entertainment.
When asked if he considers himself an artist, Sims responds, “I wear an artist hat sometimes.” But generally, he sees his work as the process of developing tools that can then be used to create art, whether that’s a big-budget film or an interactive display on the ground floor of CSAIL. But he also points out, regarding his interest in artificial evolution, how “some things are potentially interesting from a scientific perspective, but the results are considered artistic. You can mix and match these things.” In short, it’s pointless to draw industry delineations in a field as exciting and dynamic as computer science. Yunyi Zhu’s reprogrammable fabrics require an understanding of cloth, photochromic dye, LEDs, and coding. Professor Durand’s current project on computational bounce flash for indoor lighting is both a hardware and software problem, not to mention deeply tied to the craft of photography. Computer-aided photos of supermassive black holes can be as awe-inspiring as a painting. And Karl Sim’s unique career defies categorization.
At this intersection of creative work and scientific invention, he’s just excited to “keep getting stuff out into the world.”