Audrey Woods | MIT CSAIL Alliances
As far as computing systems go, it’s hard to do better than the human brain. Even as AI models begin to outperform people at specific tasks, human brains still surpass AI in many ways, especially general intelligence, and only require a tiny fraction of the energy consumed by modern AI models to do so.
MIT CSAIL Professor Nir Shavit believes computer scientists can learn from what evolution created, leveraging inspiration from neurobiology to create sparser, more efficient, and increasingly powerful algorithms. With an extensive background in high performance computing, Professor Shavit wants to inspire his colleagues to think outside the box, melding biology, theory, and creativity to improve AI and amplify the power of technology.
Professor Shavit’s career has had two distinct trajectories. When he first decided to pursue computer science, he was focused on “really hardcore theory.” He jokes that he was “doing parallel computation before there were parallel machines.” Professor Shavit went on to become a global expert in performance engineering, developing the first scaling laws for neural networks and creating highly performant algorithms. He won the 2004 Gödel Prize in theoretical computer science for his work applying tools from algebraic topology to model shared memory computability. By all metrics, Professor Shavit had reached the pinnacle of his field.
So a little over ten years ago, he decided to jump into the entirely different—although not unrelated—field of neurobiology, specifically the subfield of connectomics, which focuses on understanding connections in the nervous system, “relevant because there’s lots of parallel computing in brains.” He believed mapping connectivity in neural tissue would help him understand how to build better AI. But this meant he had to go back to basics, taking undergraduate biology classes, graduate coursework in neurobiology, reading lots of papers, and generally spending two years coming up to speed. Professor Shavit says that his neurobiology colleagues told him, “We don’t want to give our data away and have you do theory on it. If you want to join, then you have to help collect the data.” So that’s what he decided to do. “Using my skills as a performance engineer and parallel computing expert, I contributed my engineering expertise to this effort.”
Now, Professor Shavit’s lab collaborates with interdisciplinary scientists to build better, faster, and more efficient algorithms by applying the knowledge of connectomics to computer science.
Professor Shavit’s connectomics work broadly falls into three buckets. First, his group is working on better ways of gathering neurological data. One of the problems in neurobiology is that the data samples are enormous, petabytes of information. “A cubic millimeter of mouse brain is about two petabytes of data,” Professor Shavit says. This means that it is both slow and difficult to gather this data. “We’re trying to improve reconstruction of neural tissue from electron microscopy,” he explains, “working on making it faster and more accurate.” For example, one of the projects in this area is to build a smart electron microscope which could gather this information more quickly.
After the data has been collected, Professor Shavit’s group is also working to understand the neural connectivity represented in the samples. Our understanding of brain tissue is infamously limited, but we do know that neural tissue is very sparse, meaning the number of connections is small, the layout of those connections extremely efficient, and the neurons only fire when necessary. Professor Shavit believes that understanding this sparsity—even without fully understanding the mechanism of a given neural sample—is key to creating sparser algorithms for AI models, which are right now vastly less efficient than biological brain tissue. “We try to understand structurally what's going on rather than try to decipher how brains think. We want to know how they compute.”
Pulling this together, the third area of Professor Shavit’s focus is applying these lessons to machine learning and using them to better understand and then improve AI models. For example, his group is studying special kinds of nodes in neural networks called Wasserstein neurons, which are more sensitive than other neurons and therefore end up being responsible for a lot of error in the network. They tend to be overloaded, detecting many different features that might be completely unrelated. By studying these neurons and combining this knowledge with their understanding of connectomics, Professor Shavit and his group are working toward building better, sparser neural networks.
Professor Shavit was in the news recently after his CSAIL-spinout Neural Magic was acquired by IBM-owned software firm Red Hat. He says that the company actually “started from this neurobiology research because we needed to process massive amounts of data quickly.” This meant creating high-performance neural network algorithms that ran on regular computers without requiring specialized AI chips. He and his colleagues were creating these algorithms for their own biology work “where we had to be efficient,” but by sharing them “we ended up being the provider of performance engineering for executing inference in neural networks.”
While he’s glad to be on the academic side of things, Professor Shavit says he’s grateful for his experience running a startup because, “It’s a very fast-moving field and you can’t really be fully in it just being an academic. You need to have a feel of what’s going on in industry.”
On the research side, Professor Shavit has recently been returning to his roots in theory. “I come from a theoretical computer science background,” he says, and after years of being “really busy” with the performance and non-theoretical side of machine learning, he’s “now digging deeper into actual theory and bringing theory to machine learning.” Specifically, he wants to get his colleagues interested in applying tools from theoretical computer science to machine learning. For example, in theoretical computer science there are methods to measure upper bounds and lower bounds, offering a way to understand what is and isn’t computable, and, if something is computable, what is the cost of that computation. There is enormous potential for this in machine learning, where users might be able to predict what a system is capable of and what the limits of that network might be without necessarily having to go through the expense of building and testing it. He hopes to see—and inspire—more crossover between theory and applied computer science going forward.
Over his career, Professor Shavit has watched computer science become “more and more interdisciplinary,” blending with other scientific fields like medicine, biology, chemistry, and physics. It’s difficult to imagine any modern scientific endeavour not affected by machine learning or AI, and Professor Shavit only sees that growing. On the flip side, he says that the computer science way of thinking—“algorithmic thinking, modular thinking, software-oriented thinking”—is affecting and shaping those fields as well, changing the way scientists of every discipline approach their research.
However, such advances in technology mean that “computer science as we know it is going away, and we better face it quickly.” Professor Shavit thinks that the days of humans doing the vast majority of coding tasks are already over because AI is better at programming than most people. Going forward, scientists in every discipline “will need to understand what I would call intelligent prompt engineering.” Professor Shavit imagines a world where future research is assisted by powerful neural networks that are akin to “sorcerers’ apprentices,” speeding up the scientific process and rapidly improving lives. And leveraging lessons from brain tissue to build faster, better, more human-like AI will make this process happen even faster.
In the meantime, Professor Shavit is thrilled to be working in academia. “Science in the university is really fun,” he says, describing the joy of discovery and the satisfaction of cultivating young talent. With his groundbreaking work at the intersection of neurobiology and computer science, Professor Shavit is sure to have many such joys ahead.
Learn more about Professor Shavit on his website or CSAIL page.