Written by Matthew Busekroos | Produced by Andrew Zucosky
Originally from Omaha, Nebraska, PhD candidate Vighnesh Subramaniam received both his bachelor’s and master’s degrees from MIT prior to continuing his education at CSAIL. Subramaniam mentioned the incredible diversity of research at MIT and CSAIL in particular as one instrumental factor in his decision to continue his education here.
Subramaniam is currently part of the Infolab, co-advised by Professor Boris Katz and Research Scientist Andrei Barbu.
“Our lab works on many ideas that cross different research fields and I’ve learned how to take some ideas from one field and apply them to others,” Subramaniam said. “More importantly, my advisors are very good at taking high level questions and translating them into low level research directions.”
Subramaniam said one project he is most excited about is a new idea to train neural networks. According to Subramaniam, neural networks have different designs but certain designs are inapplicable for some tasks, due to the difficulty of training such networks. In deep learning, most people would state that these untrainable networks lack the correct prior to make them applicable to the task i.e. they don’t have the right design prior that makes them trainable, Subramaniam added.
“We have developed a method to make these networks trainable via alignment with a network that has the better design,” he said. “This alignment critically happens with a network that has no knowledge, such as, the network is untrained but is designed better, prior to making it trainable. By making these networks trainable, we can begin to unlock new information about the mysteries of neural network design and expand our ability to train networks that have better properties for latency or throughput. Most excitingly, this project borrows ideas used in neuroscience and applies them in a way that is useful for studying neural networks.”
Subramaniam said neural networks have been incredibly impactful from technologies, including ChatGPT or Dalle. At a practical point though, Subramaniam said these networks are expensive to train and run.
“More importantly, there are many difficulties behind neural network design and training like hyperparameter tuning,” he said. “My hope is that the research outlined above gives more understanding behind neural network design, optimization, and interpretability. I believe the ideas above outline ways of comparing neural networks. This could have a massive impact on research in neuroscience where large neural networks are often compared with the brain. If we can figure out mathematically sound ways of comparing neural networks, we could extend such work to the human brain as well to unlock new understanding of the mechanistic properties of the brain.”
Subramaniam said he is excited about focusing on deeper questions about unknown systems of intelligence.
“I’ve always found the unique capacity for language and cognition in humans interesting and have wanted to understand how these properties emerge and how we can build machines that have these capabilities,” he said. “I think the AI landscape today is exciting not only because there are some incredibly capable systems like ChatGPT, but because we can begin to ask deeper questions about the nature of intelligence from a representational and algorithmic perspective. My goal through a PhD is not only to build great applications but also to build a deeper understanding of the principles behind the technology we’re building.”
Subramaniam is still a few years away from completing his PhD, but he said he enjoys doing research and hopes that any future job he takes would involve research in some capacity.
For more on Vighnesh Subramaniam, check out his group’s paper on untrainable networks here: https://untrainable-networks.github.io/