Professor Pulkit Agrawal runs the Improbable AI Lab at CSAIL, where they have the goal of building machines that can automatically and continuously learn about their environment. Dr. Agarwal is an Assistant Professor in the department of Electrical Engineering and Computer Science (EECS) at MIT. The Improbable AI Lab is affiliated with the Laboratory for Information and Decision Systems (LIDS) and involved with NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI).
Gabriel Margolis is a PhD candidate in EECS at MIT CSAIL, advised Prof. Pulkit Agrawal. His research concerns the learning-based acquisition of motor skills for locomotion and manipulation. Previously, Margolis obtained BS ('20) and MEng ('21) degrees at MIT.
Abstract: Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more.
Paper | Code | Learn More
Anthony Simeonov is a Ph.D. student in the EECS department at MIT, advised by Professor Alberto Rodriguez and Professor Pulkit Agrawal. He is interested in robot manipulation and physical interaction. His work is supported by the NSF GRFP. He received his S.M. in EECS from MIT and my B.S. in Mechanical Engineering from UC San Diego, where he worked with Professor Michael Yip. Prior to CSAIL; Simeonov worked as research intern at Disney Research Los Angeles and NVIDIA's Seattle Robotics Lab.
Abstract: We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors.
Paper | Code | Learn More
Hear from PhD candidate Aviv Netanyahu on her distribution shift and imitation learning research. Netanyahu is a PhD student in EECS at MIT CSAIL, advised by Prof. Pulkit Agrawal, as part of the Embodied Intelligence Community of Research. She earned her MSc from the Faculty of Mathematics and Computer Science at the Weizmann Institute of Science, advised by Prof. Shimon Ullman, where she researched Computer Vision. Netanyahu holds a BSc degree in Mathematics and Computer Science from the Hebrew University of Jerusalem.
Abstract: Machine learning systems, especially with overparameterized deep neural networks, can generalize to novel test instances drawn from the same distribution as the training data. However, they fare poorly when evaluated on out-of-support test points. In this work, we tackle the problem of developing machine learning systems that retain the power of overparameterized function approximators while enabling extrapolation to out-of-support test points when possible. This is accomplished by noting that under certain conditions, a "transductive" reparameterization can convert an out-of-support extrapolation problem into a problem of within-support combinatorial generalization. We propose a simple strategy based on bilinear embeddings to enable this type of combinatorial generalization, thereby addressing the out-of-support extrapolation problem under certain conditions. We instantiate a simple, practical algorithm applicable to various supervised learning and imitation learning tasks.
