Audrey Woods, MIT CSAIL Alliances | August 3, 2026
Before joining MIT in 2025, Assistant Professor Omar Khattab had already created several transformative tools that made their mark on industry. DSPy, the first and most widely used declarative programming framework for building language systems, has been downloaded over 55 million times since its release. ColBERT, a retrieval model whose “late interaction” paradigm helped shape the modern landscape of neural search, was downloaded over 10 million times in May 2026 alone. These and other systems Professor Khattab created have been successfully applied at companies like Microsoft AI, Amazon AWS, Shopify, Dropbox, VMware, and Databricks, where Khattab worked as a Research Scientist before joining MIT.
Now, as leader of MIT OASYS or “Open Source AI Systems,” Khattab and his team are continuing to create open-source AI tools that deliver real results. With a focus on “AI at the last mile,” a term from logistics where the final leg of delivery is famously the hardest, his lab studies how to most effectively draw out that last bit of critical data from human users and adapt model behavior to the specific needs that current AI models struggle to address.
LAST MILE AI & THE ILLUSION OF FREE LUNCH
Khattab says that in the last few years, the power of foundation models has become “staggering. Their raw capability is far deeper than we would have ever dreamed.” But this raw power creates an illusion of “free lunch,” where these models are able to solve all our problems and be applied out of the box to every esoteric need, niche domain, or particular business problem. There’s a big gap between a model that produces generally reasonable answers and one that responds to the specific needs of a company or codebase. “That extra last mile is the challenge that’s really difficult right now.”
“As we get more ambitious in terms of what we ask of AI models, the amount of human effort and hand-holding and human in the loop increases. It’s not decreasing.”
Rather than targeting any single vertical, Khattab’s OASYS lab builds general-purpose tools with large open-source communities to maintain and extend them. Think of a hospital tasking a model to write discharge summaries. Each specific hospital might have its own formatting rules, billing codes, medication markdowns, and liability-sensitive phrasing. To get a standard LLM to do this, the user would have to upload lengthy instructions, perhaps even all of the hospital’s legal and compliance documentation, which makes each query expensive and likely leads to context rot. Or imagine a legal analyst looking to answer a specific question about a 500-page merger due diligence packet using AI, which would traditionally require uploading the whole document along with the prompt. Khattab’s work aims to address these inefficiencies and allow non-experts working with specific use-cases to design the specialized AI tools they need to solve their last mile problems.
BEYOND PROMPTING: RLMS & PEDAGOGICAL REINFORCEMENT LEARNING
Khattab’s OASYS students are taking several different approaches to overcome the current limitations of LLMs and AI agents. First, they are addressing the issue of the extremely long prompts and/or enormous document attachments with Recursive Language Models (RLMs). Conceptualized by Khattab’s student Alex Zhang, RLMs treat the prompt as a programmatic variable, kind of like a file on disk. Then the model behaves like an agent in a coding environment, where it writes small bits of code to recursively invoke itself on chunks of the prompt, retrieving the necessary data to answer the request. Not only does this offer a theoretically unlimited context window for a given query, it also improves accuracy and generalizability because each sub-version of the model is only focused on a small piece of text. On classic “needle in a haystack” tests, a conventional model trained to find two hidden needles fails at eight. An RLM handling the same challenge “extrapolates perfectly on the tasks that we’ve tried.”
One of the biggest industry applications of RLMs so far has been to evaluate AI agent behavior in production. Companies running language model agents “have hundreds of thousands of agent trajectories getting logged every day. One of the hardest things to do when you have language model agents in production systems is understanding their behavior at scale. You want to understand what mistakes they are making over and over that are maybe getting lost in the logs.” Pointed at an arbitrarily long string of agent logs, an RLM “can begin to do things like spawn hundreds of subcalls, hundreds of sub-agents, that will read different parts of this, find local patterns, and aggregate them hierarchically.”
Another way last mile information is conveyed is through feedback, where an agent is deployed in a test environment to see where it breaks and then learns from the results. Reinforcement learning (RL) is the standard machinery for this, but Khattab argues that popular algorithms for RL “are actually leaving a lot of signal on the table.” If a task is hard enough that all of a model’s attempts fail in a row, there is essentially nothing to learn from and the model will “just stumble around forever, trying random things.”
His lab’s answer is Pedagogical Reinforcement Learning, developed by postdoctoral researcher Souradip Chakraborty and former visiting PhD student Noah Ziems, which is inspired by how people actually study. “If you give me a really hard math problem, maybe I’ll try it once, I’ll try it twice, but if I fail to answer, I might take a look at what the official answer says. Now that I know this answer, I can ask: what could I have done, based on the knowledge I had previously, so that I could have actually come to that answer on my own?” Pedagogical RL gives the model the question along with privileged information about the answer, then constrains it so that every reasoning step must remain plausible to a copy of the model that doesn’t know the answer, a property the team can measure directly by tracking spikes in the model’s “surprise” measurements. The result is a model that can act as its own teacher on problems it could never have solved by trial and error.
BENCHMARKING THE PROBLEM: OBLIQ-BENCH & MACHINE STUDYING
To highlight just how inefficiently today’s systems handle large bodies of documents, like company policies, new scientific literature, or the codebase of a massive framework released after a model was trained, Khattab’s lab has designed and released two benchmarking tools.
OBLIQ-Bench, built by PhD student Diane Tchuindjo, is a dataset of hard search queries whose answers share almost no surface forms with the query. Imagine an analyst searching a corporate social media archive for posts that are upset about new user interfaces. The trouble, Khattab explains, is that “none of these tweets would say, ‘I’m upset about the user interface.’ They might say ‘I love that they just removed the button I used to rely on,’” sarcasm a full language model might pick up but a retrieval system will almost certainly miss because search “has to work in milliseconds. It can’t read all the documents every time through a language model.” The team found that essentially all existing retrieval paradigms struggle on these queries. “Current systems for conditioning on last mile information that takes the form of a corpus are more limited than you would think.”
The second effort is Machine Studying—led by PhD student Jacob Li—which is “kind of like a riff on machine learning.” Instead of optimizing a known reward, the model is handed a corpus and asked to develop expertise in it without knowing what the test will look like. The lab formalizes expertise as a trade-off between inference-time effort and quality. “A smart enough student can show up to an open book exam and, if you give them unlimited time, maybe answer most of the questions. But it’s only someone who has developed some genuine expertise, who can leverage that studying time to massively accelerate their ability to perform in a difficult environment pretty quickly.” Similarly, an intelligent enough model can study minimally but increase in expertise rapidly. “If one model can spend $10 to gain one unit of expertise in the domain, and another model needs to spend a million dollars, clearly something about the first model is more amenable to last mile learning.” On the lab’s test domains, “current algorithms are actually pretty disappointing,” which is precisely the problem his lab works on. “We hope that OBLIQ-Bench and Machine Studying drive the next wave of algorithms that can learn from a big corpus.”
In a short amount of time, Khattab and his students have made significant contributions to MIT’s open source offerings. Could the tools coming out of the OASYS lab help with your last mile challenges? Learn more about Professor Khattab on his website, his MIT EECS page, or his group’s website.
