Architectural Evaluation of Processing-In-Memory Systems | Tanner Andrulis

Tanner Andrulis | Graduation Date: 05/01/2027
PI Lead:
Professor of the Practice Joel Emer, MIT EECS; Associate Professor Vivienne Sze, MIT EECS

Tanner is a Graduate Research Assistant at MIT advised by Profs. Vivienne Sze and Joel Emer in the EEMS group. He researches the design and modeling of accelerators for tensor applications and machine learning, focusing on novel analog and processing-in-memory systems.

Abstract: Processing-In-Memory (PIM) accelerators are a promising approach to efficiently run Deep Neural Networks (DNNs) as they move compute into memory and reduce high DNN data movement costs. Unfortunately, research has mainly focused on devices (e.g., memristors), circuits (e.g., analog converters), or architecture (e.g., dataflow) in isolation. It is desirable to see how innovation at any level, such as new devices, may change the efficiency and performance of whole accelerators. This would enable fair comparison of innovations and yield insight into the vast number of ways to combine them. We present a framework that models PIM at an architectural level. With fast simulation and easy-to-change PIM device, circuit, and architecture models, our framework enables researchers to see how innovations affect the efficiency and performance of PIM accelerators. Further, we simulate up to 10,000x faster, enabling fast evaluation of different PIM accelerators and exploration of the vast design space.

View PDF

Finger Interfaces for Next Generation Computers | Wenqiang (Winston) Chen

Wenqiang (Winston) Chen
PI Lead:
Professor Wojciech Matusik, MIT EECS

Wenqiang (Winston) Chen is a postdoctoral associate at MIT CSAIL. He received his PhD from the University of Virginia. His research lies at the intersection of Cyber-Physical Systems (CPS), ubiquitous and mobile sensing, and human-computer interaction (HCI). In particular, his research specializes in developing Vibrational Interaction (VibInt) systems to perceive and infer information from human bodies, robots, and environments through vibrations. VibInt has been proposed to advance a wide variety of research areas, such as wearable interactions, robotics, smart health, smart homes, privacy and security. He has published his research in various top conferences and journals (e.g., Mobicom, Ubicomp, and Transactions on Mobile Computing), obtained five patents, and won the IEEE SECON 2018 Best Paper Award, the ACM SenSys 2020 Best Demo Award and the ACM Mobicom 2022 S3 workshop Best Paper Award. (All as the first author) Winston is also a co-founder of VibInt AI, a startup working on wearable devices using VibInt technologies, and his research IPs have been used in thousands of commodity devices.

Abstract: VibInt is a cutting-edge AI algorithm designed to facilitate seamless interaction with next-generation computing devices, including smartwatches and smartglasses. Its capability to detect hand vibrations and recognize finger and wrist gestures leverages the existing accelerometer and gyro sensors within smartwatches. Thus, it offers an enhanced user experience without necessitating additional hardware modifications.

View Poster

Predicting clinical trial duration via statistical and machine learning models | Joonhyuk Cho

Joonhyuk Cho | Graduation Date: 05/31/2026
PI Lead:
Professor Andrew Lo, MIT Sloan School of Management

Joonhyuk is a second year Ph.D. student in MIT EECS department, supervised by Andrew W. Lo. Her research focus is on healthcare finance and fintech. More specifically, she is interested in computational analysis and prediction of clinical trial to catalyze financial investing on drug development area.

Abstract: We apply survival analysis as well as machine learning models to predict the duration of clinical trials using the largest dataset so far constructed in this domain. Gradient boosting trees yield the most accurate predictions and we identify key factors that are most predictive of trial duration. This methodology may help clinical researchers optimize trial designs for expedited testing, and can also reduce the financial risk of drug development, which in turn will lower the cost of funding and increase the amount of capital allocated to this sector.

View PDF

Giving Computers the Sense of Touch | Michael Foshey

Michael Foshey
PI Lead:
Professor Wojciech Matusik, MIT EECS

Michael Foshey, a Mechanical Engineer and Project Manager at MIT Computer Science and Artificial Intelligence Laboratory's Computational Design and Fabrication Group, specializes in developing innovative sensors for studying human-robot interaction and implementing machine learning algorithms for optimizing and understanding new sensing modalities.

Abstract: Tactile perception is the ability to interpret information received through the sense of touch. Humans have specialized sensory receptors distributed throughout their skin, allowing us to perceive pressure and vibration. Furthermore, our brain can quickly process the information our sensory receptors perceive and react to. Our ability to sense touch and respond rapidly to it is fundamental to our daily lives. We rely on it heavily to navigate the world around us and identify objects. In our research, by making new advances in tactile sensing and machine learning, we developed a cyber equivalent to the human's tactile perception unlocking a new set of technological opportunities in human health monitoring and robotics. To make these new systems possible, we have developed a set of tactile sensing arrays that are thin, flexible, and lightweight, making them easily integrated into different robotic devices. Furthermore, we have also developed a set of computational tools that can quickly analyze the tactile signals to extract high-level information that can be acted on.

View PDF

NeuSE: Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objects | Jiahui Fu

Jiahui Fu | Graduation Date: 08/31/2023
PI Lead: Professor John Leonard, MIT Mechanical Engineering

Jiahui Fu is a fifth-year PhD student at the Marine Robotics Group in CSAIL, advised by Prof. John Leonard. Her research focuses on long-term object-based SLAM in low-dynamic environments. Jiahui is generally interested in SLAM, perception, and robotics, as well as their intersection with machine learning. Recently, Jiahui has been working on change detection and object-level consistent scene understanding using neural implicit representations.

Abstract: We present NeuSE, a novel Neural SE(3)- Equivariant Embedding for objects, and illustrate how it supports object SLAM for consistent spatial understanding with long-term scene changes. NeuSE is a set of latent object embeddings created from partial object observations. It serves as a compact point cloud surrogate for complete object models, encoding full shape information while transforming SE(3)-equivariantly in tandem with the object in the physical world. With NeuSE, relative frame transforms can be directly derived from inferred latent codes. Using NeuSE for object shape and pose characterization, our proposed SLAM paradigm can operate independently or in conjunction with typical SLAM systems. It directly infers SE(3) camera pose constraints compatible with general SLAM pose graph optimization while maintaining a lightweight object-centric map that adapts to real-world changes. Our approach is evaluated on synthetic and real-world sequences featuring changed objects and shows improved localization accuracy and change-aware mapping capability, when working either standalone or jointly with a common SLAM pipeline.

View PDF | Marine Robotics Group | Project Page

GAPSLAM: Blending Gaussian Approximation and Particle Filters for Real-Time Non-Gaussian SLAM | Qiangqiang Huang

Qiangqiang Huang | Graduation Date: 08/30/2023
PI Lead: Professor John Leonard, MIT Mechanical Engineering

Qiangqiang Huang is a PhD student at the Marine Robotics Group, part of CSAIL at MIT, where he works on inference algorithms for SLAM. His PhD advisor is Professor John Leonard. He received his B.E. and M.S. degrees from Tsinghua University. His research lies in the intersection of robotics, Bayesian inference, computer vision, and machine learning. Specifically, he focuses on developing full posterior inference algorithms that enable uncertainty-aware robotic perception. Applications of the algorithms include evaluating the uncertainty in localization and mapping, as well as supporting robots to plan how to reduce uncertainty for safe navigation.

Abstract: Inferring the posterior distribution in SLAM is critical for evaluating the uncertainty in localization and mapping, as well as supporting subsequent planning tasks aiming to reduce uncertainty for safe navigation. However, real-time full posterior inference techniques, such as Gaussian approximation and particle filters, either lack expressiveness for representing non-Gaussian posteriors or suffer from performance degeneracy when estimating high-dimensional posteriors. Inspired by the complementary strengths of Gaussian approximation and particle filters–scalability and non-Gaussian estimation, respectively–we blend these two approaches to infer marginal posteriors in SLAM. Specifically, Gaussian approximation provides robot pose distributions on which particle filters are conditioned to sample landmark marginals. In return, the maximum a posteriori point among these samples can be used to reset linearization points in the nonlinear optimization solver of the Gaussian approximation, facilitating the pursuit of global optima. We demonstrate the scalability, generalizability, and accuracy of our algorithm for real-time full posterior inference on realworld range-only SLAM and object-based bearing-only SLAM datasets.

View PDF

Empowering Users on Social Media for Better Content Credibility | Farnaz Jahanbakhsh

Farnaz Jahanbakhsh | Graduation Date: 09/01/2023
PI Lead:
Professor David Karger, MIT EECS

Farnaz Jahanbakhsh is a PhD student at MIT CSAIL advised by David Karger. Her research in in the field of Human-Computer Interaction and Social Computing and is aimed at re-imagining the design of social media and more broadly the web to achieve greater end-user empowerment. Her work has been published in premier venues including ACM CHI and CSCW. Before coming to MIT, she completed her Master's in CS at the University of Illinois at Urbana-Champaign (UIUC) and prior to that, her Bachelor's in Computer Engineering at Sharif University of Technology in Iran.

Abstract: As misinformation raged on online social spaces and threatened people's lives and even democracy, platforms rose as the authority on misinformation detection and moderation. However, concerns about freedom of speech and listening rights and the autonomy of individuals in deciding what content to consume, as well as the misalignment in incentives between the users and the platforms should give us pause in accepting this centralized moderation as the optimal solution. My work presents an alternative approach to misinformation moderation-a democratized one that empowers every user to have a say in what content they consider misinforming and what they want to do with such content. My work explores the following questions: 1) how to alter the design of social media platforms to enable users to have a say in what is misinformation, 2) how this user empowerment changes the accuracy of shared content, 3) how to design tools that enable this user empowerment on the web and work on all platforms without needing support from them, 4) how to leverage AI not to impose “the truth" on users, but to help amplify users’ assessments, and 5) how to enable users beyond labeling content accuracy, and empower them to modify and “fix'' online content.

View PDF

Learning Proximal Operators to Discover Multiple Optima | Lingxiao Li

Lingxiao Li | Graduation Date: 05/30/2024
PI Lead: Associate Professor Justin Solomon, MIT EECS

Lingxiao Li is a fourth-year Ph.D. student at MIT in the Computer Science and Artificial Intelligence Lab (CSAIL) advised by Justin Solomon. Previously, he obtained a Master's degree in Mathematics and Bachelor's degrees in Computer Science and Mathematics, all at Stanford University. His research interest is applying geometric tools to tackle problems in optimization, statistics, and computer graphics problems.

Abstract: Finding multiple solutions of non-convex optimization problems is a ubiquitous yet challenging task. Most past algorithms either apply single-solution optimization methods from multiple random initial guesses or search in the vicinity of found solutions using ad hoc heuristics. We present an end-to-end method to learn the proximal operator of a family of training problems so that multiple local minima can be quickly obtained from initial guesses by iterating the learned operator, emulating the proximal-point algorithm that has fast convergence. The learned proximal operator can be further generalized to recover multiple optima for unseen problems at test time, enabling applications such as object detection. The key ingredient in our formulation is a proximal regularization term, which elevates the convexity of our training loss: by applying recent theoretical results, we show that for weakly-convex objectives with Lipschitz gradients, training of the proximal operator converges globally with a practical degree of over-parameterization. We further present an exhaustive benchmark for multi-solution optimization to demonstrate the effectiveness of our method.

View PDF

CausalSim: A Causal Framework for Unbiased Trace-Driven Simulation | Arash Nasr-Esfahany

Arash Nasr-Esfahany | Graduation Date: 05/03/2025
PI Lead:
Associate Professor Mohammad Alizadeh, MIT EECS; Professor Devavrat Shah, MIT EECS

Arash is a fourth-year PhD student at MIT, advised by Mohammad Alizadeh. He is interested in modeling large-scale computer systems with causality and machine learning. He has received the best paper award at NSDI'23. Before MIT, he did his undergrad in the Electrical Engineering department, Sharif University of Technology.

Abstract: We present CausalSim, a causal framework for unbiased trace-driven simulation. Current trace-driven simulators assume that the interventions being simulated (e.g., a new algorithm) would not affect the validity of the traces. However, real-world traces are often biased by the choices algorithms make during trace collection, and hence replaying traces under an intervention may lead to incorrect results. CausalSim addresses this challenge by learning a causal model of the system dynamics and latent factors capturing the underlying system conditions during trace collection. It learns these models using an initial randomized control trial (RCT) under a fixed set of algorithms, and then applies them to remove biases from trace data when simulating new algorithms. Key to CausalSim is mapping unbiased trace-driven simulation to a tensor completion problem with extremely sparse observations. By exploiting a basic distributional invariance property present in RCT data, CausalSim enables a novel tensor completion method despite the sparsity of observations. Our extensive evaluation of CausalSim on both real and synthetic datasets, including more than ten months of real data from the Puffer video streaming system shows it improves simulation accuracy, reducing errors by 53% and 61% on average compared to expert-designed and supervised learning baselines. Moreover, CausalSim provides markedly different insights about ABR algorithms compared to the biased baseline simulator, which we validate with a real deployment.

View PDF | CasualSim

Who2chat: System for Academic Networking in Virtual Social Hours Enabling Coordinating, Sensemaking and Social Signaling | Soya Park

Soya Park | Graduation Date: 05/31/2024
PI Lead:
Professor David Karger, MIT EECS

Soya Park is a PhD student. She is interested in studying and designing collaborative technology inclusive toward workers with lower status. She is advised by Professor David Karger in the lab’s Haystack Group. The Haystack Group blends approaches from human-computer interaction, social computing, databases and information management.

Abstract: Academic networking is socio-technically challenging, however, fruitful for researchers' success. We introduce a system called Who2chat to tackle the challenge and facilitate connections of researchers. Who2chat allows academic researchers to create their profile and express interests, find researchers with similar interests, lower social barriers, and coordinate and start video chats, all within a single interface. We engaged in an iterative design process by deploying at academic conferences. In our preliminary deployment (N=80), we found that researchers often have difficulty finding other researchers who share their interests, and they are shy about reaching out to other researchers. Inspired by this, we implemented social-signaling features to Who2chat and ran our first deployment (N=220). Our results highlight that the interface allows users to express their willingness to meet more researchers and thus help them feel confident in joining conversations. However, it led to large conversations where too many researchers are in and prevented them from having meaningful conversations. We then developed and deployed our second interface (N=80) which addresses feedback from the previous deployment. As a result, we found that users were able to meet more people and engaged in more meaningful conversations than the previous deployment. Our findings demonstrate an interface design for social networking in academic settings and how to lower social barriers in virtual networking.

View PDF

An Out-of-Core GPU singular value decomposition illustrates Julia capabilities for large datasets | Evelyne Ringoot

Evelyne Ringroot | Graduation Date: 06/01/2024
PI Lead: Professor Alan Edelman, MIT Mathmatics

Evelyne Ringoot is a graduate student at the MIT Center for Computational Science and Engineering. Her research focuses on High-Performance Computing GPU implementations for numerical linear algebra. She aspires to accelerate the foundational algorithms underlying all applied computational fields. Evelyne graduated with great distinction from the MSc Civil Engineering at VUB Brussels and is a Hoover fellow of the Belgian American Educational Foundation.

Abstract: As the amount of available data in the world grows exponentially, calculations involve increasingly larger datasets. In particular, the Singular Value Decomposition of matrices is of interest as it is an important building block in HPC applications; SVD reveals the best low-rank approximation of matrices. We implement a Julia-native out-of-core QR-based algorithm for the calculation of the full SVD of matrices larger than typical consumer GPU as proposed by (K. Kabir, e.a., 2017).

View PDF

Codon: Compiling Python for 10-100x Performance Gains | Ariya Shajii

Ariya Shajii | CSAIL Alumnus (2021)

Dr. Ariya Shajii is the founder of Exaloop, which leverages innovations in compilers and programming languages to make scalable, high-performance computing accessible to domain experts in a range of fields and disciplines. Dr. Shajii completed his PhD at MIT CSAIL under Professor Saman Amarasinghe and Professor Bonnie Berger, focusing on the intersection between computational genomics, high-performance computing, and compilers.

Abstract: Codon is a high-performance, zero-overhead, extensible Python compiler that achieves orders-of-magnitude performance improvements over standard Python, rivaling the performance of C/C++. Codon supports native parallelism, GPU programming, and many other features that enable users to scale without leaving the comfort of Python. Learn more at https://exaloop.io.

View PDF | Codon | Exaloop

OnionChopper: A Modular Arithmetic Hardware Accelerator for Private Information Retrieval | Georgia Shay

Georgia Shay | Graduation Date: 05/31/2023
PI Lead:
Assistant Professor Mengjia Yan, MIT EECS

Georgia Shay is a Master of Engineering in Computer Science and Electrical Engineering student at MIT, graduating in May 2023. Her work has been in designing hardware for a privacy-related application. She has also been involved in hardware design through her TAship in an introductory hardware design course. After graduation she will be joining a processor design team for full time work.

Abstract: Private information retrieval (PIR) is a protocol which allows a user to retrieve data from a database on a server without the server being able to deduce which records were retrieved. Due to the homomorphic cryptography systems required to make these protocols work and large amount of data processing required per user query, these algorithms tend to run much slower than needed for real-time applications such as streaming movies or voice calling. To improve these speeds to ones more tolerable for user applications, we design OnionChopper: a small, fast, and energy efficient hardware accelerator on which to offload the heaviest computational work.

View PDF

An Artificial Intelligence-Based Industry Peer Grouping System | Manish Singh

Manish Singh | Graduation Date: 07/16/2023
PI Lead: Professor Andrew Lo, MIT Sloan School of Management

Manish Singh is currently a Ph.D. student in the Electrical Engineering and Computer science department at the Massachusetts Institute of Technology. He is advised by Prof. Andrew W. Lo and is a part of the Laboratory of Financial Engineering and Computer Science & AI lab. His research focuses on applying data science and artificial intelligence to healthcare finance, investment management, and sustainable investing. He holds a Bachelor of Technology with a major in Electrical Engineering and a minor in Computer Science from the Indian Institute of Technology Delhi, as well as an S.M. in Electrical Engineering and Computer Science from MIT.

Abstract: In this poster, we present a data-driven peer grouping system using artificial intelligence (AI) tools to capture market perception and, in turn, group companies into clusters at various levels of granularity. In addition, we develop a continuous measure of similarity between companies; use this measure to group companies into clusters and construct hedged portfolios. In the peer groupings, companies grouped in the same clusters had strong homogeneous risk and return profiles, while different clusters of companies had diverse, varying risk exposures. We extensively evaluated the clusters and found that companies grouped together by their method had higher out-of-sample return correlation but lower stability and interpretability than companies grouped by a standard industry classification system. We also develop an interactive visualization system for identifying AI-based clusters and similar companies.    

View PDF

Physics-Based Planning for Automated and Generalizable Assembly | Yunsheng Tian

Yunshen Tian | Graduation Date: 05/23/2024
PI Lead:
Professor Wojciech Matusik, MIT EECS

Yunsheng Tian is a fourth-year PhD student in the Computational Design & Fabrication Group at MIT CSAIL advised by Prof. Wojciech Matusik. His research lies in the intersection of computer graphics, robotics and machine learning. Before coming to MIT, he obtained the bachelor’s degree from Nankai University, China, advised by Prof. Bo Ren and Prof. Ming-Ming Cheng. He had also worked at Microsoft Research Asia and The University of Hong Kong as a research intern.

Abstract: Assembly planning is the core of automating product assembly, maintenance, and recycling for modern industrial manufacturing. Despite its importance and long history of research, planning for mechanical assemblies when given the final assembled state remains a challenging problem. This is due to the complexity of dealing with arbitrary 3D shapes and the highly constrained motion required for real-world assemblies. In this work, we propose a novel method to efficiently plan physically plausible assembly motion and sequences for real-world assemblies. Our method leverages the assembly-by-disassembly principle and physics-based simulation to efficiently explore a reduced search space. To evaluate the generality of our method, we define a large-scale dataset consisting of thousands of physically valid industrial assemblies with a variety of assembly motions required. Our experiments on this new benchmark demonstrate we achieve a state-of-the-art success rate and the highest computational efficiency compared to other baseline algorithms. Our method also generalizes to rotational assemblies (e.g., screws and puzzles) and solves 80-part assemblies within several minutes.

View PDF

Suboptimal Controller Synthesis for Cart-Poles and Quadrotors via Sums-of-Squares | Lujie Yang

Lujie Yang | Graduation Date: 05/30/2025
PI Lead: Professor Russ Tedrake, MIT EECS, Aero/Astro, MechE

Lujie Yang is a Ph.D. candidate at MIT EECS. She is a member of the Robot Locomotion Group at MIT CSAIL; led by Russ Tedrake.

Abstract: Sums-of-squares (SOS) optimization is a promising tool to synthesize certifiable controllers. However, many examples in the literature fail to generate impressive dynamic behaviors on complicated robotic systems. Here, we improve upon previous approaches, and show that we can apply SOS to synthesize controllers with bounded suboptimal performance for various underactuated robotic systems by finding good approximations of the value function. We summarize a unified SOS framework to synthesize both under- and over- approximations of the value function for continuous-time, control-affine systems, use these approximations to generate suboptimal controllers, and perform regional analysis on the closed-loop system driven by these controllers. We then extend the formulation to handle hybrid systems with contacts. We demonstrate that our method can generate tight under- and over- approximations of the value function with low-degree polynomials, which are used to provide stabilizing controllers for continuous-time systems including the inverted pendulum, the cart-pole, and the 3D quadrotor as well as the planar pusher, a hybrid system. To the best of our knowledge, this is the first time that a SOS-based time-invariant controller can swing up and stabilize a cart-pole, and push the planar slider to the desired pose. Demo code at https://deepnote.com/workspace/lujieyang/project/hjb- sos.

View PDF | Robot Locomotion Group

Pensieve: Microarchitectural Modeling for Security Evaluation | Yuheng Yang

Yuheng Yang | Graduation Date: 06/01/2027
PI Lead:
Assistant Professor Mengjia Yan, MIT EECS

Yuheng Yang is a PhD student at MIT EECS advised by Prof. Mengjia Yan. He works on using formal methods to design secure hardware, with a focus on mitigating timing side channels and speculative execution attacks. Recently, he has been studying microarchitecture modeling approaches and developing Pensieve, a security-oriented microarchitecture model to assist early-stage security evaluation with model checking. Before starting PhD, he received a BS degree from the University of Chinese Academy of Sciences with RISC-V core tape-out experience.

Abstract: Traditional modeling approaches in computer architecture aim to obtain an accurate estimation of performance, area, and energy of a processor design. With the advent of speculative execution attacks and their security concerns, these traditional modeling techniques fall short when used for security evaluation of defenses against these attacks. In this project, we developed Pensieve, a security evaluation framework targeting early-stage microarchitectural defenses against speculative execution attacks. At the core, it introduces a modeling discipline for systematically studying early-stage defenses. This discipline allows us to cover a space of designs that are functionally equivalent while precisely capturing timing variations due to resource contention and microarchitectural optimizations. We implement a model checking framework to automatically find vulnerabilities in designs. We use Pensieve to evaluate a series of state-of-the-art invisible speculation defense schemes, including Delay-on-Miss, InvisiSpec, and GhostMinion, against a formally defined security property, speculative non-interference. Pensieve finds Spectre-like attacks in all those defenses, including a new speculative interference attack variant that breaks GhostMinion, one of the latest defenses.

View PDF

FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback | Alan Zhao

Alan Zhao | Graduation Date: 05/05/2025
PI Lead:
Professor Edward Adelson, MIT Brain & Cognitive Sciences

Alan is a PhD student with the Perceptual Science Group at MIT CSAIL. He is advised by Professor Edward H. Adelson. Before joining MIT, he worked at Nuro Inc. as a full-time software engineer on autonomous vehicle behavior (planning & maneuvers), for a year. He received an M.S. in Robotics degree from The Robotics Institute at Carnegie Mellon University, where he was advised by Professor Oliver Kroemer at the Intelligent Autonomous Manipulation (IAM) lab. His research interest mainly lies in robotic manipulation and robot learning. In particular, he's interested in designing tactile sensors and using them to devise manipulation policies. Over the past few years, he has been working on reducing uncertainties for robotic assembly tasks by learning precise and task-oriented grasps, and learning contact-rich manipulation tasks by composing hierarchical controllers.

Abstract:

 

View PDF | Alan's Website

Olli: Accessible Data Visualizations for Blind and Low Vision Users | Jonathan Zong

Jonathan Zong | Graduation Date: 05/31/2024
PI Lead: Assistant Professor Arvind Satyanarayan, MIT EECS

Jonathan Zong is a computer scientist and visual artist who uses design to create artifacts that empower marginalized individuals and communities while advancing scholarly conversations about the technical and ethical aspects of building interactive systems. His work has been recognized by Forbes 30 Under 30 and the MIT Morningside Academy for Design Fellowship.

Abstract: Despite decades of visualization research and recent legal requirements to make web-based content accessible, web-based visualizations remain largely inaccessible to people with visual disabilities. Charts on mainstream publications are often completely invisible to screen readers (an assistive technology that transforms text and visual media into speech) or are rendered as incomprehensible strings of “graphic graphic graphic”. We present Olli, an open source library that converts visualizations into a keyboard-navigable structure accessible to screen readers. Olli enables screen reader users to access textual descriptions of data at varying levels of detail, from high level summaries to individual data points. To augment the human intelligence of blind data analysts, we leverage recent advances in large language models (LLMs) to generate additional natural language analyses that summarize complex trends and patterns in data, while also incorporating knowledge about current events and social context. Olli is available as open-source software at mitvis.github.io/olli.

View PDF