Refreshments are available starting at 10:30 a.m. The seminar will begin at 10:45 a.m.
Abstract
Large language models support a whole gamut of tasks in natural language. In contrast, unification has been more challenging in computer vision, partly due to the heterogeneity of tasks in the visual domain and a scarcity of robotics data in the physical world. How do we create powerful unified systems for vision and robotics that can be as capable and creative as their language counterparts? Aniruddha Kembhavi will first present his work on Unified IO, the first single neural model to perform a large and diverse set of AI tasks spanning classical computer vision, image synthesis, vision-and-language, and natural language processing, and his follow-up work, Unified IO 2, that brings in video, audio, and action. Then he will present his recent surprising finding in robotics: Imitating shortest path planners in simulation can produce agents that can proficiently navigate, explore, and manipulate objects in the real world, with no human data or RL and with just RGB sensing—made possible by his recent works on ProcTHOR, HoloDeck, and ObjaVerse. Finally, he will present a compelling alternative to building unified models, Visual Programming, which uses language models as code generators to invoke smaller specialized vision models. This paradigm is efficient and effective, leverages models sourced from the entire community, and scales easily to large sets of diverse tasks.
Speaker Biography
Aniruddha Kembhavi is the senior director of computer vision at the Allen Institute for AI in Seattle. He is also an affiliate associate professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Kembhavi is interested in research problems at the intersection of vision, language, and embodiment. He received his PhD from the University of Maryland under the supervision of Professor Larry Davis and also spent several years at Microsoft Bing building large-scale machine learning systems for image and video search. His work has won several awards, including Best Paper Award at the 2023 Conference on Computer Vision and Pattern Recognition and an Outstanding Paper Award at the 2022 Conference on Neural Information Processing Systems.