VICS - Visual Interaction Cues for Local Based HCI

People - Dr. Gregory Hager, Dr. Darius Burschka, Dr. Jason Corso, Dr. Grant Ye

Sponsored by the National Science Foundation.

Description The Visual Interaction Cues (VICs) project is focused on developing new techniques for vision-based human computer interaction (HCI). The VICs paradigm is a methodology for vision-based interaction operating on the fundamental premise that, in general vision-based HCI settings, global user modeling and tracking are not necessary. For example, when a person presses the number-keys while making a telephone call, the telephone maintains no notion of the user. Instead, it only recognizes the action of pressing a key. In contrast, typical methods for vision-based HCI attempt to perform global user tracking to model the interaction. In the telephone example, such methods would require a precise tracker for the articulated motion of the hand. However, such techniques are computationally expensive, prone to error and the re-initialization problem, prohibit the inclusion of an arbitrary number of users, and often require a complex gesture-language the user must learn. In the VICs paradigm, we make the observation that analyzing the local region around an interface component (the telephone key, for example) will yield sufficient information to recognize user actions.

The principled techniques of the VICs paradigm are applicable in general HCI settings as well as advanced simulation and virtual reality. We are actively investigating 2D, 2.5D, and 3D environments; we've developed a new HCI platform called the 4D Touchpad (figure below) where vision-based methods can complement the conventional mouse and keyboard.

In the VICs project, we study both low-level image analysis techniques and high-level gesture language modeling. In low-level image analysis, we use deterministic (color, shape, motion, etc.), machine learning (e.g. neural networks), and dynamic modeling (e.g. Hidden Markov Models) to model the spatio-temporal characteristics of various hand gestures. We have constructed a highlevel language model that integrates a set of low-level gestures into a single, coherent probabilistic framework. In the language model, every low-level gesture is called a Gesture Word, and each complete action is a sequence of these words called a Gesture Sentence.

Examples of the algorithms being studied follow (videos are below):

Adaptive Background Modeling

Intelligent Buttons

The 4D Touchpad - A new platform for the development of next-generation interfaces.
It facilitates unencumbered interaction with an architecture that provides core functionality for general vision-based interfaces.

The new 4DT system. Interaction components are rendered over a flat monitor.

System calibration and hand segmentation of the new 4DT system.

PUBLICATIONS

• Darius Burschka, Guangqi Ye, Jason J. Corso, Gregory D. Hager, Practical Approach for Integrating Vision-Based Methods into Interactive 2D/3D Applications, CIRL Technical Report CIRL-TR-2005-01, 2005.
• Guangqi Ye, Jason J. Corso, and Gregory D. Hager. Real-Time Vision for Human-Computer Interaction, chapter 7: Visual Modeling of Dynamic Gestures Using 3D Appearance and Motion Features, pages 103-120. Springer-Verlag, 2005.
• Jason J. Corso and Guangqi Ye and Gregory D. Hager. Analysis of Multi-Modal Gestures with a Coherent Probabilistic Graphical Model. Virtual Reality, 8(4):242-252, September 2005. The original publication is available at www.springerlink.com.
• Guangqi Ye, Jason J. Corso, Darius Burschka and Gregory D. Hager. VICs: A Modular HCI Framework Using Spatio-Temporal Dynamics. Machine Vision and Applications, 16(1):13-20, 2004. The original publication is available at www.springerlink.com.
• Guangqi Ye, Jason J. Corso, and Gregory D. Hager. Gesture Recognition using 3D Appearance and Motion Features. CVPR 2004 workshop on real-time vision for human-computer interaction.
  [ Talk at CVPR 2004 RTV4HCI workshop ]
  
• Jason Corso, Darius Burschka, and Gregory D. Hager. The 4D Touchpad: Unencumbered HCI With VICs. In Proceedings of CVPRHCI. Madison, Wisconsin, 2003.
  [ Talk at CVPR 2003 HCI workshop ]
  
• Guangqi Ye, Jason Corso, Darius Burschka, and Gregory D. Hager. Vics: A modular vision-based hci framework. In Proceedings of 3rd International Conference on Computer Vision Systems. Pages 257-267, Graz, Austria, 2003.
  [ Talk at ICVS 2003 ]
  
• Jason Corso, Darius Burschka, and Gregory D. Hager. Direct plane tracking in stereo image for mobile navigation. Proceedings of the International Conference on Robotics and Automaiton, pages 875-880,2003.
  [ Talk at ICRA 2003 ]
  
• Jason Corso and Gregory D. Hager. Planar surface tracking using direct stereo. Technical report, 2002. CIRL Lab Technical Report.
• Guangqi Ye and Gregory D. Hager. Appearance-based visual interaction. Technical report, 2002. CIRL Lab Technical Report.

Demo Movies -

• Gesture-Based Interface for X Window [download]
• Real-time Gesture Recognition Demo [download]
• Planar Surface Tracking [download]
• The 4D Touchpad Demo 1 [download]
• Planar Surface Tracking [download]
• Adaptive Background Modeling Demo 1 [download]
• Adaptive Background Modeling Demo 2 [download]
• Action Learning Demo 1 [download]
• Composite of Vision-based Interactive Work [download]

More Information - At Jason Corso's Project Page