ACADEMIC LINKS
Research
CV
Publications
Classes
News

MY FUN PAGES
Personal
Pictures
Restaurant Review
Baltimore Activities
My Lifetime Book List
Links


HOMEPAGE
Homepage

  

                                          [ Current Research | Masters Research | Undergraduate Research]       

Current Research

              Gesture Recognition and Surgical Skill Evaluation
               Johns Hopkins University, Computational Interaction and Robotics Laboratory (CIRL)
               Professor Gregory Hager
               January 2007-Present

Previous Research

               Haptics for Robot Assisted Minimally Invasive Surgery
               Johns Hopkins University, Haptics Exploration Laboratory
               Professor Allison Okamura
               August 2004-December 2006

               Haptic Integration of IBM Manipulator
               Santa Clara University, Robotics Systems Laboratory (RSL)
                Professor Christopher Kitts
                September 2003-May 2004

                Underwater Robotics
                Santa Clara University, Robotics Systems Laboratory (RSL)
                Professor Christopher Kitts
                September 2001-May 2003

               Parallel Computing
               University of Delaware, HIPERSPACE
               Professor Lori Pollock
               June 2003-August 2003
 

~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~

CURRENT RESEARCH

I will be starting research on the surgical skill evaluation project. Can we create a motion vocabulary analogous to phonemes in speech recognition to objectively evaluate surgical performation? The gesture recognition group consists also of graduate students Henry Lin, Zach Pezzamenti, and Tiffany Chen. Details to come...

~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~

MASTERS RESEARCH

Haptics for Robot-Assisted Minimally Invasive Surgery

[ What is Haptics | Da Vinci System Overview | Sensory Substitution for Medical Devices | Sensory Substitution vs. Direct Haptic Feedback | Sensory Substitution for Virtual Fixtures]

For a crash course in my research, click HERE to see the invited talk I gave at Santa Clara University.


What is Haptics and Teleoperation?

Haptics (pronounced HAP-tiks) is defined as "relating the sense of touch; tactile".  The study of haptics emerged from advances in virtual reality. It is a form of human-computer interaction (HCI) providing an environment that one can explore through direct interaction with their senses. It is the science of relating tactile sensations to robotics and getting an end using to "feel" what the robot feels.  By using special equipment such as joysticks, gloves, etc., the users can receive feedback from the forces felt on the other end.  There are two types of haptic feedback: One is a user interacting with a virtual environment and the other is is when a user is controlling a tele manipulation system.  A tele manipulation system is one where the user controls the robot from a distance.  The user would move the robot on one end and the robot will mimic the action. An example of haptics interacting with a virtual environment is when a user “feels” a click each time they scrolled through a menu item on a computer monitor, or when the joystick for the N64 shakes whenever Mario hits the rough edges of a track in MarioKart .  An example of a telemanipulation system is where a user sits at one end of the room moving his hand These examples show when a user feels what as though they are interacting with the virtual environment.  Haptics is used to help make robotic tasks feel more realistic.

Teleoperation is defined as operating a machine at a distance, much like a remote control. Teleoperated robots allow humans to go places they physically can not go and explore unknown worlds such as underwater, space, hazardous environments, and the human body. By using a teleoperated robot, a user can sit at the master's console and control a slave device to mimic his/her actions. My previous research experiences in robotics have led me to learn tools to conduct research and understand how medicine and engineering go hand in hand to help benefit the world.

There are many applications to this research.  It can be used for medical robotics to help a surgeon using a robot or as a simulation program to help train doctors.  It can also be used for autonomous exploration of hazardous or remote environments, undersea salvage, enabling technologies, and manufacturing and design.  The possibilities are endless. My desire to impact the field of robotics and improve the quality of people's lives motivates me to obtain results that have an immediate, positive impact on society.


   The SensAble Technologies PHANTOM haptic device (photos from www.sensable.com)
 

What is the Da Vinci Robot?

Over the past ten years, tele-operated robot-assisted surgical systems are increasingly being used in minimally invasive surgery (MIS). Surgical robots, like the da Vinci System (Intuitive Surgical Inc.), allow MIS techniques to be performed more quickly and accurately while still looking and feeling like traditional open surgery by providing surgeons with heightened flexibility and precision. For the patient there is less pain, less blood loss and faster recovery time. The da Vinci surgical system is the first laparoscopic surgical robot in existence.  Laparoscopic surgery uses a tube containing a tiny camera that allows the surgeon to see inside the abdomen on a high resolution video screen. Incisions made during this type of surgery tend to be smaller than those made during conventional types of surgery. The da Vinci system is also the first commercial surgical robotic system to be FDA approved and is widely used in clinical hospitals all over the world.  It enables surgeons to perform minimally invasive surgery (MIS) and aids in reducing trauma, postoperative pain and surgical complications for patients by operating with the flexibility of open surgery through tiny ports.

There are two major components of the da Vinci system: the viewing console, where the user sits and manipulates the robot and the surgical arm tower that moves the instruments to perform the surgery. To operate the device, a surgeon sits at the console several feet away from the arm tower, peering through an eyepiece that displays 3D images from a high powered camera while remotely manipulating robotic instruments that performs the surgery.  The instruments are designed to duplicate the dexterity of the surgeon’s forearms and wrists.  The surgery is performed by three incision holes.  The first incision is for a tiny camera that displays the images on the console to the user.  The other two ports are for the surgical tools which can bend back and forth, side to side and move in a full circle.  The robot is controlled by the surgeon’s hands which are connected to manipulation controls on the other side of the room.  This robot allows minimally invasive surgical techniques to be performed more quickly and accurately while still looking and feeling like open surgery. It is designed to provide surgeons with the flexibility of traditional open surgery while operating through tiny ports and the precision of the surgery is heightened.  For the patient, there’s less pain, less blood loss and faster recovery time.

To recap, surgical procedures routinely performed today using MIS techniques may be performed more quickly and easily with the da Vinci Surgical System.

The pictures above are from Intuitive's website at www.intuitivesurgical.com

Oprah's video clip on the daVinci robot:
https://ftp.intusurg.com/anonymous/public/
 

Sensory Substitution for Medical Robotics

With the high costs in the OR, creating surgical systems that integrate novel computer and human/machine interface technologies will revolutionize surgical procedures, extending the surgeon's abilities to achieve better outcomes at lower costs.  Haptic (force and tactile feedback) has been proposed as a way to further enhance the performance of these systems. A limitation to the current generation of MIS robots is the lack of haptic feedback where the operator relies solely on visual feedback to determine the amount of force being applied. Implementing direct haptic feedback to the surgeon's hands remains impractical for clinical application because of the cost and time of applying force sensors to disposable tools and the current limitations in sensing and control technologies. The goals of this work are to develop an intuitive augmented reality system for feedback of force information through sensory substitution, and to evaluate its performance in a surgical task such as knot tying. Work done in collaboration with graduate students Tope Akinibiyi.

Video footage from our project.  The first one is the robotic arms being controlled by the surgeon tying an instrument tie.  The second is a 2D representation of what the surgeon sees in his console.

pictures\DavinciAction.mpeg
pictures\VisualOverlayVideo.mpg

     
With Sensory Substitution Overlay (left), Without Sensory Substitution Overlay (right)
 Click on picture above to see video clip.

Sensory Substitution with Virtual Fixtures

Virtual fixtures are software implemented restrictions on the users moment into a region or by influencing movement along a certain path. They provide better-than-human levels of accuracy and precision which require a human in the loop. I will be exploring how to implement 6 dof virtual fixtures into the new API of the da Vinci system for three surgical task: (1) blunt dissection, (2) knot tying, and (3) suturing.  We wish to show that virtual fixtures augment and improve the execution of surgical task involving fine suture manipulation. Work done in collaboration with graduate student Lawton Verner.

Funding Source: NIH Grant R01 EB002004, Whitaker R6-02-911
Special Thanks to the Minimally Invasive Surgery Training Center (MISTC) and Dr. David Yuh.

~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
UNDERGRADUATE RESEARCH

       [ Haptics | Underwater Robotics | Parallel Computing | Sound Manipulation]

 

Haptic Integration of IBM Manipulator

[PDF of Senior Design Report]

As the first design team at Santa Clara University to experiment with force feedback and integrate haptic feedback into a robotic arm, helping make tasks more realistic for robotic operators.  The main objective of our interdisciplinary team of six was to modify an existing IBM robotic arm, model number 7545, focusing on the integration of haptics into the operation and control of the robot. This integration was to allow the user to feel differences in the rigidity of the objects picked up by the robot gripper. Therefore, the system would be providing cutaneous feedback to the user, so that the user will be able to respond to forces. For example, if the gripper was to pick up a rock, the user could distinguish the hardness of that object as opposed to squeezing a spring, which would feel pliable. Another objective was to perform the proper maintenance required for the robot since it was not functional and out of use. After getting the robot in a functional state, a new user interface was designed and manufactured that would not only move the arm and gripper, but also be simplistic and user friendly.

 
The pictures of user end gripper (left) and a picture of the IBM manipulator with robotic gripper (right)

Senior Design Team (Force Feelin')

Program Slicing for OpenMP Shared Memory Parallel Programs  - Summer 03   

 I spent ten weeks in the summer after my junior year doing research on parallel computing at the University of Delaware as part of the NSF CRA-W program.  After studying the current program slicing algorithm, I designed and implemented a user friendly slicing tool for OpenMP written by Matthew Bridges.  This privilege gave me the opportunity to experience the day to day life of a Ph.D student.   The interface was made so that a parallel programmer who knows little about the underlying infrastructure could run it. Slicing is used for software development and maintenance activities such as program understanding, software testing, and debugging. By extending and modifying an interprocedural slicing algorithm for sequential programs, and an intraprocedural slicing algorithm for parallel programs, we have developed a technique for static interprocedural slicing of shared memory parallel programs, written using OpenMP explicitly parallel constructs. OpenMP is the standard for explicitly parallel shared memory programming.  To see a journal of my experience that summer, click HERE
 

Underwater Robotics - Fall 01->Spring 03

During my sophomore year, I built a rudimentary, low-cost tethered vehicle is made of PVC tubing and uses bilge pumps as thrusters along with six other students.  Our interdisciplinary team installed a camera and an internet control system allow students to fully control the vehicle and to rapidly prototype new ideas on a simple engineering platform.   Throughout the year, we constructed a low-cost but robust Remotely Operating Vehicle which consisted of  PVC pipes as a frame, six thrusters for propulsion and maneuvering, a tether which supplies controlled power from a 12 volt DC power source, floats, and a controller using push buttons to control the thrusters.  The robot sent a stream of video so that a recording was displayed on deck.   I configured and tested a safety system for the robot.  This involves programming a BasicStamp microcontroller, interfacing relays and setting up communications between the basic stamp and DSP.  


Picture of Seafox (courtesy of SCU RSL)

That summer, I worked independently with Professor Christopher Kitts to do research and expand the vehicle’s capabilities.  Utilizing my electronics experience, soldering and finishing up the wiring and circuitry, I used my knowledge of C coding and programmed the underwater ROV to be controlled through the Internet, through the relays, through the Z world microcontroller, and through the hand held controller.

 Junior year I progressed to work on the Santa Clara University’s Triton undersea robot during my junior year.  This shallow-water (<1000 ft) 3 Hp tethered vehicle is being used for a variety of marine science studies as well as for several robotic technology studies.  I became in charge of coding.  I analyzed and tested the capacity of the code and then worked to manipulate the abilities of the robot.  


  
  

Pictures of Triton (Courtesy of SCU RSL)

Sound manipulation and web development with Prof. Ogunfunmi - Fall 00->Spring01

Starting fall quarter freshman year, I took the initiative of doing research learning C++ and the principles of digital signal processing.  My project expanding to working with web-based applications and manipulating sound clips.

 

 

 


Additional Resources :
Home Page | JHU Engineering | Haptics Lab | | ERC Lab |

Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21211