|
Current Research
Gesture Recognition and Surgical
Skill Evaluation
Johns Hopkins University,
Computational Interaction
and Robotics Laboratory (CIRL)
Professor Gregory Hager
January 2007-Present
Previous Research
Haptics
for Robot Assisted Minimally Invasive Surgery
Johns Hopkins University,
Haptics Exploration Laboratory
Professor Allison
Okamura
August 2004-December 2006
Haptic
Integration of IBM Manipulator
Santa Clara University, Robotics
Systems Laboratory (RSL)
Professor
Christopher Kitts
September 2003-May 2004
Underwater Robotics
Santa Clara University, Robotics
Systems Laboratory (RSL)
Professor
Christopher Kitts
September 2001-May 2003
Parallel Computing
University of Delaware, HIPERSPACE
Professor Lori Pollock
June 2003-August 2003
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
CURRENT RESEARCH
I will be starting research on the
surgical skill evaluation project. Can we create a motion
vocabulary analogous to phonemes in speech recognition to
objectively evaluate surgical performation? The gesture recognition
group consists also of graduate students Henry Lin, Zach Pezzamenti,
and Tiffany Chen. Details to come...
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
MASTERS RESEARCH
Haptics for Robot-Assisted Minimally Invasive Surgery
[ What is Haptics | Da Vinci System Overview | Sensory Substitution for Medical Devices | Sensory Substitution vs. Direct Haptic Feedback | Sensory Substitution for Virtual Fixtures]
For a crash course in my research, click
HERE to see the invited talk I gave at Santa Clara University.
What is Haptics and Teleoperation?
Haptics (pronounced HAP-tiks) is defined as "relating the sense of
touch; tactile".
The study of haptics emerged
from advances in virtual reality. It is a form of human-computer
interaction (HCI) providing an environment that one can explore
through direct interaction with their senses. It is the science of relating tactile
sensations to robotics and getting an end using to "feel" what the
robot feels. By using special equipment such as joysticks,
gloves, etc., the users can receive feedback from the forces felt on
the other end. There are two types of haptic feedback: One is a
user interacting with a virtual environment and the other is is when a
user is controlling a tele manipulation system. A tele
manipulation system is one where the user controls the robot from a
distance.
The user would move the robot
on one end and the robot will mimic the action. An example of haptics interacting with a virtual environment is when a user “feels”
a click each time they scrolled through a menu item on a computer
monitor, or when the joystick for the N64 shakes whenever Mario hits
the rough edges of a track in MarioKart . An example of a
telemanipulation system is where a user sits at one end of the room
moving his hand These examples show
when a user feels what as though they are interacting with the virtual
environment. Haptics is used to help make robotic tasks feel more realistic.
Teleoperation is defined as
operating a machine at a distance, much like a remote control.
Teleoperated robots allow humans to go places they physically can not go and explore unknown worlds such as underwater, space, hazardous environments, and the human body. By using a teleoperated
robot, a user can sit at the master's console and control a slave
device to mimic his/her actions. My previous research experiences in
robotics have led me to learn tools to conduct research and understand
how medicine and engineering go hand in hand to help benefit the
world.
There are many applications
to this research. It can be used for medical robotics to help a
surgeon using a robot or as a simulation program to help train
doctors. It can also be used for autonomous exploration of
hazardous or remote environments, undersea salvage, enabling
technologies, and manufacturing and design. The possibilities
are endless. My desire to impact the field of robotics and improve the quality of
people's lives motivates me to obtain results that have an immediate,
positive impact on society.
The SensAble
Technologies PHANTOM haptic device (photos from
www.sensable.com)
What is the Da Vinci
Robot?
Over the past ten years, tele-operated robot-assisted surgical systems are increasingly being used in minimally invasive surgery (MIS). Surgical robots, like the da
Vinci System (Intuitive Surgical Inc.), allow MIS techniques to be
performed more quickly and accurately while still looking and feeling
like traditional open surgery by providing surgeons with heightened
flexibility and precision. For the patient there is less pain, less
blood loss and faster recovery time.
The da Vinci surgical system is the first laparoscopic surgical robot in existence.
Laparoscopic surgery uses a tube containing a tiny camera that allows
the surgeon to see inside the abdomen on a high resolution video
screen. Incisions made during this type of surgery tend to be smaller
than those made during conventional types of surgery. The da Vinci
system is also the first commercial surgical robotic system to be FDA
approved and is widely used in clinical hospitals all over the world.
It enables surgeons to perform minimally invasive surgery (MIS) and
aids in reducing trauma, postoperative pain and surgical complications
for patients by operating with the flexibility of open surgery through
tiny ports.
There
are two major components of the da Vinci system: the viewing console,
where the user sits and manipulates the robot and the surgical arm
tower that moves the instruments to perform the surgery. To operate
the device, a surgeon sits at the console several feet away from the
arm tower, peering through an eyepiece that displays 3D images from a
high powered camera while remotely manipulating robotic instruments
that performs the surgery. The instruments are designed to
duplicate the dexterity of the surgeon’s forearms and wrists.
The surgery is performed by three incision holes. The first
incision is for a tiny camera that displays the images on the console
to the user. The other two ports are for the surgical tools
which can bend back and forth, side to side and move in a full circle.
The robot is controlled by the surgeon’s hands which are connected to
manipulation controls on the other side of the room. This robot
allows minimally invasive surgical techniques to be performed more
quickly and accurately while still looking and feeling like open
surgery. It is designed to provide surgeons with the flexibility of
traditional open surgery while operating through tiny ports and the
precision of the surgery is heightened. For the patient, there’s
less pain, less blood loss and faster recovery time.
To
recap, surgical procedures routinely performed today using MIS
techniques may be performed more quickly and easily with the da Vinci
Surgical System.
 
The pictures above are from
Intuitive's website at
www.intuitivesurgical.com
Oprah's video clip on the
daVinci robot:
https://ftp.intusurg.com/anonymous/public/
Sensory Substitution for Medical Robotics
With the high costs in the OR,
creating surgical systems that integrate novel computer and
human/machine interface technologies will revolutionize surgical
procedures, extending the surgeon's abilities to achieve better
outcomes at lower costs. Haptic (force and tactile feedback) has
been proposed as a way to further enhance the performance of these
systems. A limitation to the current generation of MIS robots is the
lack of haptic feedback where the operator relies solely on visual
feedback to determine the amount of force being applied. Implementing
direct haptic feedback to the surgeon's hands remains impractical for
clinical application because of the cost and time of applying force
sensors to disposable tools and the current limitations in sensing and
control technologies. The goals of this work are to develop an
intuitive augmented reality system for feedback of force information
through sensory substitution, and to evaluate its performance in a
surgical task such as knot tying.
Work done in collaboration with graduate students
Tope Akinibiyi.Video footage from our
project. The first one is the robotic arms being controlled by
the surgeon tying an instrument tie. The second is a 2D
representation of what the surgeon sees in his console.
pictures\DavinciAction.mpeg
pictures\VisualOverlayVideo.mpg
With Sensory Substitution Overlay (left), Without Sensory Substitution
Overlay (right)
Click on picture above to see video clip.
Sensory Substitution with Virtual Fixtures
Virtual fixtures are software implemented restrictions on the
users moment into a region or by influencing movement along a certain
path. They provide better-than-human levels of accuracy and precision
which require a human in the loop. I will be exploring how to implement
6 dof virtual fixtures into the new API of the da Vinci system for
three surgical task: (1) blunt dissection, (2) knot tying, and (3)
suturing. We wish to show that virtual fixtures augment and
improve the execution of surgical task involving fine suture
manipulation. Work done in collaboration with
graduate student
Lawton Verner.
Funding Source: NIH Grant R01 EB002004, Whitaker R6-02-911
Special Thanks to the Minimally Invasive Surgery Training Center (MISTC)
and Dr. David Yuh. ~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
UNDERGRADUATE RESEARCH
[
Haptics | Underwater Robotics
| Parallel Computing |
Sound Manipulation]
Haptic Integration of IBM Manipulator
[PDF
of Senior Design Report]
As the first design team at Santa Clara
University to experiment with force feedback and integrate haptic
feedback into a robotic arm, helping make tasks more realistic for
robotic operators. The main objective of our interdisciplinary
team of six was to modify an existing IBM robotic arm, model number
7545, focusing on the integration of haptics into the operation and
control of the robot. This integration was to allow the user to feel
differences in the rigidity of the objects picked up by the robot
gripper. Therefore, the system would be providing cutaneous feedback
to the user, so that the user will be able to respond to forces. For
example, if the gripper was to pick up a rock, the user could
distinguish the hardness of that object as opposed to squeezing a
spring, which would feel pliable. Another objective was to perform the
proper maintenance required for the robot since it was not functional
and out of use. After getting the robot in a functional state, a new
user interface was designed and manufactured that would not only move
the arm and gripper, but also be simplistic and user friendly.
The pictures of user end gripper (left) and a picture of the IBM
manipulator with robotic gripper (right)
Senior Design Team (Force Feelin')
Program Slicing for OpenMP
Shared Memory Parallel Programs
- Summer 03
I spent ten weeks in the
summer after my junior year doing research on parallel computing at
the University of Delaware as part of the NSF CRA-W
program. After studying the current program
slicing algorithm, I designed and implemented a user friendly slicing
tool for OpenMP written
by Matthew Bridges.
This privilege gave me the opportunity to experience the day to day
life of a Ph.D student.
The interface was made so that a parallel programmer who knows little
about the underlying infrastructure could run it. Slicing is used for
software development and maintenance activities such as program
understanding, software testing, and debugging. By extending and
modifying an interprocedural slicing algorithm for sequential
programs, and an intraprocedural slicing algorithm for parallel
programs, we have developed a technique for static interprocedural
slicing of shared memory parallel programs, written using OpenMP
explicitly parallel constructs. OpenMP is the standard for explicitly
parallel shared memory programming. To see a journal of my experience
that summer, click
HERE
Underwater Robotics - Fall 01->Spring 03
During my sophomore year, I built a rudimentary,
low-cost tethered vehicle is made of PVC tubing and uses bilge pumps
as thrusters along with six other students. Our
interdisciplinary team installed a camera and an internet control
system allow students to fully control the vehicle and to rapidly
prototype new ideas on a simple engineering platform. Throughout the year, we
constructed a low-cost but robust Remotely Operating Vehicle which
consisted of PVC pipes as a frame, six thrusters for
propulsion and maneuvering, a tether which supplies controlled power
from a 12 volt DC power source, floats, and a controller using push
buttons to control the thrusters. The robot sent a stream of video so
that a recording was displayed on deck. I configured and tested a safety system for the
robot. This involves programming a BasicStamp microcontroller,
interfacing relays and setting up communications between the basic
stamp and DSP.
Picture of Seafox (courtesy of SCU RSL)
That summer, I worked independently with Professor Christopher
Kitts to do research and expand the vehicle’s capabilities. Utilizing my electronics experience, soldering and finishing
up the wiring and circuitry, I used my knowledge of C coding and
programmed the underwater ROV to be controlled through the Internet,
through the relays, through the Z world microcontroller, and through
the hand held controller.
Junior year I progressed to work
on the Santa Clara University’s Triton undersea robot during my junior
year. This shallow-water (<1000 ft) 3 Hp
tethered vehicle is being used for a variety of marine science studies
as well as for several robotic technology studies.
I became in charge of coding. I
analyzed and tested the capacity of the code and then worked to
manipulate the abilities of the robot.
Pictures of Triton (Courtesy of SCU RSL)
Sound manipulation and web development with Prof. Ogunfunmi -
Fall 00->Spring01
Starting fall quarter freshman year, I took the initiative of doing
research learning C++ and the principles of digital signal
processing. My project expanding to working with web-based
applications and manipulating sound clips.
|
