13-09-2012, 03:33 PM
IMAGE PROCESSING METHODS for INTERACTIVE ROBOT CONTROL
[attachment=33723]
Abstract
In this paper we describe a straight forward technique for tracking a human hand based on images acquired by an active stereo camera system. There are many applications for Image Processing like surveillance, navigation, and robotics. Robotics is a very interesting field and promises future development so it is chosen as an example to explain the various aspects involved in Image Processing.
We demonstrate the implementation of this method on an anthropomorphic assistance
robot as part of a multi-modal man-machine interaction system: detecting the hand-position, the robot can interpret a human pointing gesture as the specification of a target object to grasp.
INTRODUCTION
Image processing pertains to the alteration and analysis of pictorial information. Common case of image processing is the adjustment of brightness and contrast controls on a television set by doing this we enhance the image until its subjective appearing to us is most appealing. The biological system (eye, brain) receives, enhances, and dissects analyzes and stores mages at enormous rates of speed.
In the field of industrial robotics, the interaction between man and machine typically consists of programming and maintaining the machine by the human operator. For safety reasons, a direct contact between the working robot
and the human has to be prevented. As long as the robots act out preprogrammed behaviors only, a direct interaction between man and machine is not necessary anyway. However, if the robot is to assist a human e.g. in a complex assembly task, it is necessary to have means of exchanging information about the current scenario between man and machine in real time. For this purpose, the classical computer devices like keyboard, mouse and monitor are not the best choice as they require an encoding and decoding of information: if, for instance, the human operator
HAND DETECTION
Keeping all this in mind, we have decided to implement another very simple recognition method: we detect the hand on the basis of its typical skin color. In most situations, this parameter turned out to be sufficiently robust for our purpose. However, we have implemented a dynamical initialization phase in which the robot can “learn” the skin color of the specific operator (see Fig. 2). In the following, we will describe the recognition method in detail.
Stereo-based Localization
Our test scenario aims at an assembly situation in which man and machine should solve a given task sharing the same workspace is located between the robot and the human. After the localization of the skin color cluster belonging to the operators hand by selecting the nearest cluster to the cameras with a sufficient size (see Fig. below), we calculate the disparity of the pixel area at the bottom of this preselected hand cluster. This area typically represents the fingertip of the user for a normal pointing gesture in this environment. The disparity provides the 3-D position by a simple transformation. This limitation on such a small pixel area enables a very fast determination of the hand position.
Tracking
By restricting the time consuming calculation of the disparity on this single point, the described method for determining the hand localization is very fast. Therefore we can obtain a new hand position so frequently that a fast tracking of the hand is possible. By a simple coordinate transformation we translate the 3-D coordinate of the finger-tip into a 2-D pan-tilt angle for the motors of the stereo camera head such that the gaze direction of the robot follows the current hand position. As stated in the introduction, this behavior allows for a very natural interactive control of the detected hand position. During the tracking of the hand, human and robot become a coupled system. If the operator recognizes that the robot has lost track of the hand, he can simply move the hand into the focus of view so that the tracking mechanism “snaps in” again.
OBJECT RECOGNITION
The purpose of hand-position-recognition is the specification of an object in the workspace by pointing to it. The detection of the hand position serves as a pre-selection of the region of interest (ROI) in the image. In our case, the ROI is defined as a triangular area in the bird’s eye view of the table which covers a sector with 30 degrees opening angle in front of the fingertip. Within this sector the object which is nearest to the fingertip is considered to be specified by the user’s pointing gesture. The specification might be supported by a verbal command to avoid ambiguities. At this point the robot has simply to detect or, if there are further ambiguities, to recognize the specified object.
INTEGRATION of OTHER SOURCE of INFORMATION
The major goal of our research is the design of a robot system which does not only have an anthropomorphic body structure but which is able to communicate with the operator on the basis of natural communication channels. Recognizing the human’s hand position to identify a target object for grasping is only one of these channels. Within the current project we design several other means of man machine interaction. We have, for instance, implemented a module for the recognition of the human’s gaze direction. This module can identify the position of the human’s eyes so that a rough estimate of the focus of attention is possible. In turn, the operator can see the gaze direction of the robot head so that a natural way of identifying the current region of interest on the table is possible. In addition we have built a speech recognition system which can identify spoken keyword commands. By means of a speech synthesizer, the robot can express its behavioral state. On the basis of natural commands it is possible to guide the robot’s manipulator to a certain target or to terminate an incorrect behavior (such as the selection of a wrong object) by a spoken command. Another communication channel is based on touch. We used a touch sensitive artificial skin by means of which the operator can correct the posture of the robot arm.
CONCLUSION and OUTLOOK
We have presented a basic concept for a vision based interactive robot control as a part of multi-modal man machine interaction system. Implementing a straight forward technique for tracking a human hand and extracting a pointing gesture we could demonstrate how man and machine become a coupled system in a very natural way. The object recognition system can use those pointing gestures to minimize the searching area or, in the common case, can search the whole scene for a special object. Thereby the object’s position and orientation is extracted very robustly. By using multiple channels of information, the whole system is able to overcome ambiguities and the robot can react in an adequate matter.
[attachment=33723]
Abstract
In this paper we describe a straight forward technique for tracking a human hand based on images acquired by an active stereo camera system. There are many applications for Image Processing like surveillance, navigation, and robotics. Robotics is a very interesting field and promises future development so it is chosen as an example to explain the various aspects involved in Image Processing.
We demonstrate the implementation of this method on an anthropomorphic assistance
robot as part of a multi-modal man-machine interaction system: detecting the hand-position, the robot can interpret a human pointing gesture as the specification of a target object to grasp.
INTRODUCTION
Image processing pertains to the alteration and analysis of pictorial information. Common case of image processing is the adjustment of brightness and contrast controls on a television set by doing this we enhance the image until its subjective appearing to us is most appealing. The biological system (eye, brain) receives, enhances, and dissects analyzes and stores mages at enormous rates of speed.
In the field of industrial robotics, the interaction between man and machine typically consists of programming and maintaining the machine by the human operator. For safety reasons, a direct contact between the working robot
and the human has to be prevented. As long as the robots act out preprogrammed behaviors only, a direct interaction between man and machine is not necessary anyway. However, if the robot is to assist a human e.g. in a complex assembly task, it is necessary to have means of exchanging information about the current scenario between man and machine in real time. For this purpose, the classical computer devices like keyboard, mouse and monitor are not the best choice as they require an encoding and decoding of information: if, for instance, the human operator
HAND DETECTION
Keeping all this in mind, we have decided to implement another very simple recognition method: we detect the hand on the basis of its typical skin color. In most situations, this parameter turned out to be sufficiently robust for our purpose. However, we have implemented a dynamical initialization phase in which the robot can “learn” the skin color of the specific operator (see Fig. 2). In the following, we will describe the recognition method in detail.
Stereo-based Localization
Our test scenario aims at an assembly situation in which man and machine should solve a given task sharing the same workspace is located between the robot and the human. After the localization of the skin color cluster belonging to the operators hand by selecting the nearest cluster to the cameras with a sufficient size (see Fig. below), we calculate the disparity of the pixel area at the bottom of this preselected hand cluster. This area typically represents the fingertip of the user for a normal pointing gesture in this environment. The disparity provides the 3-D position by a simple transformation. This limitation on such a small pixel area enables a very fast determination of the hand position.
Tracking
By restricting the time consuming calculation of the disparity on this single point, the described method for determining the hand localization is very fast. Therefore we can obtain a new hand position so frequently that a fast tracking of the hand is possible. By a simple coordinate transformation we translate the 3-D coordinate of the finger-tip into a 2-D pan-tilt angle for the motors of the stereo camera head such that the gaze direction of the robot follows the current hand position. As stated in the introduction, this behavior allows for a very natural interactive control of the detected hand position. During the tracking of the hand, human and robot become a coupled system. If the operator recognizes that the robot has lost track of the hand, he can simply move the hand into the focus of view so that the tracking mechanism “snaps in” again.
OBJECT RECOGNITION
The purpose of hand-position-recognition is the specification of an object in the workspace by pointing to it. The detection of the hand position serves as a pre-selection of the region of interest (ROI) in the image. In our case, the ROI is defined as a triangular area in the bird’s eye view of the table which covers a sector with 30 degrees opening angle in front of the fingertip. Within this sector the object which is nearest to the fingertip is considered to be specified by the user’s pointing gesture. The specification might be supported by a verbal command to avoid ambiguities. At this point the robot has simply to detect or, if there are further ambiguities, to recognize the specified object.
INTEGRATION of OTHER SOURCE of INFORMATION
The major goal of our research is the design of a robot system which does not only have an anthropomorphic body structure but which is able to communicate with the operator on the basis of natural communication channels. Recognizing the human’s hand position to identify a target object for grasping is only one of these channels. Within the current project we design several other means of man machine interaction. We have, for instance, implemented a module for the recognition of the human’s gaze direction. This module can identify the position of the human’s eyes so that a rough estimate of the focus of attention is possible. In turn, the operator can see the gaze direction of the robot head so that a natural way of identifying the current region of interest on the table is possible. In addition we have built a speech recognition system which can identify spoken keyword commands. By means of a speech synthesizer, the robot can express its behavioral state. On the basis of natural commands it is possible to guide the robot’s manipulator to a certain target or to terminate an incorrect behavior (such as the selection of a wrong object) by a spoken command. Another communication channel is based on touch. We used a touch sensitive artificial skin by means of which the operator can correct the posture of the robot arm.
CONCLUSION and OUTLOOK
We have presented a basic concept for a vision based interactive robot control as a part of multi-modal man machine interaction system. Implementing a straight forward technique for tracking a human hand and extracting a pointing gesture we could demonstrate how man and machine become a coupled system in a very natural way. The object recognition system can use those pointing gestures to minimize the searching area or, in the common case, can search the whole scene for a special object. Thereby the object’s position and orientation is extracted very robustly. By using multiple channels of information, the whole system is able to overcome ambiguities and the robot can react in an adequate matter.