With the development of information technology in our society, we can expect that computer systems to a larger extent will be embedded into our environment. These environments will impose needs for new types of human-computer-interaction, with interfaces that are natural and easy to use. In particular, the ability to interact with computerized equipment without need for special external equipment is attractive.
Today, the keyboard, the mouse and the remote control are used as the main interfaces for transferring information and commands to computerized equipment. In some applications involving three-dimensional information, such as visualization, computer games and control of robots, other interfaces based on trackballs, joysticks and datagloves are being used. In our daily life, however, we humans use our vision and hearing as main sources of information about our environment. Therefore, one may ask to what extent it would be possible to develop computerized equipment able to communicate with humans in a similar way, by understanding visual and auditive input.
The purpose of this project is to develop new perceptual interfaces for human-computer-interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces based on keyboards, mouses, remote controls, data gloves or speech. Examples of applications of hand gestures analysis include:
* Control of consumer electronics
* Interaction with visualization systems
* Control of mechanical systems
* Computer games
Main advantages of using visual input in this context are that visual information makes it possible to communicate with computerized equipment at a distance, without need for physical contact with the equipment to be controlled. Compared to speech commands, hand gestures are advantageous in noisy environments, in situations where speech commands would be disturbing, as well as for communicating quantitative information and spatial relationships. The idea is that the user should be able to control equipment in his environment as he is, and without need for specialized external equipment, such as a remote control,
The project combines CVAPs expertise in computer vision with CIDs experience in designing and evaluating of new human-machine interfaces. Initially, the focus is on developing algorithms for recognizing hand gestures and to build prototype systems that make it possible to test perceptual interfaces in practical applications. An important component of the work is to perform continuous user studies in close connection with the development work.
It may be worth emphasizing that that the aim is not to recognize the kind of expressive gestures that are tightly coupled to our speech, or sign languages aimed at inter-human communication. The goal is to explore hand gestures suitable for various control tasks in human-machine interaction. Multi-modal interfaces including hand gestures, face and gaze tracking and speech recognition will also be considered.
Detailed information
A prototype system for computer vision based human computer interaction -- Includes a demonstration of how a user can control different types of consumer electronics using hand gestures.
Camera mouse control -- A simple demonstration of how it the cursor on a computer screen can be controlled by hand motions, and how hand gestures can be used for interacting with programs via a cursor controlled by hand gestures.
Simultaneous hand tracking and hand posture recognition -- Includes a demonstration where the estimated hand motions and the recognized hand postures control the motion of drawing device. (In the video clip below, the type of hand posture controls the type of action that is performed, while the estimated hand motion controls the motion of the pencil, the cursor or the drawing.)
The 3-D hand mouse -- A demonstration of how it is possible to measure the three-dimensional motion (translations and rotations) of a human hand, and to use such motion estimates to control the three-dimensional motion of other computerized devices. (In the video clip below, the cube moves according to the estimated motion of the hand.)
