n order to depict a taxonomy of multimodal human-computer interaction we will have to clarify a number of concepts and issues. The first assumption is that there are minimally two separate agents involved, one human and one machine. They are physically separated, but are able to exchange information through a number of information channels. As schematically shown in figure 1.1 , we will make the following definitions.
Figure 1.1 : A model for the identification of basic processes in human-computer interaction. Note that in fact two loops exists: the intrinsic feedback as in eye-hand coordination, and the extrinsic loop, imposed by the computer.
There are two basic processes involved on the side of the human user: Perception and Control. Note that we take the perspective of the human process throughout this document. With respect to the Perceptive process, we can make a distinction between:
* Human Input Channels (HIC) and
* Computer Output Media (COM)
Within the Control process, we can make a distinction between:
* Human Output Channels (HOC) and
* Computer Input Modalities (CIM)
Then, within both of the agents, a cognitive or computational component can be identified, which processes the incoming input information and prepares the output. Also, at this intermediate cognitive level, intentional parameters will influence the processing, either implicitly, such as by design, in the case of non-intelligent agents, or explicitly, as in humans or in more sophisticated agents containing an explicit representation of goals and ``beliefs''. With respect to the machine, it should be noted that here the design is known, whereas for the human cognitive apparatus, the architecture must be inferred and cannot be observed directly.
Instead of the word modality at the human input side, and the word media at the human output side, we have chosen for the word channel, which also allows for a more clear distinction between the abbreviations (HOC CIM COM HIC HOC ...) which can also be pronounced as:
* HOC: human output to computer
* CIM: computer input from man
* COM: computer output to man
* HIC: human input from computer
Perception
Now we will deal with perception, i.e., the process of transforming sensorial information to higher-level representations which can be used in associative processes (memory access) and cognitive processes such as reasoning. In the perceptual process, two basic components can be identified as described earlier (figure 1.1 , right part). First there is the human sensory system with its typical characteristics and constraints, providing the ``human input channel'' (HIC) processing functions. Second, there are those components of a computer system which provide a multidimensional signal for the human user: ``computer output media'' (COM).
Control and Manipulation
In this chapter, we will describe the processes of control and manipulation as performed by users of a computer system. Again, two components can be discerned (Figure 1.1 , left part). The first component we will approach consists of the ``human output channels'' (HOC). Theories on motor control will be reviewed briefly. The second component consists of the ``computer input modalities'' (CIM) that are suitable to be interfaced to the human output activity patterns. In addition, event handling architectures which are relevant for different computer input modalities will be investigated. Finally, our approaches towards bi- and multimodal control schemes will be introduced.
