This project envisages establishing a new generation of high-fidelity haptic display technologies. The newly developed systems will not only cover haptic interaction but also attempt to complement haptic information by visual and auditory input. To achieve these goals, two main threads will be followed: On one side, the consortium will explore and develop new technologies, which will be used to significantly improve haptic displays. On the other side the psychophysical basis of human haptic perception will be investigated. One goal is to exploit haptic illusions to overcome fundamental technological limitations. Four demonstrators covering typical application scenarios with a critical technological challenge will be developed: Haptic interaction with biological tissues, haptic texture rendering and recognition, the simulation of rigid objects with clearly defined, sharp edges, and multi-modal volumetric exploration systems.

2 Project objectives

It is no coincidence that in many languages, deeply involving emotions and feelings are described as “touching”. Touch appears indeed to be the most direct, less intellectually mediated sense. As such, it should be a fundamental ingredient of any system aimed at providing compelling sensations of remote presence. The perspective in which this proposal was conceived is that of understanding and enabling an experience of presence not limited to “being there”, but extended to “being in touch” with the virtual surroundings.

Multi-modal human-machine interfaces have been established as one of the most important, fundamental components of information technology of the XXI century. They already have decisive impact on all areas of our professional and private life, covering for example everyday communication, manufacturing, trade, financial services, health care, or entertainment.

Technological innovations leading to high-fidelity recording and synthesis of natural signals for stimulating the human sensory system (most notably sound and image generation devices) transformed our way of life and created huge markets for professional applications and entertainment industry products. While the corresponding display technologies are still in vigorous development, the ultimate limits of current approaches restricting man-machine information exchange to the auditory and visual channels are clearly manifested. Especially highly immersive systems, which are at the forefront of the development, suffer increasingly from the lack of devices, which can stimulate our other senses with similar fidelity as current audio and visual display technology.

Due to the importance of haptic sensation for general human perception, several devices for generating force and cutaneous feedback have already been developed. They mostly utilize classical electromagnetic actuator technology and have been successfully used in some highly specific applications areas like vibration devices in game consoles or force driven virtual instruments for simulating minimally invasive surgical procedures in VR-based training simulators. While the potential for further improvements of the underlying technology is still not completely exhausted, this approach is doomed to fail when trying to address the full spectrum of high fidelity haptic feedback. This situation, which is more and more becoming the bottleneck for the further development on the field, is due to our very limited understanding of the basic psychophysiology of the sense of touch, as well as to the lack of feasible technological solutions to create versatile haptic display devices. This becomes especially evident, if we compare the current state-of-the-art in the areas of visual and haptic information exchange between humans and machines.

Even the very limited applicability of currently available devices created substantial commercial interest, which clearly indicates the enormous worldwide market potential for high definition haptic displays. It is therefore rather surprising, that no serious concerted academic or industrial effort has been undertaken to address the underlying problems to be solved. The reason for this reservation is the very substantial risk connected with the corresponding research and development, caused by the lack of appropriate basic scientific and technological foundations as mentioned above. We have to experiment with fundamentally new approaches, which will significantly promote the state-of-the-art on the long term, however with very limited hope for return of investment on a commercially attractive time scale. Further risks are connected with the hardly predictable social acceptance of new technologies, which will strongly influence the role haptic displays will play in our future life. Only an interdisciplinary, concerted, multi-national effort supported by appropriate public funding can cope with the challenges ahead and alleviate the risk involved.

To this end, two major threads can be identified in our envisioned research. In the area of psychophysical investigations, this project aims at creating a solid scientific basis for the coming decades, allowing us to think and build a more effective and touching experience of presence. Our investigation will span from “early touch” topics – that is, how elementary tactile information (force, texture, indentation, temperature, frequency, etc.) is processed by humans to form haptic perception, in the perspective of what is known about early vision (edge and junction detection, shape from colour and shade, optic flow, contrast, etc.) – to the existence of high-level, abstract representations of forms (ideas) in supra-modal neural correlates, which are common to all senses (in particular, to touch and vision).

In the area of technological development, we strive to create the technological basis for engineers to conceive and build better systems for direct interaction with humans through touch, thus overcoming apparent limitations of existing devices.

The long term objective of the study is to develop systems that offer a totally convincing multi-modal and multidimensional sensorial immersion in a virtual or augmented environment that can be actively explored and perceived by the operator.

This will be done by both, providing possibly realistic primary stimuli for the human sensory system as well as understanding and utilizing the psychophysical mechanisms of multi-modal "haptic illusions".

These two threads are intimately connected, not only because biological sciences will inspire and enable technology improvement, but also because new devices will have to be engineered for experimentally testing specific psychophysical hypotheses. The envisioned contribution will remove the most disturbing obstacles in the way of future progress of information technology.

The general objectives will be addressed by four major lines of activity:

1. For the foundation of our research we will investigate how to generate the sensation of haptic presence and how to integrate it with vision. We will try to find meaningful and useful abstractions of tactile data, playing a similar role to that of “early vision” primitives. In particular, we will verify the conjectured role of tactile flow in the psychophysiology of touch and in the technology of haptic displays. Furthermore, we will determine what visuo/haptic cues are sufficient, and which are necessary, for which applications. In this context possible areas of investigation will be the relative importance of shear forces for shape recognition in comparison to texture recognition or the role of spatial and temporal resolution for specific fields of application. Specificity of results to tasks and to subjects will be carefully evaluated. Apart from this we will research on how humans combine different haptic cues (kinesthetic, cutaneous, thermal) with visual ones. Specifically, what are the relations in the human brain between cortical areas involved in specific functions (e.g., object recognition) for visual and tactile inputs? Are there abstract, sensory-independent representations of object forms in our brain, with precise neural correlates shown by for example functional MRI imaging? In this course of research we will adopt a wide spectrum of investigational approaches, such as psychophysical test, neuro-imaging (fMRI), or behavioural studies.

2. Mechanical mimicking in artificial devices alone will hardly be able to provide fully satisfactory solutions for high-fidelity haptic feedback, due to the inherent complexity of the necessary devices and stringent physiological and technological constraints. Even if future breakthroughs may overcome some of these limitations, the resulting solutions may not be economically feasible in the foreseeable future. We will therefore investigate "perceptual tricks" related to the sense of touch in order to make haptic displays more efficient. We will address the fundamental question, how can we generate an illusion of a rich haptic stimulus without getting all the physical parameters completely right. To find answers to that question we will go along two lines: One line of research will explore "illusions" within the haptic modality. The other line will investigate the interactions between the sense of touch with other sense modalities, such as vision and audition.

3. The current limits of technology have to be significantly extended in order to be able to generate direct kinesthetic feedback with highest possible fidelity. Also, new ways for generating force feedback will be explored. This activity will be based on a broad study of possible physical principles for force generation. The potential of the emerging possibilities for the realization of practically and commercially usable haptic feedback systems will be evaluated. Special emphasis will be laid on the most challenging problem of free-hand interaction with virtual objects. A candidate technology using magnetorheological fluids in a controlled 3D volume with magnetic field profiling will be more closely investigated striving for a first prototypical implementation. Additionally, basic studies will be carried out to make clear which human perception mechanisms have to be stimulated and which approaches are suitable to produce high quality tactile and proprioceptive feedback

4. To demonstrate how and to which degree the current shortcomings in haptic displays can be alleviated, demonstrators will be developed, addressing four representative areas for haptic feedback, namely free hand soft tissue interaction, haptic texture generation, rendering of rigid objects with high stiffness and sharp edges, and exploration of volumetric data in an interactive multi-modal environment. All these interaction types pose different benchmark problems that have not yet been solved to a satisfying degree. These systems will allow to integrate and test the emerging findings and solutions in the context of practically highly relevant application areas and should contribute to the identification of potential difficulties and pitfalls in the social acceptance of the envisioned technology.

3 Participant list

List of Participants

Partic. Role*	Partic. no.	Participant name	Participant short name	Country	Date enter project**	Date exit project**
C	1	Technische Universitaet Berlin	TUB	D	Start of project	End of project
P	2	Swiss Federal Institute of Technology, Zurich	ETH	CH	Start of project	End of project
P	3	Max Planck Institute for Biological Cybernetics	MPI-T	D	Start of project	End of project
P	4	University of Pisa	UNIPI	I	Start of project	End of project
P	5	Universite d’Evry Val-d’Essonne; Laboratoire Systemes Complexes (Fre-CNRS)	LSC	F	Start of project	End of project
P	6	University of Birmingham	UBIRM	GB	Start of project	End of project

4 Contribution to programme/key action objectives

As of today typical Human-Computer-Interfaces (HCIs) consist of a graphical display, audio-speakers, and a computer mouse. The significance of an adequate and user-friendly HCI becomes clear if one considers that in the industrial nations more than 40% of the households are already equipped with personal computers (PCs). Today's human-computer interaction mainly relies on the human visual sense. Speech in- and output exists satisfactorily only for very special and limited applications; haptic interaction is even more limited. This situation is in stark contrast to human real world interactions. That is, the human sensory and perceptual abilities are far from being optimally used by today's HCIs.

A substantial part of this proposal is high-risk and therefore longer term research, in particular the investigations towards new force and tactile feedback actuation principles, the psychophysical analysis and evaluation methods to be developed are widely open research questions. The planned research will push beyond mere multi-media towards high-definition haptic and multimodal human system-interaction. The TOUCH-HAPSYS proposal will therefore foremost have implications on the future and emerging technologies open domain (VI.1.1). An extended basic knowledge about the visuo-haptic information processing in the human brain enhances the understanding of the immersive sensation of “being in touch” which itself supports the FET-Presence domain (KA VI. 2.2).

By exploiting non-mechanical physical principles, such as magnetic and electrical force fields, and by investigating new mechanically active materials, such as polymeric gels, artificial muscles, and biomolecular actuators for the design of new multimodal HCIs a breakthrough for interactive haptic technologies is expected. These high risk approaches will be supplemented by the development of middle risk interfaces, such as redundant force-feedback devices with more than 6 degrees-of-freedom (DOF). One critical point for the success of this proposal is the innovative concept which will be based on an integration of technology with the intensive use of the properties of the human perceptual system. With the unique and complementary consortium consisting of engineers, cognitive scientists, physicists, and biologists, TOUCH-HAPSYS integrates a wide range of disciplines. The technological enhancements provided by TOUCH-HAPSYS are most relevant for the 2002 IST program and involve several fields of applications foreseen herein. The most relevant outcomes from TOUCH-HAPSYS perfectly match the goals of KA IV: Essential Technologies and Infrastructures, esp. KA IV.7.2: Integration of sensors and actuators for increased intelligence, interaction and networking. The focus of TOUCH-HAPSYS is on the simulation of complex, real-time, interactive multidimensional haptic environments which can be embedded in a rich virtual environment, KA IV.4.1, KA IV.6. One scenario is to distribute such multipurpose haptic devices and connect them over a network, such that a workgroup with members distributed around the world can physically work together with their hands on the same object, KA IV.2.3. This will be beneficial, e.g. in rapid virtual prototyping and various medical applications.

In view of the medical sector TOUCH-HAPSYS can easily be adapted to aid handicapped individuals, especially the blind (KA I.2.1). High-definition haptic systems will, for example, also be used in the future by health professionals (KA I.1.2) for tele-surgery (KA IV.2.3), virtual freehand surgery training, and for general medical education. TOUCH-HAPSYS also contributes to KA III (Multimedia Content and Tools). This will have a strong impact on innovative forms and formats of digital content for virtual and mixed realities. (KA III 5.2). It will provide radically new concepts for human-machine interaction combining the sense of touch with the other sensory modalities (KA IV.6), such as vision and audition (KA III.5.2). The integration between tactile perception and visual capture systems is based on sophisticated equipment where the software architecture (KA IV.3.1) will play an important role. The development of haptic simulators is akin to problems encountered in modern control theory and robotics, and this theoretical and experimental approach will have a strong impact on simulation and visualisation (KA IV 4.1) when developing real time, large-scale simulation and visualisation technologies, tools, applications and environments. TOUCH-HAPSYS envisions a revolution of the functionality, usability, and acceptability of future haptic displays. It will have a deep impact on multimedia content presentation for the needs of businesses and public organisations. This technology offers a management and exchange of digital content, together with higher levels of interactivity.

The TOUCH-HAPSYS project clearly shows also strong links to the 6th framework program of the EC. It supports the effort to provide easy-to-use natural and multimodal human-machine interfaces for everyday use to a wider group of EC-citizens, regardless of his or her educational background, mother tongue, and age. TOUCH-HAPSYS addresses societal challenges, it will radically change the work and learning environment, and it will revolutionise the way humans interact with digital information. Within TOUCH-HAPSYS we develop new technology, tailored to human perception, which brings the sense of touch to any standard home computer and which should be easy to use by everyone in the future, just like a regular PC today - this will be an essential step in carrying out the vision of 'ambient intelligence'.

5 Innovation

In 1965 the Computer Graphics pioneer Ivan Sutherland envisioned the building of an “ultimate display”, a multi-modal synthetic environment, which included force and touch feedback [Sutherland65]. He was one of the first researchers to realize the significance of the sense of touch, especially for the enhancement of virtual worlds. As the philosopher Bertrand Russell put it in [Russell69]:

“… it is touch that gives our sense of ‘reality’...Not only our geometry and physics, but our whole conception of what exists outside us, is based upon the sense of touch.”

Nowadays, the importance of the sense of touch for human-computer interaction has finally been realised and it is widely believed that haptic interfaces will be a major improvement for human-system interaction. The rapid increase of research activities in this field during the past decade supports this expectation. Especially highly immersive environments, which are at the forefront of the development, could benefit from devices, which are able to stimulate the haptic sense with the same fidelity as currently available audio and visual display technology. Examples of such innovative immersive setups that would greatly impact today’s interaction paradigms are communication and remote collaboration systems, product planning and development applications, web-based commerce and applications for training and education (including web-based learning). The key factor is the realistic simulation of interacting with the VE, so there is a sense of being able to touch objects and to manipulate them. It is clear, that only if we enable people to interact with VEs, will we achieve a convincing sense of presence in these virtual worlds, that is a sensation of “being in touch”.

Nevertheless, although the need for high-definition haptic systems has clearly been identified, up to now no haptic system has been created that is used on a day-to-day basis. This is due to the fact that haptic technology is still in its infancy and most of the available devices are just mere expensive toys. The reason for this lies in limited device characteristics and the non-observance of the psychophysical background of haptic-based human-system interaction. While the potential for further improvements of the technology underlying current devices (mainly classical electromagnetic actuation) is still not completely exhausted, this solution approach is doomed to fail when trying to address the full spectrum of high-fidelity haptic feedback. Especially, the necessary combination of kinesthetic and tactile feedback has not yet been addressed thoroughly in research. These shortcomings emphasize the strong need for highly innovative and high-risk research projects that aim at bridging the existing gap in technology as well as in psychophysical understanding.

For these reasons, we focus our research on the one hand on multidisciplinary investigations of the basis of haptic and visual-haptic presence with the major aim on the integration of sensory information over multiple haptic channels and on the combination of visual and haptic cues in support of perception and action. On the other hand we focus on the design and development of novel high-fidelity haptic interfaces and haptic interaction paradigms, which can be used to create practical systems that may eventually find their way into everyday life. In order to avoid past mistakes we take special care to integrate technological development and psychophysical research at a very early stage. We believe that the visionary goal of building convinving haptic interfaces can only be achieved by combining two major elements: mimicking real sensations by high-fidelity haptic devices and exploiting the psychophysical foundations.

According to our defined work packages and milestones, six major areas can be identified where we expect innovative improvements:

· Better understanding of the sense of touch

· Combination of haptic and visual information

· Next generation haptic hardware

· Psychophysical background of “haptic illusions”

· Multi-modal integrated interfaces

· High-fidelity haptic benchmark demonstrators

The first area of innovation will be an improved understanding of the sense of touch. For several reasons this sensory channel has been subject to substantially less extensive investigation than vision and therefore lags behind in the state of science. These include the greater diversity of physical mechanisms involved in providing haptic information, the more intimate integration of perception and action, and the lack of man-machine interfaces for the transmission of touch information. Recently, new invasive and non-invasive methods (such as single and multiunit recording, fMRI, EEG, & TMS) were developed which allow the investigation of somatosensory neural processing of tactile and proprioceptive input arising in passive and active touch (passive touch: experimenter imposed stimulus; active touch: self-directed stimulation from exploratory movement). These methods provide us with important tools for advancing the state of our understanding of haptic perception. Furthermore, we will investigate whether there is a concept analogous to optical flow, which encodes important information for softness discrimination, has a connection with the psychophysics of human touch, and is amenable to implementation in VR/Haptic displays and/or prosthetics. The conjecture is that a concept of “tactile flow” can be defined as the flow associated with the displacement of iso-stress curves on the surface of the contacting fingertips, as the resultant contact force is varied.

The second line of innovations concerns the combination of haptic and visual information to elicit the sense of presence. The scientific study of the perceptual quality evoked by sensory stimulation, psychophysics, traditionally focuses on elemental stimuli within a single sense modality. Generally, however, the sense of presence is acknowledged to depend on simultaneous information from different sensory modalities. The main goal here is to answer the question how the sense of immersion and presence in VEs is improved by including touch. Information derived from vision and touch can either be redundant, such as information about size, position, or direction; or the information from the two channels complements each other, such as information about colour, temperature, or weight of an object. Here, we will be concerned with combinations of haptic and visual information relating to both, different, or identical physical attributes of the environment.

The third innovation group identified in our workpackages is hardware related. On the one hand we seek to build novel devices that will enhance force feedback generation and on the other new systems will be developed that are going to be used for psychophysical studies on “haptic illusions”. An important component of the first line of research is the study of new principles for force generation, e.g. non-mechanical physical principles such as magnetic and electrical force fields, or new mechanically active materials, such as polymeric gels, artificial muscles, or biomolecular actuators. Here we expect major innovations that might encourage and enable other groups to conceive and develop new highly innovative systems for direct interaction with humans through touch in the coming decades. In the framework of TOUCH-HAPSYS we will right from the start explore some of these unknown areas by investigating the potential of magnetorheological material actuators, which might lead to innovative new ways for force generation. The second thread comprises the development of novel tactile interfaces that support the necessary “haptic illusion” studies and will later be combined with the previously mentioned high fidelity force feedback displays. Two exemplary devices are especially worth to mention in this category, a shear force actuator and a vibrotactile multi-bandwidth, multi-layer actuator. The combination of tactile feedback with kinesthetic feedback according to psychophysical guidelines has not yet been investigated before and holds the potential of achieving the desired high-fidelity haptic feedback stimulating several channels of our haptic sense.

The next major advancement in current haptic research brought about by TOUCH-HAPSYS will be the investigation of the psychophysical principles of haptic illusions. The main focus here is to weaken the mentioned technological limitations by “cheating” the perceptual system. Perceptual illusions already helped in the design of advanced visual and auditory displays. Therefore, we will investigate the application of perceptual illusions introduced by interactions between different haptic cues and different sensory modalities to advance the designs of new high-fidelity haptic systems. Furthermore, we strive to come up with a mathematical framework as the basis for systematic design guidelines for future haptic (illusion) displays. The ability to generate an illusion of a rich haptic stimulus without getting all the physical parameters completely right, will be a fundamental new innovation in the field of haptics, opening completely new ways of human-system interaction.

The substantial improvements we expect from the results mentioned above enable us to follow a third path towards innovations in human-system interaction. TOUCH-HAPSYS will provide radically new concepts in this area combining the sense of touch with the other sensory modalities, such as vision and audition. In order to study the interdependency of haptic illusion approaches with technology stimulating other sensory channels an advanced multidimensional and multi-modal display will be built. We envision a revolution of the functionality, usability, and acceptability of future haptic displays.

The final part that holds great potential for innovation are the benchmark demonstrators based on high-fidelity haptics. Although the prototype systems serve as platforms to apply and evaluate the new designed hardware and interaction paradigms, they still represent a major leap in immersive haptic systems themselves because of their posing critical benchmarking scenarios. We address four representative areas for enhanced haptic feedback, namely free hand soft tissue interaction, haptic texture generation, rendering of rigid objects with high stiffness and sharp edges, and multi-modal volumetric exploration systems. All these interaction types pose different problems that have not yet been solved to a satisfying degree. These systems will allow to integrate and test the emerging findings and solutions in the context of practically highly relevant application areas and should contribute to the identification of potential difficulties and pitfalls in the social acceptance of the envisioned technology. It has to be noted that all four demonstrators can function as a basis for highly innovative applications, for instance computer aided medical education. While a large number of systems are already available for teaching anatomy using virtual human models, their usage is still strongly limited by the lack of adequately realistic interfaces for the handling and manipulation of virtual anatomy. The envisioned development could fundamentally change this disappointing situation allowing the replacement of real cadavers by virtual corpses in everyday anatomical education.

FOR A LIST OF REFERENCES, PLEASE SEE THE APPENDIX.

6 Community added value and contribution to EU policies

Key objectives of the IST WP4002 program are “Services to the Citizens”, “Multimedia Contents and Tools”, and “Essential Technologies”. TOUCH-HAPSYS will contribute to all these European goals by:

· A deep understanding of the information processing of the multidimensional sense of touch.

· Understanding the interaction of touch and vision.

· Giving new insights into the cognitive body schema.

· The improvement of man-machine interfaces.

· Inventing new methods and procedures in hardware- and software engineering using basic science.

· Opening the digital world to additional user groups.

· Creating chances in the medical and the educational domain.

· Supporting the next generation of eWork and eCommerce.

A predominant recent development in professional and daily life is the rapidly increasing use of computers, mobile computing/communication, and multimedia technologies. This project will go one step further by exploring and enabling high-definition haptic systems for human interaction, which in addition to multimedia (audio-visual interaction) will pave the way for new multimodal human-computer interaction paradigms. One aim of the proposal is to push the limits of haptic system technology and psychophysical understanding of multimodal human-machine interaction. In particular haptic (force, touch, temperature) interaction allows for totally new ways of human-computer and human-machine communication. Touch is a very intuitive, partly subconscious, and multidimensional sense that in contrast to lingual, text, or multimedia controlled interfaces avoids a lot of intercultural problems and therefore can be used by a wide community of people with different cultural, educational, and employment backgrounds from the inexperienced child to the elderly impaired person. At a medium-term perspective, haptic and integrated visuo-haptic systems will have a great impact on future digital services of general interest and completely revolutionize education, tele-markets, and e-markets; one example application is web-based fabric commerce. The long-term effect will imply a better acceptance of technological devices in larger groups of the population. Thus a positive effect on the market potential of haptic applications can be expected.

An important new added value by this project is to provide understanding of multimodal, particularly haptic and visuo-haptic, human interfaces from a psychophysical and mathematical viewpoint. High-risk, new, physical and psychophysical principles are to be employed to bring about a significant advance of haptic actuation systems. By selective cue enhancement an optimal usability for each special application could be managed in future.

Based on this intellectual value we will significantly advance the state of the art in haptic systems by also exploiting such knowledge for haptic illusion and system integration. Virtual reality simulators for minimally invasive surgery training have attracted much attention in the recent years, and an almost mature technology has been brought forward (mostly in the US, see e.g. [Phantom], [Immersion]) that will help to make minimally invasive surgery (MIS) commonplace in the next few years. On the other hand, challenging applications, such as training for free-hand surgery, or interactive segmentation of radiologic data that would enable medical treatment by expert teams that are working at different locations have not been tackled yet, an area in which Europe could take the lead. However, the needed actuator technology is more complex than the one available today. The objective of this proposal is to improve this situation.

Furthermore, we would like to stress the leading role that Europe would take in the widely unexplored area of psychophysical analysis and modeling of haptic (force, touch, and temperature) and multimodal (touch, vision, and audition) human-machine interaction. In a medium-term perspective the value added by fundamental knowledge and technological results during this project will establish a technological advantage, stimulate further research, development, and business chances in the area of haptic and multimodal systems.

Our results will broaden the field of application of haptic and multimodal displays and enhance the chance that such devices can be integrated into everyday goods and on this way will step into the mass consumer-market. The technique of haptic illusion by causing a desired haptic perception deliberately with physically different stimulation principles will be one of the keys towards a broad use of haptic systems.

A set of exemplary innovative application driven demonstrators in crucial areas of soft tissue interaction for medical applications, haptic texture/pattern generation e.g. in web-based commerce, and rigid (stiff) object interaction for digital mockups will serve for the evaluation for the new scientific results obtained by the consortium.

The current project will provide a key technology of significant potential to the IST program. Some parts may be worth protecting by patents because of their potentially high commercial impact.

The budgets allocated to R&D in the field of haptic systems by the US government and by commercial companies exceed the level of funding that a single member state in the EU can afford to spend on the subject. Moreover, the problems that must be solved in order to create truly intuitive interfaces supporting a truly ambient intelligence are so fundamental that they exceed state borders and national funding schemes. The problems are so huge and so complex that no single research group could ever hope to solve them on its own. By taking the problems to the European level the financial and human resources can be brought together that are needed to stay at the frontier of the developments.

In the past, psychophysicists and engineers have thrived in almost completely independent cultures. Today, it is acknowledged by an increasing number of leading scientists that in the field of haptics neither the basic researchers nor the system designers can hope for a real breakthrough, unless completely new research paradigms have been developed. Mutual fertilisation between the different communities seems to offer enormous opportunities. In many respects the involved subject areas are now addressing the same research topics, for example the intuitive, easy to use haptic interface for everyday applications for everybody.

TOUCH-HAPSYS combines a consortium of indeed diverse disciplines unique in Europe that are willing to put together their complementing expertise in a project that exploits new physical and cognitive principles of haptic interaction. Prototype devices will be evaluated and refined by the usage of cognitive science methods. On the other side, new devices will open up the chance of new insights into cognitive processes. The new interdisciplinary approach to the problem has already been recognised by members of the consortium and other European researches leading to establishing the EuroHaptics conference that started in 2000 and annually brings together a similar mixture of experts at European and International level. TOUCH-HAPSYS will present latest results on the EuroHaptics. The idea of TOUCH-HAPSYS is to serve as a first platform for further interdisciplinary projects. Therefore, the workshops, especially the final workshop, will be opened for a wider community.

EU level funding of that platform will be enormously important to make the collaboration visible in the so far separated scientific communities, and to help the partners to produce the results that can convince other scientists of the value of the collaboration.

FOR A LIST OF REFERENCES, PLEASE SEE THE APPENDIX.

7 Contribution to Community social objectives

By combining the search for new physical principles for force generation with the psychophysical exploration of the human cognitive perception and action potential TOUCH-HAPSYS focuses on the development of fundamentally new solutions in haptic human-machine interaction. The resulting intuitive haptic devices will have a deep influence on improving the quality of life in a modern information society. They will significantly enhance the impact and application potential of existing human-machine interfaces. Human centered guidelines will allow to develop more intuitive and easy-to-use immersive virtual and augmented reality systems. Consequently, they will not only improve the design and the safety of many working environments (workplaces in production lines or in vehicles, medical treatment, etc.) but will also improve convenience in everyday life, e.g. easier to use interfaces in homes, in cars, or in automatic controlled public institutions. By understanding the dynamics of the multidimensional sense of touch and also the dynamics of the visuo-haptic integration a mismatch of sensorial cues can be avoided in future applications. Intuitive haptic interfaces will provide simple and comfortable, feel-at-ease access for every user, regardless of his or her educational background and age. The long-term effect will imply a better acceptance of technological devices in larger groups of the population.

The European population is growing older and the percentage of the retired and/or disabled people is raising every year. Former studies have shown that elderly persons have a really different body schema and with growing age get more and more trouble not only with their sight but also with high precision body movements that uses grip and touch. Companies selling white goods already recognized these problems (recognized this growing market) and are beginning to design production runs that take into account the sensorial needs of elderly people. The results of TOUCH-HAPSYS will provide guidelines towards efficient design of immersive visual-haptic devices/environments.

Web-based commerce attracts rapidly growing interest in the population. It is, however, at the moment restricted by the mainly audio-visually controlled PC-interfaces. To illustrate the point, imagine a virtual fashion store. Having checked the size, the fit and the look of a garment, the customer would now like to sample the quality of the virtual velvet material between his/her fingers before making a purchase decision. TOUCH-HAPSYS will contribute at this point by studying the generation and psychophysical validation of haptic texture/pattern perception.

The next generation of high definition haptic devices will permit a real breakthrough in the field of medical education, training, and diagnosis. The use of current force feedback devices is limited to training of minimally invasive surgical procedures and cannot cope with more generic situations (like in open surgery). An improvement is urgently needed. New devices and techniques invented by TOUCH-HAPSYS will be used to improve the training of medical students especially in basic anatomical education and skill training. Additionally, tele-palpation using intuitive haptic displays is a qualitative step forward for the remote monitoring of the state of health.

Ambient intelligence and augmented reality are visionary keywords for future research funding programs. However, most projects in this field are studying only the integration of vision and hearing. Touch is by far not studied neither integrated to the extent, which would even approximately correspond to its role in general human perception. Higher levels of interactivity are a key feature of an ambient intelligence landscape and will enable more natural and effective use of an emerging universal information structure. In practice it will enhance the attractiveness of web-based open and distance learning (WBODL) tools. Cognitive, validated instruments developed by TOUCH-HAPSYS will also allow augmented reality systems to go beyond visual augmentation and address touch related problems in newly developed systems.

We also envision significant contributions at the service of computational neuroscience, cognitive science, computer science, and other areas by the planned balanced experimental and theoretical research for designing novel sensors and displays.Finally, TOUCH-HAPSYS will create devices which will permit completely new sensations and procedures of haptic interaction in schooling, entertainment, and user agent technologies (e.g., videogames, interactive working groups) and introduce a real paradigm shift in the way of education in general.

8 Economic development and S&T prospects

The research results and the technological advancements which result from TOUCH-HAPSYS will be utilised in three ways: 1) to improve Human-Computer interaction, 2) as a tool to accumulate knowledge in the scientific community, and 3) as a fundament for economic and commercial applications.

1) Human-computer interaction:

The acquired knowledge and the developed technology within the TOUCH-HAPSYS project will be for establishing design principles as well as for building high-definition haptic displays to substantially improve Human-Computer interaction (HCI). Foremost, not only we plan to build high-definition integrated haptic systems, but we seek to establish a novel approach for improving HCI by referring to the neuronal and psychological mechanisms of active and passive touch. This consists of a deep understanding of the human haptic information processing (including visuo-haptic integration) which will guide the technology (and not vice versa). We will establish a constant knowledge transfer from the consortium to the scientific community and to the industry by regular meetings, publishing in scientific journals, and a public workshop at the end of the TOUCH-HAPSYS project. We will demonstrate the TOUCH-HAPSYS approach to the public using four challenging testbed benchmark applications. This should initiate a completely new technological area, which has the potential to radically change the way we interact with the computer and with digital information in general.

The hardware modules, as well as the general models of human interaction behaviour and the guidelines for the design of high-definition haptic systems to be developed in TOUCH-HAPSYS will be made available to companies for commercial use. The consortium will promote its results first of all by means of the four demonstrator systems, that will be build in such a way that some of them can be shown in exhibitions and at conferences. The consortium will actively seek media coverage of the project itself and of the demonstrators, especially through TV and the popular press.

The human factors guidelines resulting from the basic research and the hardware development in TOUCH-HAPSYS will be presented in relevant commercial conferences and to industry panels.

2) Knowledge accumulation in the scientific community:

The EU has positively considered research in haptics in recent years under Framework IV and V. Results of all these existing projects will be carefully reviewed in TOUCH-HAPSYS, and will constitute the basis for the work of this project.

The outcome of the TOUCH-HAPSYS project, that is the high-fidelity haptic system, will find excellent applications within the scientific community itself. To illustrate this, consider the dramatic advance in knowledge on visual and auditory information processing during the last century. For the sense of touch, a similar progress could not yet be observed. This misbalance is attributed to the advanced display technology for visual and auditory stimuli. As of today, there is no adequate display technology available which addresses the sense of touch. TOUCH-HAPSYS seeks to change this situation fundamentally, by making every effort to promote their systems in the scientific community. This hopefully triggers a step in knowledge accumulation in the domain of touch, which at the end might be the most valuable contribution of TOUCH-HAPSYS to the scientific community.

In addition to regular publications in scientific journals and to scientific conferences, TOUCH-HAPSYS will organise a scientific symposium at a major EU conference to disseminate its results (more specifications in the Dissemination and Use Plan, D1.4). This symposium will also be used to provide the basis and to define the objectives for a long-term R&D enterprise under the next Framework Programme. Within the TOUCH-HAPSYS project a free available website will be established, to bring current results back into the scientific community, including the presentation of state of the art, of results, publications and data.

The most important long-term use of TOUCH-HAPSYS is its contribution to the organisation and structuring of the European Research Area.

3) Economic and commercial application:

The potential of high-definition haptic display systems and the scientific results of TOUCH-HAPSYS are expected to be of immediate interest to a wide range of companies in various industries. The current proposal is primarily aimed at the fundamental problems that must be solved to allow the development of high-definition haptic displays. It is, however, expected that both psychophysical and the technological approach developed within TOUCH-HAPSYS together with the four benchmarking applications, which have demonstration character within TOUCH-HAPSYS, will be of significant interest to a range of different companies throughout the European community. It can thus be expected that the methods mentioned below (as examples) will be of short-term interest for commercialization.

Short-term use:

The consortium intends to create an informal accompanying board from industrial partners, in order to bind potential users already in an early stage, and to provide opportunity to influence the development and get first handle on later commercial exploitation. The following methods will be of short term interest for commercialization:

- Psychophysically motivated human factors guidelines for the construction of high-fidelity haptic and visuo-haptic displays systems, WP 2 and WP 3..

- Novel actuator technologies as described in WP 4 and WP 6.

- New multimodal, multidimensional high-definition haptic display systems.

- Demonstrator technology developed in WP 7.

Long-term use:

The most important long-term commercial utilization of the results from TOUCH-HAPSYS are in the vision to integrate haptics into an intelligent, multi-modal, high-definition human-machine interface. This – still imaginary – easy-to-use interface, tailored to human perception, should bring us the sense of touch to our home computer. This will be an essential step in carrying out the vision of ‘ambient intelligence’.

In summary, the foundations for a revolutionised way of interaction between humans and technical systems in the future information society will be initiated by this project and will trigger a span of research, development, and business activities in a timeframe of about 2008-2012.

Dissemination:

According to its fundamental research nature, the dissemination policy of this project is totally open (all deliverables are of public nature), and indeed encouraged. All the accumulated knowledge produced in this proposal will be brought into the scientific community by regularly publishing in high-impact scientific journals, and by regular presentations at prime international conferences. Early in the course of the project TOUCH-HAPSYS provides a plain presentation that is suitable to attract attention across the boarders of the scientific community (D1.3). Additionally a freely available portion of the project specific website will bring current results back into the community, including the presentation of state of the art, of results, publications and data. The management board will most likely decide that parts of the developed software can be made freely available as OpenSource. Knowledge, experiences and new (industrial) contacts that are accumulated during the project will enable the consortium to map a deliberate Technology Implementation Plan (D1.10). A final workshop at the end of the project will be organised in association with a major EU conference to present the results to the scientific community. The workshop will be open especially for young scientists, users, and industry. This workshop should lay the basis to define concrete objectives for future R&D activities under the 6^th Framework Programme.

9 Workplan

9.1 General description

Each workpackage (WP) focuses on one aspect of the overall goal to build a new generation of high-fidelity haptic display systems. WP 1 is to manage, coordinate the consortium, and to organise regular meetings and workshops as well as PRESENCE networking activities. The goal of WP 2 and 3 is on the understanding the fundamentals of the touch- and visuo-haptic related aspects of Presence. WP 2 will study the physics of biological contact phenomena and use psychophysical and neurophysiological methods to study early cortical processing of haptic perception. We will determine which task-specific and subject-specific cues are sufficient, and which are necessary to generate a sensation of being present and in touch. Whereas WP 2 focuses on pure haptic perception WP 3 is directed towards the exploration of multimodal perception. We will investigate the interactions of touch and vision in virtual environments aiming at optimising the feeling of immersion and presence. We will concentrate on object representation and recognition, attention, and the integration of information from vision and touch. In WP 4 three lines of action can be identified. At the first part of this WP possibilities of conceptually new devices by applying new ways for force generation are explored. The goal is to generate a totally new device which allows freehand interactions between an operator and the virtual environment. The second line of action aims at the improvement and enhancement of available technology in order to build new advanced and high-definition interactive haptic systems. At the last part of WP 4 fMRI compatible sensors and actuators are designed and realised in order to permit an experimental validation of the theories developed in WP 2 and 3. WP 5 is born out of the realisation that even the most advanced technology itself alone cannot succeed without taking the cognitive, behavioural, and sensory abilities of the human person interacting with the advanced haptic system into account. Therefore, in WP 5 we will, in line with the fundamental research in WP 2 and 3, explore the possibilities to use haptic illusions in order to compensate for technological limitations. In WP 6 this knowledge will be applied to the design and development of new systems. The resulting integrated systems will then be evaluated in close collaboration of the consortium partners. Finally, WP 7 will seek to build integrative testbed, benchmark demonstrators for crucial application areas using the knowledge and experience acquired during this project. The four demonstrators are related to typical key applications each facing a different technological challenge. The first demonstrator will simulate the haptic interaction with biological tissues, a problem which is prominent if medical training is to be improved by the use of virtual environments. The second demonstrator will haptically render textiles and the recognition performance of human operators in haptic texture/pattern interaction will be explored. A typical application of this would be the remote presentation of sales goods like e.g. fabrics via the internet. The third demonstrator will tackle the problem of creating rigid and stiff objects with clearly defined, sharp edges. This is a well known difficult problem for any virtual simulation which contains hard objects, as is, for example, needed in virtual prototyping. The fourth demonstrator, a multi-modal volumetric exploration system, aims at enabling the operator to explore of complex 3-dimensional volumetric data intuitively in order to solve complex tasks as for instance the segmentation of 3-dimensional radiological images or the interpretation of activation patterns resulting from fMRI experiments.

Taken together, all these application domains will provide a critical test for the scientific and technological knowledge acquired in this project. This project will be the fundament of a new generation of high-definition haptic displays which will widely be used in the future by everyone.

The work to be done in each package is analysed in detail in the following.

Workpackage 1: Management

The objective of this WP is to provide administrative, organisational and technical management and facilitate effective coordination between the partners. It includes the overall financial, scientific and technological management; the organisation of the tasks in the workpackages; the coordination of the work between the workpackages; organization of PRESENCE networking activities; and the continuous information exchange with the CEC.

In the initiation phase, the project management structure will be defined by appointing local scientific and workpackage managers, and installing the management board. The project will start with an initial workshop meeting. Another crucial workshop coinciding with MS 1 and 2 is to decide which of the high-risk parts of the project are likely to succeed and to make decisions for subsequent project tasks to aim at demonstration of achieved results towards high-definition haptic displays. Furthermore, hypotheses on pure haptic and multimodal perception are formulated and the experiment design for their assessment is specified involving a decision for fMRI compatible actuator technology to be constructed in following tasks. MS 4 and 5 along with another workshop are again crucial decision points to decide about haptic system integration for the three planned demonstrators and another high-risk actuator capability evaluation. These workshops will serve as the basis for our risk management strategy. Progress on high-risk areas will be critically evaluated by the project partners and invited external experts will be asked to identify weak points in the originally planned development. On the basis of these interim evaluations the consortium will reconsider the appropriateness of the planned approach. Decisions about fall-back strategies will be made according to the experience collected. The breadth of the scientific and technological fundament of the proposed work will ensure, that alternatives can be found for approaches which do not come up to our current expectations. Resource allocation and the project schedule will be accordingly adjusted.

Regular meetings and reports will be organized every six months. The management WP will install a BSCW server for the preparation of project documentation, reports, dissemination of information and deliverables. In addition to a regular meeting, informal collaboration, and information exchange between the partners is planned. Some of the more formal workshops will be organised in conjunction with International Symposia on topics related to TOUCH-HAPSYS.

Specific coordination issues will include to set up the contractual framework including the contract with the EC and the Consortium Agreement as well as to collect summary progress reports. Upon acceptance of the review report by the EC, each partner will submit a cost statement to the Coordinator who will submit a consolidated cost statement to the EC. Project reviews will be arranged every six months, at the coordination meetings, where interim and final reports will be edited and issued. Besides a technology implementation plan that indicates potential foreground rights and exploitation intentions, including a timetable for exploitation, a final project report will be presented at the end of the project. This report will be suitable for publication and outline how the project was conducted, its objectives, and results.

The Coordinator will chair the Steering Committee meetings (call, agenda, minutes, and follow-up); ensure a coherent dissemination strategy for all results and optimise promotional activities and opportunities. He will monitor the progress assessment, coordinate contact activities of TOUCH-HAPSYS with other projects, act as the liaison to the EC monitor, and manage the interfaces between the workpackages in order to promote collaboration and cooperation within TOUCH-HAPSYS.

The management WP will undertake the coordination of the dissemination, protection, and exploitation policies of the Consortium. A dissemination strategy will be drafted covering the precise nature and timing of related activities. The strategy will include a dissemination workshop (the final project workshop will be open and invitations will be extended), articles/reports, demonstration s, and Internet presentations.

Workpackage 2: Touch basics

Haptics is the combination of cutaneous and proprioceptive information subserving touch perception. Among others, Ledermann and Klatzky and colleagues have reported extensively on haptic information processing [Ledermann]. A key finding is that some haptic dimensions are more elemental (e.g., texture) than others (e.g., geometric form). Such dimensions are handled differently, both in terms of the active exploration movements made by the fingers, or in terms of detection and identification of haptic objects the. The objectives of WP2 are to study early cortical processing of haptic perception to determine which cues are sufficicnet, and which are necessary, for specific tasks. This fundamental knowledge about the touch modality is significant for generating a realistic touch sensation using touch feedback devices, such as these used in virtual environments. Specifically, the results of this workpackage are needed in WP 3 and can be fruitfully used in WP5 in which the integration of visual and haptic information into immersive percepts is explored and measures for the presence sensation are determined. In our exploration of the “touch basics”, we will include the study of performance in elderly subjects and neurological patients as biological models for resistance of system performance to degradation under component failure.

WP2 will be concerned with sensory processes contributing to the perception of elemental tactile and proprioceptive qualities. The consortium with its strength on engineering and basic psychological research provides the opportunity to advance haptic devices with special requirements for fMRI research and clinical examination. Tactile actuators for fMRI investigations shall be developed in WP 4 in close cooperation of engineering aspects and design guidelines deri