:

Physio Info Tronics for Perceptualization Environments

An Anthrotronic Interface System To The Emerging Information-Communication Matrix.

Submitted by: Dave Warner MD PhD

MindTel / Institute for Interventional Informatics

Advanced research in the area of sensors and information systems continue to produce an ever emerging array of networked information and communication (info-com) technologies, which in turn produce an ever increasing volume of information which must be perceived and understood by a human operator. We propose to develop an interface to the network through which human communication of and interaction with this information is facilitated. This paradigm of interface technology is based on new theories of human-computer interaction, which are both physiologically and cognitively oriented. This optimization of human interaction with vast networked systems information assets incorporates multi-sense rendering technologies, giving sustained perceptual effects, and incorporates other natural user interface devices which measure multiple (physical and physiological) parameters simultaneously and use them as inputs. Such a biologically optimized interactive information interface method has the potential to facilitate hyper-effective communication. This increase in effectiveness will impact both human-computer and human-human communication through "enhanced expressivity” and extended perceptual dimensionality.

We propose to research, prototype and evaluate an integrative interface matrix that couples the data streams emerging from sensors and micro informatic technologies to the human nervous system in ways that increase the perceptual dimensionality and expressive capacity. We propose to develop an Anthrotronic (human scale instrumentation system) interface matrix that will allow for the exploitation of the human nervous system in ways that increase the humans ability to “grok” (To grok is to perceive and comprehend some sensory information or idea with some deep level of understanding.) and communicate the information being generated and transmitted by the vast multi domain info-com system. We will research, prototype and evaluate technologies that enable Controllability and Exploitability of the multi-channel, multifunction concurrence of dynamically interconnectable bio-couplers to the info-com system.

Exploiting Natural systems of human information processing through developing “Physiologically oriented interface systems”.

Knowledge of sensory physiology and perceptual psychophysics is being used to optimize our future interactions with the information from sensor networks and other bio infotronic systems. An understanding of the human neurophysiology allows for exploitation of predictable, adaptive capabilities. The assertion is that the information flow between external sources and direct experience is biased, restrained, constrained, limited, enhanced, and facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.

Thus we are proposing to integrate the conscious human user into this system as a computational resource.(not just a user)

By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, "a sensorial combinetric integrator". We will identify the optimal perceptual state space parameters in which information can best be rendered. That is what types of information are best rendered to each specific sense modality, a sense specific optimization of rendered information.

Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Models of information processing based on the capacity of these Neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile modalities. Then by using the natural bio-modulated energy as a signal source for input; we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multi-modal feedback and closed-loop interactions. ERGO = Controllability and Exploitability of the multichannel, multifunction concurrence of dynamically interconnectable bio-coupler based hybrids.

This research effort is concerned with developing a "reference architecture" (a formalized conceptual framework for research and technology development) for designing Physio-informatically robust interactive human computer interface systems to the info-com systems. The purpose of the reference architecture will be to provide insight into the various components of the system in the context of how they might affect the flow of information as information is passed through them. The primary focus will be to consider the flow of information between the human and the com-system in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during/caused by the intentional /volitional interaction with information between a conscious human and an info-com system.

We will assume that interface systems that support Human com-system interaction can be modeled and developed as systems where information flows between various components of the system in a specific manner. Emergent technology for coupling biologic systems to informatic systems is enhanced by knowledge/acknowledgement of physioinformatics restraints/constraints/capacities in the design and iterative refinement of the physicality/functionality of the technology. Our adaptive strategy will be to develop technology systems that are coupled to the distributed -concurrently multi functional - neuro info matrix that conveys perceptible and expressible information streams.

We propose to develop an Anthrotronic interface to informatic systems that is are either biologically/physiologically based (primarily neurologic i.e. neuro informatic) information systems and/or informatic systems which are designed to support interaction (dynamic exchange of information) with such systems. The intent of this work is to develop a systems-based, physiologically robust, technology matrix that will exploit the various technologies being developed.

The basis for this work comes from the researcher’s extensive experience in physio-informatics, gained through developing interface technologies for persons with severe disabilities, distributed medical intelligence systems, DARPA programs for instrumenting humans for controlling distributed robotics, and landmine detection systems.

Budget Estimate

We are seeking $875,000 per year for 3 years in direct costs

This will include all labor, facilities, travel and material costs.

 Statement of Work

TASK 1

A plan of implementation and demonstration will be determined in the initial period of Phase 1.

TASK 2

We propose to develop and integrate a set of advanced human-to-computer input devices

into a single interface system. This integration of data input devices into a single

system will include EEG, EMG, EOG, (bioelectric signals from brain, muscle and eye),

dynamic bend sensors, pressure sensors, audio and video digitizers, and other devices

as they are relevant. We will develop methods for data fusion to enable meaningful

correlations across various input modalities. The devices will be connected to an

external module which will route the data both to a central multitasking server

and to the rendering subsystem for immediate feedback. The server will be intelligent

enough to automatically implement a custom configuration of input-device parameters,

interface functionality, and relevant records based on the device(s) connected

and the identity of the operator(s) currently on the system.

TASK 3

We will integrate a comprehensive set of visual, aural, tactile, proprioceptive, and

somatosensory rendering devices into the Grok-Box system to give the user an

integrative experiential interaction with the complex data types. We will utilize

interface technology that renders computer information onto multiple human sensory

systems to give a sustained perceptual effect (i.e., a sensation with a context).

We propose to combine these different rendering modalities with somatotopic placement,

thus providing for spatial coding of the rendered information. We will demonstrate

how the implementation of vision, hearing, and touch technologies can allow for

simultaneous sensation of multiple independent and dynamic data sets that can be

integrated physiologically into a single perceptual state.

TASK 4

We will develop an interactive environment that combines new ways to render complex

information with the advanced human-to-computer input devices. Interactive interface

technology will be implemented such that it renders content specific information onto

multiple human sensory systems giving a sustained perceptual effect, while monitoring

human response in the form of physiometric gestures, speech, eye movements, and various other inputs. We will refine quantitative measurement of activity during purposeful tasks.

TASK 5

We will refine the interface system to match the human nervous system's ability to

transduce, transmit, and render to consciousness the necessary information to interact

intelligently with information. We will implement a data analysis subsystem designed to

enhance the ways that relevant data may then be rendered optimally to the operator's

sensory modalities. Linear and nonlinear multivariate analysis tools will be utilized

for the processing of multiple data sets in a variety of ways, including graphical analysis

(phase portraits, compressed arrays, recurrence maps, etc.) and sound editing (mixing, filtering). Automated detection of trends and correlations using fuzzy logic may be performed in the background or in a post-processing mode. The system will be designed so that the user may then be alerted by the system if it detects areas worthy of further investigation

TASK 6

We will design an experimental protocol to evaluate the Grok-Box system for

providing mission critical support. We will interact with the appropriate

agencies to ensure the specific requirements of this task to meet mission

critical objectives are met.

TASK 7

We will generate an ongoing reporting and demonstration of the developing

functionality of the system. We will provide seminars and workshops for the

purposes of disseminating and furthering this work.