Physio Info Tronics for Perceptualization Environments
An Anthrotronic Interface System To The Emerging Information-Communication Matrix.

Abstract


Advanced research in the area of sensors and information systems continue to produce an ever emerging array of networked information and communication (info-com) technologies. These in turn produce an ever increasing volume of information which must be perceived and understood by a human operator. We propose to develop an interface to the network through which human communication of and interaction with this information is facilitated. This paradigm of interface technology is based on new theories of human-computer interaction, which are both physiologically and cognitively oriented. This optimization of human interaction with vast networked systems information assets incorporates multi-sense rendering technologies, giving sustained perceptual effects, and incorporates other natural user interface devices which measure multiple (physical and physiological) parameters simultaneously and use them as inputs. Such a biologically optimized interactive information interface method has the potential to facilitate hyper-effective communication. This increase in effectiveness will impact both human-computer and human-human communication through extended perceptual dimensionality and "enhanced expressivity.”

Introduction
The amount and diversity of information being produced in the many fields of human knowledge grows rapidly. Whether it be academic researchers creating and publishing new information based on already existing material they have utilized in the development of new theories and methods, or the information that is generated, used and recorded during more sudden and sporadic occurrences of emergency medical response, for example, the volume and differences in information increase. Complicating this state of affairs are the integrations of information across boundaries and events made possible by emerging communications technologies. Cross pollinization and syntheses of the diverse data means that there are more and more possibilities for action when all this information is brought to bear on contexts of need.  Interacting profitably with the many sources of information using the many means of communication now available requires some new models of how humans may interact with information resources. In this text, our purpose is to articulate one such model. Our research into human-computer interactions, particularly as it developed conceptually with the needs of the disabled user, allowed for some very effective ideas about how to optimize the human’s ability to exploit the powers of information & communications systems. Anthrotronic is a term we use to emphasize the human as central in the design and use of information and communication technologies. The human ability to perceive, process, and act on information is directly related to the physiological structures and functions of the nervous system. The orientation of the model we propose is therefore towards a tayloring of information represenation to the unique capacities of the nervous sytem to take in and respond to information.


History and Problem

The early stages of this research occurred in the neurology and rehabilitation departments of a major medical school. All led to a new focus on perception and expression as the key problems in re-thinking how humans might better interact with information systems. Historically, these problems were sidelined by the emphasis on faster more powerful computation and greater storage. Human factors was, therefore, left out of the thinking and design of information systems. First we give a brief background on how these issues were formulated in the particular contexts of medical diagnostics and rehabilitation. These were merely the starting point for the interrogation of the logic of human-information interaction which we are fully engaged in at the present time.

Perception:
Electroencephalograms (EEG) and Magnetic Resonance Imaging (MRI) devices were both used to gather data. Electrophysiological output from the brain is what is thereby measured and output as squiggly lines (EEG) or crude image maps of the head (MRI). Experimentation with electrocardiograms (ECG) was also undertaken. This research led to the conclusion that the perceptual forms of the data generated by these technologies was inadequate to the task of providing precise diagnostic information to the clinician.  Also, the form of the data locked out anyone but the highly experienced clinician in making decisions over obscure significance in the data. Based on this experience, the perceptualization of data became a general problem to be explored and addressed.

Expression:
Neuro-diagnostic studies of the brain also triggered the awareness that electrophysiological output comes not just from the brain, but from the muscles of the entire body. The body is emanating certain kinds of energy as measurable patterns at all times. EEG, EKG, and other physiological monitors are simply ‘reading’ this energy as data as it streams off the body. The proximity of pediatric rehab to the neurology department occassioned the question as to whether physiological output such as this could in fact be exploited as an input source. That is, if the body is giving off certain energies in certain patterns in relationship to some mental or physical behaviors, then it is conceivable to use them to control some device like a computer. The output need only be captured in a way which allows them to be converted into input for driving a device.

Based on these insights the issues of perceptualization and expression within information systems were raised and shown to be problems in need of solution for a range of users in many different fields. Rehabilitation and clinical/emergency medicine have been the areas we have worked to penetrate with both an articulation of the problem and a generic conceptual framework for its solution. More recently we have engaged with the humanitarian and disaster response communities to show the ways in which existent practices of information gathering, representing and decision support are inadequate.  Let it be noted that the only reason we can make such claims is because technology has come into being that allows wholly other ways of interacting with information. Our work has been to research these technologies and within particular contexts to test and refine our ideas about how to use them. The generic issues we now turn to.


Anthrotronic Principles of Human-Information Interaction

Anthrotronic is human-centered and more accurately mind-centered thinking about information and communications systems design. Rather than beginning from the perspective of power and storage, these are included in the larger framework of beginning from the needs and abilities of particular users as a ‘mind in the system’ taking in and outputing information. Our assumption is that current interaction with vast and diverse information resources for varying and urgent purposes is hindered by

1) A neglect of the multiple sensory systems of the human body, and
2) Perceptually inferior preparations of information for those senses which
are used.

The visual sense has been given top priority in the area of human-information interaction. All our data is made available through the eyes, with few exceptions. There are at least two other senses which are completely cut out of the interaction with information: the ear and the skin which has seldom, if ever been exploited as a means of information gathering. Of course, the blind have benefitted greatly from it. Even the way information has been prepared for the visual sense is often inferior, as it has been presented as text, numeric characters, and crude graphics. In our research, the medical expert is always dealing either with printed textual material, or else with the opaque visual forms of data produced by medical instruments like the EEG, EKG, and such others. Text, numbers, symbols, squiggly lines, and graphs have perceptual qualities which are inadequate to time and content requirements of those using the information (eg. doctors, emergency medical personnel, etc.). The information, as typcially represented, is just perceptually defficient, to the neglect of the extraordinary capacity of our brain to capture and process information from our senses. We are not saying that vision or text are bad ways of accessing information. Rather, we are making a conceptual point that while these work well for some needs, the critical nature of some information requirements is hindered by them. That is when great quantities and diversities of information  must be accessed, represented in all their richness, and then used in a rapid manner. Our goal is to offer conceptual (and eventually technological) solutions for information demand by rethinking the sensory and perceptual possibilities for how to render information to the human body for decision support.

Solution
Our goal is to take these concepts and begin to develop a system that brings together different multi-sensory representations of information into one unified dynamic interface.  The approach to the problems raised is based in part on ideas in the study of sensory physiology.  A mind-centered orientation to human-information interaction asks first, “How does the human nervous system through the senses gather raw data and then present it as information to the mind?”  The answers to this question will help us create powerful interfaces structures and functions between minds and data.  Thus we are proposing to integrate the conscious human user into this system as a computational resource: a mind (not just a user) in the loop.  By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, "a sensorial combinetric integrator".   That is, the mind and body inside the network interface we are talking about will be a locus of percpetion and expression; a reader and a responder in any information and decision intensive process. 

 

To progress in this, we will identify the optimal perceptual parameters in which information can best be rendered for each of the three senses. That is, what types of information are best rendered to each specific sense modality, optimization of represenation based on the unique processing properties of the sense in question. Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures.  Feature extraction is defined by Kendal et al as “the selective detection and accentuation by sensory neurons of certain features of a stimulus” (1991, 338).  Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile sense perception. Then by using the natural electrophysiological energy as a signal source for input (referred to above), we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multi-modal feedback and closed-loop interactions and, therefore, controllability and exploitability of the multichannel, multifunction concurrence of dynamically interconnectable bio-coupler based hybrids (dave. Please translate from after the therefore in this last sentence).

 

We will endeavor to address and solve the deficiencies in information representation and decision support through both the perceptualization of information and the enhancment of expressivity made possible in a single interactive interface which can be deployed in any place equipped to benefit from it.  Information will be rendered for three senses rather than one. The eyes, ears, and skin will all become aveneus for gathering data, with vocal an haptic signals allowing for command inputs.  Perceptualization of information is an idea which assumes that under high intensity demand for diverse and voluminous information, it is best to divide the different information up.  After processing the different data through data mining techniques, the goal is to take some of that data and put it into a form which is highly accessible to the visual sense. Likewise, take some of the data and make it accessible to hearing; and so with the tactile, or haptic surfaces of the body. Large quantities of diverse data may be transformed into multi-sensory forms of information.  Here are the basic forms of perception and expression.

 

1) geometry, color, texture, and dynamics becoming meaningful features of
information which has been 'visualized'
2) tone, pitch, timbre, volume, duration, location becoming meaningful
features of information which has been 'sonified', or converted into sound
forms.
3) touch, felt position, motion, and force becoming meaningful features of
information which has been converted into forms which come into contact with
hands, fingers, arms, or skin and muscle sensations of the body known as
tactile or haptic manifestations. Think of the braille used by the blind,
and imagine an ellaboration of this idea for accessing information without
sound or image but with physical impressions across body surfaces.
  
Data Mining & Knowledge Discovery
Data mining processing is part of the core functionality of the interface. In order to transform data into the different perceptionalizations available to the user, an powerful  means of turning repositories of data into novel and powerful information is required. "Gold mining is a process for sifting through lots of ore to find valuable nuggets. Data mining is a process for discovering patterns and trends in large datasets to find useful decision making information … There are many different definitions of data mining. Almost all of them involve using the today's increased computing power and advanced analytical techniques to discover useful relationships in large databases." "Data mining is a 'knowledge discovery process of extracting previously unknown, actionable information from very large databases'" [Aaron Zornes, The META Group] "Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techiques" [Gartner Group] quotes from www.spss.com.  It is important to note that data can come in many forms.  Some indeed is in databases and data warehouses.  However, a good deal of the data that a user of this interface will be interacting with is being generated on the fly in crisis situations.  Medical monitoring at the site of a human emergency will be streaming raw data into the grok-it interface for the user to perceive and respond to in emergency time.  Or if physicians are doing clinical work over the web there will be simultaneous processing of record data from databases as well as on the fly generation of patient data.  Counteracting the effects of bioterrorism would be another example of requiring data from both large stationary stores and that coming from a vast array of different kinds of sensors at the and near the site of emergency.

  
Objective

We propose to research, prototype and evaluate an integrative interface matrix that couples the data streams emerging from sensors, micro informatic technologies and databases to the mind via an intelligent exploitation of the nervous system towards the increasing of perceptual dimensionality and expressive capacity. This Anthrotronic (human scale instrumentation system) interface matrix will allow for the harnessing of the human nervous system in ways that increase the user’s ability to “grok” (to perceive and comprehend some sensory information or idea with a deep level of understanding) and communicate the information being generated and transmitted by the vast multi domain information-communications system. Further, we will research, prototype and evaluate technologies that enable Controllability and  Exploitability of the multi-channel, multifunction concurrence of dynamically interconnectable bio-couplers to the info-com system.  The foundation for this goal is the proposition that the information flow between external sources (representation) and direct experience (mind) is biased, restrained, constrained, limited, enhanced, and facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.

Finally, this research effort is concerned with developing a "reference architecture" (a formalized conceptual framework for research and technology development) for designing Physio-informatically robust interactive human computer interface systems to the informations-communications systems. The purpose of the reference architecture will be to provide insight into the various components of the system in the context of how they might affect the flow of information as information is passed through them. The primary focus will be to consider the flow of information between the human and the com-system in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during/caused by the intentional/volitional interaction with information between a conscious human and an info-com system.

 

Conclusion

For several years we have worked to intelligently advance the physioinformatic thesis of human-information interaction.  Not only have our ideas been well received and supported by both governmental and private institututions, but we have already developed some powerful core technologies, both software and hardware, which would be utilized in any development we would do in the next stage of this work.  The opportunity to bring into prototype a fully functioning grok-it interface system is to the benefit of everyone with needs for the kind of information interaction we are here envisioning.