Volume 1. Section I. Administrative

1. BAA #: N66001-98-X-6905

2. Technical Topic Area: Human-Computer Interaction

3. Proposal Title: Physio-Info-Tronics for Perceptualization Environments: An Anthrotronic Interface System for the Emerging Information-Communications Matrix

4. Technical Point of Contact:

Dave Warner M.D. Ph.D

(800) 950-0849

davew@mindtel.com

davew@well.com

5. Administrative Point of Contact:

Ed Lipson

6. Cost Summary of proposed research:

7. Contractor’s type of business: “Other Small Business”

8. Duns #:

9. Cage Code

Volume 1. Section II. Detailed Proposal

A. Innovative Claims

The amount and diversity of information being produced in the many fields of human knowledge is growing rapidly. Technology will increase the transfer rate of this information and globalization will empower smaller countries, groups, and individuals to generate and express data in novel areas of interest and formats. For humans wishing to sift through and make meaningful the enormous amount of relevant and irrelevant data flow from below, on, and above the planet requires a new synthesis model. In this text, our purpose is to articulate one such model.

Our research into human-computer interaction, particularly as it developed conceptually with the needs of users with disabilities, has allowed for some innovative and very effective ideas on how to optimize the human’s ability to exploit the powers of emerging information & communications systems. Anthrotronics (meaning “human instrumentation systems”) is a term we have introduced to emphasize the human as central in the design and use of information and communication technologies. The orientation of the model we propose is therefore towards a tailoring of information representation to the unique capacities of the nervous system to perceive and respond to information. http://www.medibolt.com/conbottom.htm.

Whether it be academic researchers creating and publishing new information based on existing material they have used in the development of new theories and methods, or the information that is generated, used and recorded during more sudden and sporadic occurrences of emergency medical response, the volume and differences in information will increase. Complicating this state of affairs are the integrations of information across boundaries and events made possible by hybridized communications technologies. Cross pollination and comparisons of the diverse data means that there are more and more possibilities for action when all this information is brought to bear on contexts of need. Interacting profitably with the many sources of information using the many means of communication now available requires new models of how humans may interact with information resources. Our research investigates the human ability to perceive, process, and act on information while understanding that this input and output is directly related to the physiological structures and functions of the nervous system within a given environment.

B. Technical Approach

Purpose and Tasks of this work.

Our purpose is to develop a generic perceptualization and command environment which

will “dock” on to a communications network and enable quick and effective interaction

with information. Specifically, our goal is to allow for 1) multisensory perception of data

coming from stored and emerging data sources (e.g., sensors); and 2) different modalities

of bodily action by which to issue command decisions back out into the network. The

picture is of one person occupying this perceptual and expressional environment and

working within its functionality to meet the needs of experts in differing contexts of both

urgent and non-urgent information requirements. Medical and humanitarian applications

of this generic system have been the contexts for our conceptual and technological

development of the environment and its capacities. Next, we give a list of tasks we will

aim to accomplish over the course of our research under SPAWAR

funding grants.

Task 1. We will conceptualize and develop a plan for the design, construction, and implementation of a multisensory perceptualization and expressional

command environment with generic functionality and therefore applicability across many contexts for rapid and information intensive needs.

Task 2. We will hold seminars and research symposia for the purpose of bringing those researchers and developers doing work related to the different technological areas associated with human-information interaction, representation, and decision support.

Task 3. We will design software systems to facilitate the coupling of diverse sensor technologies with those systems responsible for raw data processing.

Task 4. We will design software systems to take processed data from sensors and communicate that data to the interface engines of the perceptual/expressional environment.

Task 5. We will develop rules for how differing kinds of data are parsed across the three senses we are working with.

Task 6. We will develop perceptualization protocols for the visual, binaural, and tactile senses of the human body.

Task 7. We will plan for and incorporate incremental hardware components as necessitated by the conceptual and software development of the project as it progresses.

History and Problem

Initial research was conducted in the neurology and rehabilitation departments of Loma Linda University Medical Center. This work led to a new focus on perception and expression as the key problems in re-thinking how humans might improve interaction with information systems. Historically, these problems were sidelined by the emphasis on faster more powerful computation and greater storage. Human factors were, therefore, largely omitted from the consideration and design of information systems. Below we provide a brief background on how these issues were formulated in the particular contexts of medical diagnostics and rehabilitation. These were merely the starting points for the interrogation of the logic of human-information interaction, in which field we are now fully engaged.

Perception

Electroencephalographic (EEG) and Magnetic Resonance Imaging (MRI) devices were both used to gather clinical data from the human brain. These are the among the tools of the clinical neurologist. Electrophysiological output from the brain is what is thereby measured and output as either “squiggly lines” (EEG) or crude image maps of the head (MRI). Experimentation with electrocardiograms (EKG) was also undertaken. This research led to the conclusion that the perceptual forms of the data generated by these technologies were inadequate to the task of providing precise diagnostic information to the clinician. Further, the form of the data prevented anyone but the highly experienced clinician from making decisions concerning obscure features in the data. Based on this experience, the perceptualization of data became a general problem to be explored and addressed. Towards the goal of better perceptualization of data we will explore three separate areas of representation of data to the senses: visual, binaural, and tactile. In the area of visual representation our objective is to transform standard visual forms of information into new and more powerful representations for the eyes. A great deal of information critical to medicine, science, business, and many other fields, is stored in vast repositories. The increasing of those stores only complicates the already difficult process of accessing and understanding needed information. Text is the visual form of much of this information. Therefore, our work will be to identify already existing technologies for the transformation of textual information into visualizations. There are many such resources on the horizon. As we identify which are the best for the purposes of the Grok Box we will want to purchase, test, and ultimately integrate them into the perceptual/expressional environment we are developing.

Binaural/auditory representations of information are another tier of the project. The

concept of auditory display is the principle for this form of perceptualization. "Auditory

display research applies the ways we use sound in everyday life to the human/machine

interface and extends these uses via technology."(Gregory Kramer, An Introduction to

Auditory Display, 1994 ). This is also known as the sonification of data. Researchers at

IBM are engaging in the exploration of sonifying data sets. Researcher David Jameson

explains that the purpose of sonification is “to augment or even replace traditional, visual

ways of transmitting computerized information. In other words, using programs to

transmit information in aural ways much as graphs, numbers, or pictures transmit data

visually.” Turning data sources into sound is a powerful way of representing

information which is able to take some of the total load of needed data and making it

more quickly available through another sense modality such as:

An atmospheric selector is used to transform the spectral frequencies of natural electromagnetic impulse radiation into the human auditory range. The resulting sound pattern is determined by meteorological conditions. A change in the sound pattern of atmospheric electrical activity permits a change in the meteorological situation in the area of the measurement station intake area to be directly detected acoustically. <http://www.sti.nasa.gov/rselect/openlit.html>

Finally, there is the technology which utilizes the perceptual capacities of human skin

surfaces (tactile) and the body’s sensitivities to pressure and tension (haptics).Tactile and

haptic perceptualization is the third mode of sensing and responding which our project

will explore and develop. In its most basic form, tactile representation of data occurs in

the form of Braille, the literary medium of the blind. Haptic interfaces are also currently

being developed and employed successfully in many surgical theatres as robotic

telemedicine becomes an accepted form of medical intervention.

There is quite a diversity of methods for rendering information to the skin and limbs of

the body. Our task will be to bring together several of the most promising technologies

currently being developed. The value of this and the former two modes of

perceptualization is extraordinary. By spreading data across three senses and doing so in

ways which truly capitalize on the nervous system’s physiologic data processing from

those senses, a single person will be in position to integrate and then respond to an

unparalleled quantity of information for some critical purpose.

Ergonomics

Ergonomics is the study of body posture and orientation and is assessed using

Electromyography (EMG) sensors and a 3-D video technique termed Digital Surface

Photogrammetry. This analysis becomes a critical human factors variable during

sustained sitting, standing or in ambulatory activities. These subtle yet potentially

limiting variables will be addressed when considering such elements as fatigue and task

distraction under stressful situations. EMG measures muscle activity and when combined

with Photogrammetry and wearable bend sensors allows for an insightful skeletal-

muscular analysis and can reveal potential problems associated with input equipment

(monitors, sensors, speakers) positioning and output control devices (joy-stick, mouse,

keyboard, microphone) interactivity.

Expression

Neuro-diagnostic studies of the brain also triggered the awareness that electrophysiological output comes not just from the brain, but from the muscles of the entire body. The body is emanating certain kinds of energy as measurable patterns at all times. EEG, EKG, and other physiological monitoring instruments simply ‘read’ this energy as data, as it emanates from the body. This occasioned the question as to whether physiological output such as this could in fact be exploited as an input source. That is, if the body is giving off certain energies in certain patterns in relationship to some mental or physical behaviors, then it is conceivable to use them to control some device like a computer. The biosignal output need only be captured in a way that would allow it to be converted into an input for driving a device.

Concerning the expressional or command aspect the Grok Box environment, we will

integrate multiple output technologies for the issuing of commands. This aspect of the

project will be less technologically intensive and experimental than are the perceptual

aspects. As mentioned previously, the human body is a source of energy patterns which

when captured via sensors may become a means of giving commands. The muscles

generate electric voltage shifts when flexed and relaxed. EMG sensors, which are

becoming more and more powerful, are an excellent method for the Grok-Box user to

give commands. Our past research indeed shows the effectiveness of this means of

controlling output. Another means of control are pressure sensors. These can be situated

underneath the feet of the user and designed with multiple areas for diverse commands as

well as combinations of different areas. In this way, a user is able to issue many

commands per minute through their feet. Also, if the user is sitting rather than standing,

then pressure sensors may be placed in the seat of the chair as well as at the points of the

elbows.

Vocalizations are another strong means of output from the user. Speaking into an array

of different sensors designed for different input is what we have conceived for the

system. So, for example, some kinds of commands will be spoken with certain phonetic

structures or strings while other commands will employ very different phonetics.

This would be based on extensive research with the voice recognition technologies

currently available (e.g., 1,2,3,4). In addition to these outputs, the hands are able to use

elaborate joystick-like controllers with multiple control parameters. Many bodily options

for expression are available as we develop this networked decision support technology.

There are expression scenarios we will examine which take the energy patterns which

emanate of the brain (like in our EEG/MRI studies) and use them as signals for

controlling the system. As our work on this project progresses we will be searching for

better and more efficient ways to instrument our perceptualization/decision-support

environment.

Based on these insights, the issues of perceptualization and expression within information systems were raised and shown to be problems in need of solution for a wide range of users in diverse fields. Rehabilitation and clinical/emergency medicine have been the areas in which we have endeavored to achieve both an articulation of the problem and a generic conceptual framework for its solution. More recently, we have engaged with the humanitarian and disaster response communities to show the ways in which existing practices of information gathering, data representation, and decision support are inadequate. http://www.medibolt.com/gb2k/video/storyboard/index.html

Technology has been emerging which allows wholly different ways of interacting with information. Our strategy has been to research these technologies and, within particular contexts, to test and refine our ideas about how to use them. We now turn to the generic issues.

Anthrotronic Principles of Human-Information Interaction

Anthrotronics constitutes human-centered and, more specifically, mind-centered thinking about information and communications systems design. Rather than beginning with the perspective of power and storage, these are included in the larger framework of beginning with the needs and abilities of particular users as a ‘mind in the system’ acquiring and outputting information. Our assumption is that current interaction with vast and diverse information resources for varying and urgent purposes is hindered by:

· Neglect of the multiple sensory systems of the human body, and

· Perceptually inferior preparations of information for those senses that are used.

The visual sense has been given top priority in the area of human-information interaction. With few exceptions, data tend to be represented visually. Two other senses which are usually omitted from the information interaction: a) hearing and b) touch, which has seldom been exploited as a means of information gathering (exception: Braille for users who are blind). Even the way information has been prepared for the visual sense is often inferior, as it has been presented as text, numeric characters, and crude graphics. The medical expert is normally dealing either with printed textual material, or else with arcane visual forms of data produced by medical instruments like EEG and EKG. Text, numbers, symbols, squiggly lines, and graphs have perceptual qualities often inadequate to time and content requirements of those using the information (doctors, emergency medical personnel, etc.).

The information, as typically represented, is perceptually deficient, to the neglect of the extraordinary capacity of our brain to capture and process information from our senses. We are not saying that vision or text are inferior ways of accessing information. Rather, we are making a conceptual point that while these work well for some needs, the critical nature of some information requirements is hindered by reliance on them, especially when vast quantities of diverse information must be accessed, represented in all their richness, and then used in a rapid manner. Our goal is to offer conceptual-and concrete technological prototypical-solutions for information demand by rethinking the sensory and perceptual possibilities for how to render information to the human body for decision support.

Solution

Our goal, based on these concepts, is to develop systems that incorporate diverse multi-sensory representations of information into a unified dynamic interface. The approach is based in part on concepts in sensory physiology. A mind-centered orientation to human-information interaction asks first, “How does the human nervous system, through the senses, gather raw data and then present it as information to the mind?” The answers will help us create powerful interfaces structures and functions between minds and data. Thus we are proposing to integrate the conscious human user into this system as a computational resource: a mind (not just a user) in the loop. By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, “a sensorial combinatoric integrator.” That is, the mind and body inside the network interface we are proposing will be a locus of perception and expression: a reader and a responder in any information and decision-intensive process.

To this end, we will identify the optimal perceptual parameters in which information can best be rendered for each of the three senses named above (vision, audition, and touch). That is, what types of information are best rendered to each specific sense modality and how can we optimize the representation based on the unique processing properties of the sense in question?

Research in human sensory physiology, specifically sensory transduction mechanisms, demonstrates that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Feature extraction is defined by Kandel et al.[1] as “the selective detection and accentuation by sensory neurons of certain features of a stimulus.” Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile sense perception. Then, by using electrophysiological signals as input (see above), we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multimodal feedback and closed-loop interactions. We will endeavor to address and solve the deficiencies in conventional information representation and decision support through both the perceptualization of information and the enhancement of expressivity made possible in a single interactive interface that can be deployed in any place equipped to benefit from it. Information will be rendered for three senses rather than one. The eyes, ears, and skin will all become avenues for gathering data, with vocal and haptic signals allowing for command inputs. Perceptualization of information is an idea which assumes that, under high-intensity demand for diverse and voluminous information, it is best to divide the information. After processing with data mining techniques, the goal is to take some of the data and put them into a form highly accessible to the visual sense. Likewise, take some of the data and make them accessible to hearing, and similarly with the tactile surface of the body. Large quantities of diverse data may be transformed into multi-sensory forms of information. Here are the basic modalities of perception and expression:

· geometry, color, texture, and dynamics representing meaningful features of information that has been ‘visualized’

http://qube1.mindtel.com/users/projects/topper/10-11-2000_Naturala//

· tone, pitch, timbre, volume, duration, location representing meaningful features of information that has been ‘sonified’ (i.e. converted into sound).

· touch, felt position, motion, and force representing meaningful features of information that has been converted into forms which come into contact with hands, fingers, arms, or other skin and muscle sensations of the body known generally as tactile or haptic manifestations. Think of the Braille, used by the blind, and imagine an elaboration of this idea for accessing information without sound or image but with physical impressions across body surfaces.

Data Mining & Knowledge Discovery

Data mining and processing are part of the core functionality of the interface. In order to transform data into the different perceptualizations available to the user, a powerful means of turning repositories of data into novel and powerful information is required. The following relevant quotes are from www.spss.com:

· “Gold mining is a process for sifting through lots of ore to find valuable nuggets. Data mining is a process for discovering patterns and trends in large datasets to find useful decision making information … There are many different definitions of data mining. Almost all of them involve using today’s increased computing power and advanced analytical techniques to discover useful relationships in large databases.”

· “Data mining is a ‘knowledge discovery process of extracting previously unknown, actionable information from very large databases’” [Aaron Zornes, The META Group]

· “Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techniques” [Gartner Group].

Data can, of course, come in many forms. Some are in databases and data warehouses. However, a good deal of the data, with which a user of this interface will be interacting, would be generated on-the-fly in crisis situations. For example, medical monitors at the site of a human emergency would be streaming raw data into the “grok-it” interface for the user to perceive and respond to immediately. Or, if physicians were doing clinical work over the Web, there would be simultaneous processing of information from databases as well as on-the-fly generation of patient data. Counteracting the effects of bio-terrorism would be another example of requiring data from both large stationary stores and data coming from a vast array of different kinds of sensors at and near the site of emergency.

Objectives

We propose to research, prototype and evaluate an integrative interface matrix that couples data streams from sensors, micro-informatic technologies, and databases to the mind via an intelligent exploitation of the nervous system towards the enhancement of perceptual dimensionality and expressive capacity. This anthrotronic (human-scale instrumentation system) interface matrix will allow for the harnessing of the human nervous system in ways that increase the user’s ability to “grok”[2] and communicate the information being generated and transmitted by the vast multi-domain information-communication system.

Further, we will research, prototype and evaluate technologies that enable controllability and exploitability of the multichannel, multifunction concurrence of dynamically interconnectable bio-couplers to the info-com system. The foundation for this goal is the proposition that the information flow between external sources (representation) and direct experience (mind) is biased, restrained, constrained, limited, enhanced, and facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.

Finally, this research effort is concerned with developing a “reference architecture” (a formalized conceptual framework for research and technology development) for designing physio-informatically robust interactive human-computer interface systems to the information-communication systems. The function of the reference architecture will be to provide insight into the various components of the system in the context of how they might affect the flow of information as it passes through them. The primary focus will be to consider the information flow between the human and the com-system in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during/caused by the intentional/volitional interaction with information between a conscious human and an info-com system.

Facilities and Equipment to be used in this work

There are many companies and researchers working in this area, but for our

purposes we are looking at technologies such as SPIRE (Spatial Paradigm for

Information Retrieval and Exploration), AVS (Advanced Visualization Systems),

Mineset (SGI’s data mining and visualization suite), and/or the Institute for Human and

Machine Cognition’s Concept Map Software. Systems such as TACTICS being

developed by Fritz et al. at the Applied Science and Engineering Laboratories of the

duPont Institute at the University of Delaware also give a robust example of tactile

perceptualization. They write,

TACTICS is a system that converts visual information, such as the abundant computer images available on the Internet, into tactile information. Specifically, it produces a meaningful tangible depiction of a complex visual image. To represent a photograph, for example, in a tactually perceivable fashion, one must reduce detail in the image while retaining meaning. The visual component of our system is implemented in software as a sequence of image manipulation processes. These algorithms segment the image into regions of similar gray-level, detect and enhance edges between regions, filter out unnecessary details or noise, and threshold the image [6]. This process produces a simplified line drawn, or coloring book style, version of the original. The basis for our hypothesis is taken from comprehensive work in tactual perception [7]. Our current implementation is in the C programming language as an extension to the University of Pennsylvania's image processing application "XV" (C).

The International Community for Auditory Display is an excellent resource for thinking

about and implementing the ideas of sonifying data. In addition, certain products already

exist which make sonification technologies available. For example, the vOICe device

which is a wearable computer technology enabling the “hearing” of information.

Conclusion

In the various traditional models of human-information interaction it is customary to

think in terms of inputs and outputs. Our aim is to develop a systems model for

interactive human-information interface systems which is more representative of the total

reality in a given situation than traditional models. Taking into account the physiology of

the human synthesizer as a factored variable for data flow and processing will yield new

insights into maximizing the benefits of any decision making paradigm. This is the

development of a physiologic based reference architecture for designing and developing

interactive computer interface systems to match the human nervous system's ability to

transduce, transmit, and render to consciousness the necessary information to interact

intelligently with information.

For several years our team worked to intelligently advance the physio-informatic thesis of

human-information interaction. Not only have our ideas been well received and supported

by both governmental and private institutions, but we have also developed powerful

core technologies, both software and hardware, which will be the foundation of technical

prototyping and development we would do in this, the next stage of our work. The

development of a fully functional grok-it interface-system prototype will benefit those

needing the types of information interaction discussed herein.

C. Proposed Products and deliverable resulting from this research

D. SOW (with cross references to Cost proposal)

E. List of personnel, qualifications, proposer’s previous accomplishments, etc.