Volume 1. Section I. Administrative
1. BAA #: N66001-98-X-6905
2. Technical Topic Area: Human-Computer Interaction
3. Proposal Title:
Physio-Info-Tronics for Perceptualization Environments: An Anthrotronic
Interface System for the Emerging Information-Communications Matrix
4. Technical Point of Contact:
Dave Warner M.D. Ph.D
(800) 950-0849
davew@mindtel.com
davew@well.com
5. Administrative Point of Contact:
Professor
Department of Physics
Syracuse University
Syracuse NY 13244-1130
315-443-9107 (fax -9103)
edlipson@syr.edu
CEO, MindTel LLC
www.mindtel.com
6. Cost Summary of proposed research:
7. Contractor’s type of business: “Other Small Business”
8. Duns #: Mindtel: 084218218; Syracuse University: 00-225-7350
9. Cage Code: Mindtel: 1R0T4; Syracuse University: 03587
Volume 1. Section II. Detailed Proposal
A. Innovative
Claims
The amount and diversity of information being produced in the many fields of human knowledge is growing rapidly. Technology will increase the transfer rate of this information and globalization will empower smaller countries, groups, and individuals to generate and express data in novel areas of interest and formats. For humans wishing to sift through and make meaningful the enormous amount of relevant and irrelevant data flow from below, on, and above the planet requires a new synthesis model. In this text, our purpose is to articulate one such model.
Our research into human-computer interaction, particularly as it developed conceptually with the needs of users with disabilities, has allowed for some innovative and very effective ideas on how to optimize the human’s ability to exploit the powers of emerging information & communications systems. Anthrotronics (meaning “human instrumentation systems”) is a term we have introduced to emphasize the human as central in the design and use of information and communication technologies. The orientation of the model we propose is therefore towards a tailoring of information representation to the unique capacities of the nervous system to perceive and respond to information. http://www.medibolt.com/conbottom.htm.
Whether it be academic researchers creating and publishing new information based on existing material they have used in the development of new theories and methods, or the information that is generated, used and recorded during more sudden and sporadic occurrences of emergency medical response, the volume and differences in information will increase. Complicating this state of affairs are the integrations of information across boundaries and events made possible by hybridized communications technologies. Cross pollination and comparisons of the diverse data means that there are more and more possibilities for action when all this information is brought to bear on contexts of need. Interacting profitably with the many sources of information using the many means of communication now available requires new models of how humans may interact with information resources. Our research investigates the human ability to perceive, process, and act on information while understanding that this input and output is directly related to the physiological structures and functions of the nervous system within a given environment.
B. Technical
Approach
Purpose and Tasks of this work.
Our purpose is to develop a generic perceptualization and command environment which will “dock” on to a communications network and enable quick and effective interaction with information. Specifically, our goal is to allow for 1) multisensory perception of data coming from stored and emerging data sources (e.g., sensors); and 2) different modalities of bodily action by which to issue command decisions back out into the network. The picture is of one person occupying this perceptual and expressional environment and working within its functionality to meet the needs of experts in differing contexts of both
urgent and non-urgent information requirements. Medical and humanitarian applications of this generic system have been the contexts for our conceptual and technological development of the environment and its capacities. Next, we give a list of tasks we will aim to accomplish over the course of our research under SPAWAR funding grants.
Task 1. We will conceptualize and develop a plan for the design, construction, and implementation of a multisensory perceptualization and expressional command environment with generic functionality and therefore applicability across many contexts for rapid and information intensive needs.
Task 2. We will hold seminars and research symposia for the purpose of bringing those researchers and developers doing work related to the different technological areas associated with human-information interaction, representation, and decision support.
Task 3. We will design software systems to facilitate the coupling of diverse sensor technologies with those systems responsible for raw data processing.
Task 4. We will design software systems to take processed data from sensors and communicate that data to the interface engines of the perceptual/expressional environment.
Task 5. We will develop rules for how differing kinds of data are parsed across the three senses we are working with.
Task 6. We will develop perceptualization protocols for the visual, binaural, and tactile senses of the human body.
Task 7. We will plan for and incorporate incremental hardware components as necessitated by the conceptual and software development of the project as it progresses.
History and Problem
Initial research was conducted in the neurology and rehabilitation departments of Loma Linda University Medical Center. This work led to a new focus on perception and expression as the key problems in re-thinking how humans might improve interaction with information systems. Historically, these problems were sidelined by the emphasis on faster more powerful computation and greater storage. Human factors were, therefore, largely omitted from the consideration and design of information systems. Below we provide a brief background on how these issues were formulated in the particular contexts of medical diagnostics and rehabilitation. These were merely the starting points for the interrogation of the logic of human-information interaction, in which field we are now fully engaged.
Perception
Electroencephalographic (EEG) and Magnetic Resonance Imaging (MRI) devices were both used to gather clinical data from the human brain. These are the among the tools of the clinical neurologist. Electrophysiological output from the brain is what is thereby measured and output as either “squiggly lines” (EEG) or crude image maps of the head (MRI). Experimentation with electrocardiograms (EKG) was also undertaken. This research led to the conclusion that the perceptual forms of the data generated by these technologies were inadequate to the task of providing precise diagnostic information to the clinician. Further, the form of the data prevented anyone but the highly experienced clinician from making decisions concerning obscure features in the data. Based on this experience, the perceptualization of data became a general problem to be explored and addressed. Towards the goal of better perceptualization of data we will explore three separate areas of representation of data to the senses: visual, binaural, and tactile. In the area of visual representation our objective is to transform standard visual forms of information into new and more powerful representations for the eyes. A great deal of information critical to medicine, science, business, and many other fields, is stored in vast repositories. The increasing of those stores only complicates the already difficult process of accessing and understanding needed information. Text is the visual form of much of this information. Therefore, our work will be to identify already existing technologies for the transformation of textual information into visualizations. There are many such resources on the horizon. As we identify which are the best for the purposes of the Grok Box we will want to purchase, test, and ultimately integrate them into the perceptual/expressional environment we are developing.
Binaural/auditory representations of information are another tier of the project. The concept of auditory display is the principle for this form of perceptualization. "Auditory display research applies the ways we use sound in everyday life to the human/machine interface and extends these uses via technology."(Gregory Kramer, An Introduction to Auditory Display, 1994 ). This is also known as the sonification of data. Researchers at IBM are engaging in the exploration of sonifying data sets. Researcher David Jameson explains that the purpose of sonification is “to augment or even replace traditional, visual ways of transmitting computerized information. In other words, using programs to transmit information in aural ways much as graphs, numbers, or pictures transmit data visually.” Turning data sources into sound is a powerful way of representing information which is able to take some of the total load of needed data and making it more quickly available through another sense modality such as:
An atmospheric selector is used to transform the spectral frequencies of natural electromagnetic impulse radiation into the human auditory range. The resulting sound pattern is determined by meteorological conditions. A change in the sound pattern of atmospheric electrical activity permits a change in the meteorological situation in the area of the measurement station intake area to be directly detected acoustically. <http://www.sti.nasa.gov/rselect/openlit.html>
Finally, there is the technology which utilizes the perceptual capacities of human skin surfaces (tactile) and the body’s sensitivities to pressure and tension (haptics).Tactile and haptic perceptualization is the third mode of sensing and responding which our project will explore and develop. In its most basic form, tactile representation of data occurs in the form of Braille, the literary medium of the blind. Haptic interfaces are also currently being developed and employed successfully in many surgical theatres as robotic telemedicine becomes an accepted form of medical intervention.
There is quite a diversity of methods for rendering information to the skin and limbs of the body. Our task will be to bring together several of the most promising technologies currently being developed. The value of this and the former two modes of perceptualization is extraordinary. By spreading data across three senses and doing so in ways which truly capitalize on the nervous system’s physiologic data processing from those senses, a single person will be in position to integrate and then respond to an unparalleled quantity of information for some critical purpose.
Ergonomics
Ergonomics is the study of body posture and orientation and is assessed using Electromyography (EMG) sensors and a 3-D video technique termed Digital Surface Photogrammetry. This analysis becomes a critical human factors variable during sustained sitting, standing or in ambulatory activities. These subtle yet potentially limiting variables will be addressed when considering such elements as fatigue and task distraction under stressful situations. EMG measures muscle activity and when combined with Photogrammetry and wearable bend sensors allows for an insightful skeletal-muscular analysis and can reveal potential problems associated with input equipment (monitors, sensors, speakers) positioning and output control devices (joy-stick, mouse, keyboard, microphone) interactivity.
Expression
Neuro-diagnostic studies of the brain also triggered the awareness that electrophysiological output comes not just from the brain, but from the muscles of the entire body. The body is emanating certain kinds of energy as measurable patterns at all times. EEG, EKG, and other physiological monitoring instruments simply ‘read’ this energy as data, as it emanates from the body. This occasioned the question as to whether physiological output such as this could in fact be exploited as an input source. That is, if the body is giving off certain energies in certain patterns in relationship to some mental or physical behaviors, then it is conceivable to use them to control some device like a computer. The biosignal output need only be captured in a way that would allow it to be converted into an input for driving a device.
Concerning the expressional or command aspect the Grok Box environment, we will integrate multiple output technologies for the issuing of commands. This aspect of the project will be less technologically intensive and experimental than are the perceptual aspects. As mentioned previously, the human body is a source of energy patterns which when captured via sensors may become a means of giving commands. The muscles generate electric voltage shifts when flexed and relaxed. EMG sensors, which are becoming more and more powerful, are an excellent method for the Grok-Box user to give commands. Our past research indeed shows the effectiveness of this means of controlling output. Another means of control are pressure sensors. These can be situated
underneath the feet of the user and designed with multiple areas for diverse commands as well as combinations of different areas. In this way, a user is able to issue many commands per minute through their feet. Also, if the user is sitting rather than standing, then pressure sensors may be placed in the seat of the chair as well as at the points of the elbows.
Vocalizations are another strong means of output from the user. Speaking into an array of different sensors designed for different input is what we have conceived for the system. So, for example, some kinds of commands will be spoken with certain phonetic structures or strings while other commands will employ very different phonetics. This would be based on extensive research with the voice recognition technologies currently available (e.g., 1,2,3,4). In addition to these outputs, the hands are able to use elaborate joystick-like controllers with multiple control parameters. Many bodily options for expression are available as we develop this networked decision support technology. There are expression scenarios we will examine which take the energy patterns which emanate of the brain (like in our EEG/MRI studies) and use them as signals for controlling the system. As our work on this project progresses we will be searching for better and more efficient ways to instrument our perceptualization/decision-support environment.
Based on these insights, the issues of
perceptualization and expression within information systems were raised and
shown to be problems in need of solution for a wide range of users in diverse
fields. Rehabilitation and clinical/emergency medicine have been the areas in
which we have endeavored to achieve both an articulation of the problem and a
generic conceptual framework for its solution. More recently, we have engaged
with the humanitarian and disaster response communities to show the ways in
which existing practices of information gathering, data representation, and
decision support are inadequate. http://www.medibolt.com/gb2k/video/storyboard/index.html
Technology has been emerging which allows wholly different ways of interacting with information. Our strategy has been to research these technologies and, within particular contexts, to test and refine our ideas about how to use them. We now turn to the generic issues.
Anthrotronic
Principles of Human-Information Interaction
Anthrotronics constitutes human-centered and, more specifically, mind-centered thinking about information and communications systems design. Rather than beginning with the perspective of power and storage, these are included in the larger framework of beginning with the needs and abilities of particular users as a ‘mind in the system’ acquiring and outputting information. Our assumption is that current interaction with vast and diverse information resources for varying and urgent purposes is hindered by:
· Neglect of the multiple sensory systems of the human body, and
· Perceptually inferior preparations of information for those senses that are used.
The visual sense has been given top priority in the area of human-information interaction. With few exceptions, data tend to be represented visually. Two other senses which are usually omitted from the information interaction: a) hearing and b) touch, which has seldom been exploited as a means of information gathering (exception: Braille for users who are blind). Even the way information has been prepared for the visual sense is often inferior, as it has been presented as text, numeric characters, and crude graphics. The medical expert is normally dealing either with printed textual material, or else with arcane visual forms of data produced by medical instruments like EEG and EKG. Text, numbers, symbols, squiggly lines, and graphs have perceptual qualities often inadequate to time and content requirements of those using the information (doctors, emergency medical personnel, etc.).
The information, as typically represented, is perceptually deficient, to the neglect of the extraordinary capacity of our brain to capture and process information from our senses. We are not saying that vision or text are inferior ways of accessing information. Rather, we are making a conceptual point that while these work well for some needs, the critical nature of some information requirements is hindered by reliance on them, especially when vast quantities of diverse information must be accessed, represented in all their richness, and then used in a rapid manner. Our goal is to offer conceptual-and concrete technological prototypical-solutions for information demand by rethinking the sensory and perceptual possibilities for how to render information to the human body for decision support.
Solution
Our goal, based on these concepts, is to develop systems that incorporate diverse multi-sensory representations of information into a unified dynamic interface. The approach is based in part on concepts in sensory physiology. A mind-centered orientation to human-information interaction asks first, “How does the human nervous system, through the senses, gather raw data and then present it as information to the mind?” The answers will help us create powerful interfaces structures and functions between minds and data. Thus we are proposing to integrate the conscious human user into this system as a computational resource: a mind (not just a user) in the loop. By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, “a sensorial combinatoric integrator.” That is, the mind and body inside the network interface we are proposing will be a locus of perception and expression: a reader and a responder in any information and decision-intensive process.
To this end, we will identify the optimal perceptual parameters in which information can best be rendered for each of the three senses named above (vision, audition, and touch). That is, what types of information are best rendered to each specific sense modality and how can we optimize the representation based on the unique processing properties of the sense in question?
Research in human sensory physiology, specifically sensory transduction mechanisms, demonstrates that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Feature extraction is defined by Kandel et al.[1] as “the selective detection and accentuation by sensory neurons of certain features of a stimulus.” Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile sense perception. Then, by using electrophysiological signals as input (see above), we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multimodal feedback and closed-loop interactions. We will endeavor to address and solve the deficiencies in conventional information representation and decision support through both the perceptualization of information and the enhancement of expressivity made possible in a single interactive interface that can be deployed in any place equipped to benefit from it. Information will be rendered for three senses rather than one. The eyes, ears, and skin will all become avenues for gathering data, with vocal and haptic signals allowing for command inputs. Perceptualization of information is an idea which assumes that, under high-intensity demand for diverse and voluminous information, it is best to divide the information. After processing with data mining techniques, the goal is to take some of the data and put them into a form highly accessible to the visual sense. Likewise, take some of the data and make them accessible to hearing, and similarly with the tactile surface of the body. Large quantities of diverse data may be transformed into multi-sensory forms of information. Here are the basic modalities of perception and expression:
· geometry, color, texture, and dynamics representing meaningful features of information that has been ‘visualized’
http://qube1.mindtel.com/users/projects/topper/10-11-2000_Naturala//
· tone, pitch, timbre, volume, duration, location representing meaningful features of information that has been ‘sonified’ (i.e. converted into sound).
· touch, felt position, motion, and force representing meaningful features of information that has been converted into forms which come into contact with hands, fingers, arms, or other skin and muscle sensations of the body known generally as tactile or haptic manifestations. Think of the Braille, used by the blind, and imagine an elaboration of this idea for accessing information without sound or image but with physical impressions across body surfaces.
Data Mining &
Knowledge Discovery
Data mining and processing are part of the core functionality of the interface. In order to transform data into the different perceptualizations available to the user, a powerful means of turning repositories of data into novel and powerful information is required. The following relevant quotes are from www.spss.com:
· “Gold mining is a process for sifting through lots of ore to find valuable nuggets. Data mining is a process for discovering patterns and trends in large datasets to find useful decision making information … There are many different definitions of data mining. Almost all of them involve using today’s increased computing power and advanced analytical techniques to discover useful relationships in large databases.”
· “Data mining is a ‘knowledge discovery process of extracting previously unknown, actionable information from very large databases’” [Aaron Zornes, The META Group]
· “Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techniques” [Gartner Group].
Data can, of course, come in many forms. Some are in databases and data warehouses. However, a good deal of the data, with which a user of this interface will be interacting, would be generated on-the-fly in crisis situations. For example, medical monitors at the site of a human emergency would be streaming raw data into the “grok-it” interface for the user to perceive and respond to immediately. Or, if physicians were doing clinical work over the Web, there would be simultaneous processing of information from databases as well as on-the-fly generation of patient data. Counteracting the effects of bio-terrorism would be another example of requiring data from both large stationary stores and data coming from a vast array of different kinds of sensors at and near the site of emergency.
Objectives
We propose to research, prototype and evaluate an integrative interface matrix that couples data streams from sensors, micro-informatic technologies, and databases to the mind via an intelligent exploitation of the nervous system towards the enhancement of perceptual dimensionality and expressive capacity. This anthrotronic (human-scale instrumentation system) interface matrix will allow for the harnessing of the human nervous system in ways that increase the user’s ability to “grok”[2] and communicate the information being generated and transmitted by the vast multi-domain information-communication system.
Further, we will research, prototype and evaluate technologies that enable controllability and exploitability of the multichannel, multifunction concurrence of dynamically interconnectable bio-couplers to the info-com system. The foundation for this goal is the proposition that the information flow between external sources (representation) and direct experience (mind) is biased, restrained, constrained, limited, enhanced, and facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.
Finally, this research effort is concerned with developing a “reference architecture” (a formalized conceptual framework for research and technology development) for designing physio-informatically robust interactive human-computer interface systems to the information-communication systems. The function of the reference architecture will be to provide insight into the various components of the system in the context of how they might affect the flow of information as it passes through them. The primary focus will be to consider the information flow between the human and the com-system in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during/caused by the intentional/volitional interaction with information between a conscious human and an info-com system.
Facilities and Equipment to be used in this work
There are many companies and researchers working in this
area, but for our purposes we are looking at technologies such as SPIRE (Spatial Paradigm for
Information Retrieval and Exploration), AVS (Advanced Visualization Systems), Mineset (SGI’s data mining and visualization suite), and/or the Institute for Human and Machine Cognition’s Concept Map Software. Systems such as TACTICS being developed by Fritz et al. at the Applied Science and Engineering Laboratories of the duPont Institute at the University of Delaware also give a robust example of tactile perceptualization. They write,
TACTICS is a system that converts visual information, such as the abundant
computer images available on the Internet, into tactile information.
Specifically, it produces a meaningful tangible depiction of a complex visual
image. To represent a photograph, for example, in a tactually perceivable
fashion, one must reduce detail in the image while retaining meaning. The
visual component of our system is implemented in software as a sequence of
image manipulation processes. These algorithms segment the image into regions
of similar gray-level, detect and enhance edges between regions, filter out
unnecessary details or noise, and threshold the image [6]. This process produces
a simplified line drawn, or coloring book style, version of the original. The
basis for our hypothesis is taken from comprehensive work in tactual perception
[7]. Our current implementation is in the C programming language as an
extension to the University of Pennsylvania's image processing application
"XV" (C).
The International Community for Auditory Display is an
excellent resource for thinking about
and implementing the ideas of sonifying data.
In addition, certain products already exist which make sonification
technologies available. For example,
the vOICe device which is a wearable
computer technology enabling the “hearing” of
information.
Conclusion
In the various traditional models of human-information interaction it is customary to think in terms of inputs and outputs. Our aim is to develop a systems model for interactive human-information interface systems which is more representative of the total reality in a given situation than traditional models. Taking into account the physiology of the human synthesizer as a factored variable for data flow and processing will yield new insights into maximizing the benefits of any decision making paradigm. This is the development of a physiologic based reference architecture for designing and developing interactive computer interface systems to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information. For several years our team worked to intelligently advance the physio-informatic thesis of human-information interaction. Not only have our ideas been well received and supported by both governmental and private institutions, but we have also developed powerful core technologies, both software and hardware, which will be the foundation of technical prototyping and development we would do in this, the next stage of our work. The development of a fully functional grok-it interface-system prototype will benefit those needing the types of information interaction discussed herein.
C. Proposed
Products and deliverable resulting from this research
D. SOW (with
cross references to Cost proposal)
E. List of
personnel, qualifications, proposer’s previous accomplishments, etc.
1. FIRST PRINCIPLES OF PHYSIO INFORMATIC SYSTEMS
The conceptual, theoretical and experimental basis for a
general systems reference architecture for Physio-informatic systems
Medical Neuroscientist
Dir. Medical Intelligence
MindTel
Physio-informatics is a new systems model for
linking human physiologic systems to information systems in the most general
way. This general systems model has been derived through an ever evolving
series of experiments and explorations.
The conceptual, theoretical and analytical basis for
establishing a general systems based
reference architecture for Physio-informatic systems necessarily crosses
many disciplines. It must be emphasized
from the onset that the following discussion of the derivation and development of a general model (aka..
reference architecture ) for describing “meaningful” information flow between humans and informatic systems is a broad
topic area which covers many scientific disciplines, engineering techniques and
a continually expanding array of technologies. Including but not limited to
Physiology, Physics, Mathematics, Philosophy, General Systems, Bio-Cybernetics
Systems, Cognitive Neuroscience, Perceptual Psycho-Physics, Perceptual State
Space Modulation, Bio-Sensors, Quantitative Human Performance, Expressional
Interface Systems, Physio-Informatics, Intelligent Interface-Metrics, User
Tracking Interface Systems, Distributed Tele-Robotic Controllers and
Intermental Networking.
A general perspective of this effort is that it is an
attempt at integrating these areas of human scientific endeavor (as mentioned
above) in a manner which will not require that future researchers in
Physio-Informatics master all of them
before they can contribute meaningfully to the process of optimizing the
coupling between humans and informatic systems in an interactive interface
system. Thus the intent of this effort is to establish a general conceptual
framework (a reference architecture) which can be used as a guiding
heuristic tool when confronted with the challenge of designing and
developing interactive interface systems for human computer interaction.
Specifically one which extends perceptual dimensionality and facilitates
enhanced expressivity.
A systems based, physiologically robust, reference architecture for designing and refining interactive human-computer interface systems in ways which increase operational throughput of information.
The term“ physio-informatics” will be used in this
dissertation to denote informatic systems which are either
biologically/physiologically based (primarily neurologic i.e. neuro informatic)
information systems and/or informatic systems which are designed to support
interaction (dynamic exchange of information) with such systems
The intent of this work is to develop a systems based,
physiologically robust, reference architecture for designing and refining
interactive human-computer interface systems in ways which increase operational
throughput of information. Extending the perceptual dimensionality of
information presented to the human and enhancing the expressional capacity of
the human to convey intent to the informatic system achieve this increased
throughput.
Interactive Human-Computer Interface Systems
In the various traditional models of human computer it is
customary to think in terms of inputs and outputs. Input from the computer to
the human and out from the human to the computer or input from the human to the
computer and output of the computer to the human. The purpose of this
dissertation is to develop a systems model for interactive human-computer interface
systems which is thought to be more representative of reality than traditional
models in that it is consistent with the phenomenological aspects. That is the
development of a physiologic based reference architecture for designing and
developing interactive human computer interface systems to match the human
nervous system's ability to transduce, transmit, and render to consciousness
the necessary information to interact intelligently with information.
"The physiologic basis of a reference architecture for
designing interactive human-computer interface systems"
The capacity of computers to receive, process, and transmit
massive amounts of information is continually increasing. Current attempts to
develop new human-computer interface technologies have given us devices such as
gloves, motion trackers,3-D sound and graphics. Such devices greatly enhance
our ability to interact with this increasing flow of information. Interactive
interface technologies emerging from the next paradigm of human-computer
interaction are directly sensing bio-electric signals (from eye, muscle and
brain activity) as inputs and rendering information in ways that take advantage
of psycho-physiologic signal processing of the human nervous system (perceptual
psychophysics). The next paradigm of human-computer interface will optimize the
technology to the physiology -- a biologically responsive interactive
interface.
Interactive information technology is any technology which
augments our ability to create / express / retrieve / analyze / process /
communicate / experience information in an interactive mode. Biocybernetics
optimizes the interactive interface, promising a technology that can profoundly
improve the quality of life of real people today. The next paradigm of
interface technology is based on new theories of human-computer interaction,
which are physiologically and cognitively oriented. This emerging paradigm of
human computer interaction incorporates multi-sense rendering technologies,
giving sustained perceptual effects, and natural user interface devices which
measure multiple physiological parameters simultaneously and use them as
inputs. Biologically optimized interactive information technology has the
potential to facilitate effective communication. This increase in effectiveness
will impact both human-computer and human-human communication, "enhanced
expressivity".
Interactive interface technology renders content specific
information onto multiple human sensory systems giving a sustained perceptual
effect, while monitoring human response, in the form of physiometric gestures,
speech, eye movements and various other inputs. Such quantitative measurement of activity during purposeful tasks
allows us to quantitatively characterize individual cognitive styles. This
capability promises to be a powerful tool for characterizing the complex nature
of normal and impaired human performance. The systems of the future will monitor
a user's actions, learn from them, and adapt by varying aspects of the system's
configuration to optimize performance. By immersion of external senses and
iterative interaction with biosignal triggered events complex tasks are more
readily achieved. This paradigm shift of mass communication and information
technologies is providing an exciting opportunity to facilitate the rapid
exchange of relevant information thereby increasing the individual productivity
of persons involved in the information industry. Areas such as
computer-supported cooperative work, knowledge engineering, expert systems,
interactive attentional training, and adaptive task analysis will be changed
fundamentally by this increase in informatic ability. The psycho-social
implications of this technologically mediated human-computer and human-human
communication are quite profound.
Providing the knowledge and technology required to empower people to
make a positive difference with information technology could foster the
development an attitude of social responsibility towards the usage of this
technology and may be a profound step forward in modern social development.
Applications which are intended to improve quality of life, such as,
applications in medicine; education, recreation and communication must become a
social priority.
Knowledge of sensory physiology and perceptual psychophysics
is being used to optimize our future interactions with the computer. By
increasing the number and variation of simultaneous sensory inputs, we can make
the body an integral part of the information system, "a sensorial
combinetric integrator". We can then identify the optimal perceptual state
space parameters in which information can best be rendered. That is what types
of information are best rendered to each specific sense modality, "a sense
specific optimization of rendered information. Research in human sensory
physiology, specifically sensory transduction mechanisms, shows us that there
are designs in our nervous systems optimized for feature extraction of
spatially rendered data, temporally rendered data, and textures. Models of
information processing based on the capacity of these neurophysiological
structures to process information will help our efforts to enhance perception
of complex relationships by integrating visual, binaural, and tactile
modalities. Then by using the natural bioelectric energy as a signal source for
input; electroencephalography, electroocculography, and electromyography
(brain, eye and muscle) we can generate highly interactive systems in which
these biological signals initiate specific events. Such a real-time analysis
enables multi-modal feedback and closed-loop interactions.
The following discussion is concerned with developing a
“reference architecture” (a formalized conceptual framework for thinking) for
designing physiologically robust interactive human computer interface systems.
The purpose of the reference architecture will be to provide insight into the
various components of the system in the context of how they might affect the
flow of information as information is passed through them The primary focus
will be to consider the flow of information between the human and the computer
in a sustained, iterative, experiential interaction In the context of this
dissertation it will be assumed that the intent of developing this reference
architecture is to map the information flow during/caused by the intentional
/volitional interaction with information between a conscious human and a
computer system An exchange of information between the an experienced
perceptual state and an external physical state is mediated by a biologic /
physiologic information transporter system This system is multi modal – multi
scale – concurrent hetero-purpose poly-dyno- morphic simul-tasking
For this discussion we will assume that interface systems
which support Human computer interaction can be modeled as a system where
information flows between various components of the system in a specific manner
Theoretical position –
Information can be mapped and represented as a specific state space parameter set.
The phenomena of interest, (perception and expression),
occurs at the anthroscopic scale.
The anthroscopic scale, the natural scale of perceptibility
and expressivity of an individual human, is “From meters to millimeters, from
decades to deci-seconds.”
The nervous system is the primary information infrastructure
for humans.
The nervous system supports the transduction transmission
representation and response to information in the environment.
Human perception and expression is mediated, for the most part,
by the nervous system.
An understanding of the human neuro physiology allows for
exploitation of predictable adaptive capabilities. The assertion is that the
information flow between external sources and direct experience is
biased/restrained/constrained/limited/enhanced/facilitated in understandable
and predictable ways by the physiological mechanisms of human information
processing.
Physio info metrics --- the quantitative measure of
the information carrying capacity of a physiologic system.
Physiologically mediated information is exchanged between
external environment and experiential awareness. The fundamental nature of the
nervous system (neuro info matrix) determines its operational capacity. Both
the physicality and the physiology contribute to the set of bio-physical
restraints. The physicality of the nervous system constrains the perception of
space, time, mass and energy. Physiology of the human nervous system restrains
perception by computational limits of the system. The complexity, functionality
and capacity of the intra-activity of the nervous system sustains perception.
ERGO - The form and function of the nervous system influence various parameters
of perception and expression. That is to say that nervous system is the biologic
structure that is considered most likely to be responsible for mediating
information flow within the human body
The Basic ideas leading to the primary foundations for this
thinking can be seen as coming from the following areas
-Action directed goals in the pursuit of new knowledge -
which start with logical analysis of observed phenomena and proceed to the point of discerning an
operational utility of continuing the pursuit in the current mode of analysis
or changing modes to seek a more fruitful mode of investigating the phenomena.
(Oppenhiemer)
In other words it is a philosophy of scientific
investigation which constantly seeks to validate the current mode of analysis
for a given set of observed phenomena so as to maintain constant progress in
the discovery process of new knowledge.
General systems theory is a useful framework for developing
complex models for investigating complex systems, like those of
Physio-Informatics, is in as far as it has certain concepts of systems models
and principles such as hierarchical order, progressive differentiation and feedback that can be defined and
characterized and elaborated on with set and graph theory which state
explicitly conditions for membership and orders of relationship.
The “open systems” approach to a general systems theory by von Bertalanffy in the late 1930’s was
instigated by a perceived need to break out of the “closed systems” model which
implicitly separates the system from its environment, as it would lead to
incorrect conclusions. His concept was that biological systems necessarily must
be considered as being open systems where both information and energy is in
continuous flow between the system and the environment. His initial formulation
of a general system was an attempt to derive principles which were valid for
open systems.
A system can be defined as an object consisting of a set of complex objects or relationships,
each of which are in some way associated with other objects with in the system
in a way that some quantities (parameters) with in those objects are associated
with quantities (parameters) of other objects within the same system.
( von Bertalanffy)
The base elements with which information is constructed is “difference”. A difference can be
interpreted as either an ontological fact or as an abstract matter. Information can be defined as a difference
which makes a difference. (Bateson
1970) Or a difference with a non zero
significance (Warner)
The relevant aspects of Information Theory concerning the
transmission effects on information
across physical structures, are considered to be important in physio-informatic
systems, but are tempered by the fact that biological systems do not
adhere to the neg-entropy formulation
of Shannon
Also of significant importance is theory of Cybernetics, the
theoretical model of feedback governed systems whose present state influences
in some way the probabilities of any future state occurring in the system. It
is interesting to note that the operators which are invoked on the system are a
result of past or currents states. This is important to establish that there is
a relationship between operators and states beyond the “transformational function”
of operators on states.
A state of any system is defined by the set values which
describe the condition of the system in any given point in time (the value of
all the state vectors). A system will have a state space which represents/contains
all possible states of that system.
For those systems whose quantities are in continuous flux a
special kind of set called a “State Space” can be constructed which has as its
elements (set members) an n-tuple of values which are the values of the
quantities at a given instant. At any given instant the system is said to have
a “state” which is determined by the values of each of the “parameters” at that
instant. (Ashby, Zadeh)
For a given system whose States are not static (in time) within a given state-space it can be asserted that a transformational function has been performed on the system which determines the “next” state the system will be in. Such a transformational function is called an operator. Thus it is correct to say that an operator acts on an initial state parameter value and produces a new state parameter value.
In an open system is can be asserted that the “evolution of
the states in time” i.e. the “state space trajectory” can be considered to be
influenced by both the current state of the system (internal factors) and the
processes of the environment (external factors) which are acting on the system.
A strange but useful mathematical modeling system for elaborating this has been established. A state space of a physio-informatic system can be described as a set of information-based states which behave in a particular manner.
The initial assumption is that all the information that is perceived about the external environment is filtered-mediated-biased by the nervous system. This is known as the “Neuro Cosmological Principle” of epistemology, which has an “Anthro-scopic scale” and an “Anthro-centric perspective”.
2.
BIO-CYBERNETICS:
A Biologically Responsive
Interactive Interface: "Adventures In The Next Paradigm Of Human Computer Interaction"
Dave Warner, Jeff
Sale, Todd Anderson, Jo Johanson
Human Performance Institute
Dept. of Rehabilitation Engineering,
Loma Linda University Medical Center (1995)
Abstract
The capacity of computers to receive, process, and transmit massive amounts of information is continually increasing. Current attempts to develop new human-computer interface technologies have given us devices such as gloves, motion trackers, 3-D sound and graphics. Such devices greatly enhance our ability to interact with this increasing flow of information. Interactive interface technologies emerging from the next paradigm of human-computer interaction are directly sensing bio-electric signals (from eye, muscle and brain activity) as inputs and rendering information in ways that take advantage of psycho-physiologic signal processing of the human nervous system (perceptual psychophysics). The next paradigm of human-computer interface will optimize the technology to the physiology -- a biologically responsive interactive interface.
BIOCYBERNETICS: Interactive Information Technology
Interactive information technology is any technology which augments our ability to create / express / retrieve / analyze / process / communicate / experience information in an interactive mode. Biocybernetics optimizes the interactive interface, promising a technology that can profoundly improve the quality of life of real people today. The next paradigm of interface technology is based on new theories of human-computer interaction which are physiologically and cognitively oriented. This emerging paradigm of human computer interaction incorporates multi-sense rendering technologies, giving sustained perceptual effects, and natural user interface devices which measure multiple physiological parameters simultaneously and use them as inputs. Biologically optimized interactive information technology has the potential to facilitate effective communication. This increase in effectiveness will impact both human-computer and human-human communication, "enhanced expressivity". Work in human-computer interaction is an ongoing endeavor in many areas. These efforts have captured the attention of several professional societies; the entertainment industry, the aerospace industry, communications and educational technologies industries, as well as medicine. These diverse areas will all be impacted in multiple ways by advances in technologies that enhance human-computer interaction.
Optimizing the human computer interface will rely on the knowledge base of physiology and neuroscience, that is, the more we know about the way we acquire information physiologically the more we know the optimum way for a human to interact with intelligent information systems. The next paradigm will see the "THINNING" of the human-computer interface to a biological sheer as the interface will map very close to the human body.
Physiologically Oriented Interface Design
Knowledge of sensory physiology and perceptual psychophysics is being used to optimize our future interactions with the computer. By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, "a sensorial combinetric integrator". We can then identify the optimal perceptual state space parameters in which information can best be rendered. That is what types of information are best rendered to each specific sense modality, "a sense specific optimization of rendered information. Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile modalities. Then by using the natural bioelectric energy as a signal source for input; electroencephalography, electroocculography, and electromyography (brain, eye and muscle) we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multi-modal feedback and closed-loop interactions.
Biocybernetic Controller
Interactive interface technology renders content specific information onto multiple human sensory systems giving a sustained perceptual effect, while monitoring human response, in the form of physiometric gestures, speech, eye movements and various other inputs. Such quantitative measurement of activity during purposeful tasks allows us to quantitatively characterize individual cognitive styles. This capability promises to be a powerful tool for characterizing the complex nature of normal and impaired human performance. The systems of the future will monitor a user's actions, learn from them, and adapt by varying aspects of the system's configuration to optimize performance. By immersion of external senses and iterative interaction with biosignal triggered events complex tasks are more readily achieved.
This paradigm shift of mass communication and information technologies is providing an exciting opportunity to facilitate the rapid exchange of relevant information thereby increasing the individual productivity of persons involved in the information industry. Areas such as computer-supported cooperative work, knowledge engineering, expert systems, interactive attentional training, and adaptive task analysis will be changed fundamentally by this increase in informatic ability. The psycho-social implications of this technologically mediated human-computer and human-human communication are quite profound. Providing the knowledge and technology required to empower people to make a positive difference with information technology could foster the development an attitude of social responsibility towards the usage of this technology and may be a profound step forward in modern social development. Applications which are intended to improve quality of life, such as, applications in medicine, education, recreation and communication must become a social priority.
Using Technology to Improve Quality of Life
The potential of this technological capability to improve quality of life can be best understood when it is actualized into the lives of real people with real needs. The Human Performance Institute at Loma Linda University Medical Center is an interdisciplinary research center which is leading the effort to utilize the latest in human computer interface technology to "make the world a better place". The primary research areas are in developing interactive interfaces which enable severely disabled individuals to lead productive lives, and in the design of environmental systems which support experiential interaction with information systems in such a way as to help maintain a state of general good health.
The following are real world cases that demonstrate the utility of this technology to change the future of disabled individuals:
Crystal, an 18 month old "C1 quadriplegic" (complete paralysis from the neck down, requires a respirator in order to breathe) was the first person to use this biocybernetic technology in a medical setting. Processing of electropotential changes along the eye and adjacent muscles into a biological signal enabled this child to interact in real time with the displays on the monitor, in short, "her eyes became her hands" in generating commands to the screen. The activity was direct, the implications profound: She was able to enter into a unified feedback loop where direct real time response to a physiological signal was used to modify and improve that psycho-physiological source. In this case, her capacity to learn and interact with the world willfully was restored.
Andy, a 10 year old C2 quadriplegic whose speech is confined to the breathing patterns of his respirator to such an extent that it requires better than a minute to make a verbal request found himself in a spatialized environment where commands from facial muscles enabled him to "fly around" in a 3d computer environment. This was the first time in 5 years where he was able to willfully control something in his environment without the aid of others. A 17 year old car accident victim who was motivated to rehabilitate his impaired psycho motor skills through an "air guitar" interactive system which converted the weak bioelectric signals from his impaired muscles into "rock and roll" music.
We have also developed the BioCar, a primitive yet functional demonstration of telerobotic devices under direct biocybernetic control. The BioCar is a simple demonstration of how the biosignals can be used to control objects within an environment. For this demonstration a remote control car from Radio Shack was modified so that it can be controlled from the parallel port of a standard IBM compatible PC. Since there are only seven discrete functions (there is no proportional control) that the car can perform (forward, forward left, forward right, stop, reverse, reverse left and reverse right) then it takes a minimum of three sets of electrodes to control all of the functions (23=8). The BioCar software is responsible for interpreting the bioelectric signals from the user and sending commands to the remote control car.
Michael, a 27 year old engineer recently paralyzed in an auto accident was able to navigate the BioCar through a very complicated course using the muscles of his face and arms. The same system that allowed him to control this toy car could be easily adapted to control his wheel chair or some type of robotic arm. The potential to empower the disabled to become functional members of society can be realized through biocybernetic interface design.
The next effort of our lab was to expand the utility of this biocybernetic controller. We modified a nintendo game to accept commands from our system as if they were coming from the regular hand controller. This simple modification allows disabled children to use whatever muscle activity they have control of to play the same games as normal children. This generalized biocybernetic controller opens up an enormous resource of compelling games which can be integrated into rehabilitative therapy. From the control of virtually nothing to really something, we can get coordinated motion from patients at a much earlier time. Instead of some arbitrary task, they can work with computer generated objects that have specific motions associated with them; getting the associated feedback of watching themselves pick up a virtual object even though you may lack the physical strength to pick up a real object.
Future efforts will focus on adapting the biocybernetic controller beyond games and toys to functional information systems. The capacity to operate interactive educational multimedia systems will open a whole new area where human expressivity can be optimized in applications that customize an educational environment to the capabilities of an individual.
Cybernetic Hedonism
The other focus of our efforts is in developing highly interactive, biocybernetic systems where biological signals can modify an environmental chambers' parameters allowing the user to bioelectrically interface with spatialized environments. We believe that such physiologically modulated environmental systems may have a health preserving function. Interfaces to control stimulation can adaptivly utilize any biosignal. The result is the capacity to create a stimulus regime that accelerates relaxation and facilitates stress reduction. This is an application of wellness maintenance technology. "The Nirvana Express"
The Microscope of the Mind
The goal is to extend these environmental control systems into new methods of investigative research. Such as a test of basic cognitive functionality or the capacity to maintain attentional focus necessary to complete an iterative series of cognitive tasks. Data fusion of sensor data with user interaction parameters will allow meaningful correlation's to be made across various performance modalities. A goal of this application is to seek to identify a qualitative difference between the two performance/behavior states and then investigate various methods of quantifying that difference in a way that can be generalized.
It is postulated a difference will be seen in the modulation of some of the natural rhythms. It is also postulated that a cognitively induced modification would be consistent in an individual but would most likely be different between individuals. The psycho-social-behavioral nature of individuals factors into initial assessment of their cognitive function. Other indicators of cognitive function are short-intermediate-long term memory, sound judgment and the ability to identify similarities in related objects. Performance of these cognitive functions is a strong indicator of the biologic health of the brain. Poor performance is highly correlated with organic brain dysfunction.
The potential of this new paradigm of biocybernetics is limited only by the imagination (and funding) of the users.
(Special thanks to Dave Gilsdorf and Patrick Keller for their ongoing efforts in making the world a better place)
3. THE
NEUROREHABILITATION WORKSTATION:
A Clinical Application of Machine-Resident
Intelligence
Dave Warner, Jeff
Sale, Stephen Price, Doug Will
Human Performance Institute
Loma Linda University Medical Center (1993)
Abstract
The Neurorehabilitation Workstation is described. The need to maintain a clinical perspective motivates the comprehensive nature of the system, which integrates multiple data acquisition devices, interface technologies, advanced analytical techniques, and multi-sensory rendering capabilities. Emphasis is placed on machine-resident intelligence embedded at several levels.
Introduction
The field of Rehabilitation applies techniques and resources from many disciplines and is constantly seeking to improve the measurement of human performance and the assessment of therapeutic efficacy. We have had considerable success recently in our attempts to transfer new technologies into the clinical setting for such purposes. Devices such as gloves to measure hand motion dynamics, surface EOG and EMG sensors for eye movement and muscle contraction, and lightweight pressure sensor arrays for gait analysis show great promise in therapy. At the same time, our efforts to make these transfers permanent have been impeded by the lack of standard platforms, interfaces, inaccessible file formats, as well as the medical community's lack of time, technical expertise, and adequate budgets. Until now no cost-effective solution appeared possible. Recent developments in human-computer interface hardware and software, data analysis, and expert systems suggest this is no longer the case. We are currently exploring a solution, the Neurorehabilitation Workstation (NRW), which integrates these technologies and methods into a comprehensive system designed specifically for the clinic. In addition, we hope it may be generic enough to act as a standard for other similar applications. The success of the NRW depends on four things; modular design (for distributed processing and adaptability), integration of several data input devices into a single platform within a common interface protocol, implementation of machine-resident intelligence (neural nets, fuzzy logic) on several levels, and creation of a development environment driven by clinical needs. We detail aspects of these features below.
Data Input
A necessary feature of the NRW is the integration of a variety of data input devices into a single system to include EEG, EMG, EOG, ECG, dynamic bend sensors, pressure sensors, audio and video digitizers, etc. The resulting capacity for data fusion allows for meaningful correlations to be made across various performance modalities. The devices and their hardware boards connect to an external module, and a high speed bus will route the data both to a central multi-tasking server and to the rendering subsystem for immediate feedback. The server should be intelligent enough to automatically implement a custom configuration of input device parameters, interface functionality, and relevant records based on the device(s) connected and the identity of the operator(s) and patient(s) currently at the system.
Data Management
The maintenance of medical record integrity is a significant issue. Such integrity is achieved through security protocols, standardized data formats, error handling, and semi-automated database archiving. The data management subsystem tasks also include linking the device data with the patient record and specifying sensor-specific data formats and structures.
Interactive Modalities/Methodologies
The user interface will be based on new theories of human-computer interaction methodologies , computer-supported cooperative work, knowledge engineering, expert systems, and adaptive task analysis The system will monitor a user's actions, learn from them, and adapt by varying aspects of the system's configuration to optimize performance. Adaptable on-line knowledge-based help using text, graphics, and animated tutorials provide interactive learning and navigation.
Data Analysis
Effective therapeutic intervention relies on a comparative evaluation of a patient's progressing or digressing state. The nature of the change in this state may often be quite subtle, even imperceptible using traditional techniques. Given that the data acquisition subsystem can detect these changes, the data analysis subsystem is designed to enhance them in ways that may then be rendered to optimize the operator's sensory modalities. Linear and nonlinear multivariate analysis tools will be capable of processing multiple data sets in a variety of ways, including graphical analysis (phase portraits, compressed arrays, recurrence maps, etc.) and sound editing (mixing, filtering). Automated detection of trends and correlations using fuzzy logic may be performed in the background or in a post-processing mode. The user may then be alerted by the system if it detects areas worthy of further investigation. Such a feature should expedite the creation of a taxonomy of lesion-specific impairments.
User Classification
We have defined five types of users. These types help define discrete levels of user functionality. Therapists, the primary users of the system, are responsible for data acquisition, data management, basic analysis, and patient-oriented interactive biofeedback modes. Technicians are responsible for simple data acquisition. Physicians will use the more comprehensive data analysis tools. Researchers will focus on the data analysis but their use of the system will be unconstrained. They will explore and develop custom analytical techniques. Patients will primarily use the therapeutic biofeedback features of the system, usually in a supervised setting.
Data Rendering Modalities
With multi-sensor data acquisition and advanced analytical characterization, the rendering capacity of the system becomes extremely vital. The NRW will implement multi-sensory rendering by combining recently developed 3D sound and tactile feedback systems with advanced visualization technologies. Research in human sensory physiology has shown the eye to be optimized for feature extraction of spatially-rendered data, the ear for temporally-rendered data, and the tactile sense for textures [9]. Thus the NRW will enhance perception of complex relationships by integrating visual, binaural, and tactile modalities. The rendering subsystem has a near real-time biofeedback mode for use in a therapeutic paradigm and a data perceptualization mode for use in an analytical paradigm. Outputs from sensing devices and analytical operations are parsed and routed to the combination of rendering modalities best suited to render that information.
Conclusion
The goals of the NRW are twofold; 1) to provide an open hardware platform and modular infrastructure which will expedite the implementation of new technologies into the clinic, 2) to augment clinical therapy with new methods of interaction and analysis. Success should result in providing neurorehabilitation, and the medical community in general, with a powerful tool for characterizing the complex nature of normal and impaired human performance.