FIRST PRINCIPLES OF PHYSIO-INFORMATIC SYSTEMS

INTRODUCTION

Physio-informatics is a new systems model for linking human physiologic systems to information systems in the most general way. This general systems model has been derived through an ever evolving series of experiments and explorations.

Physio-informatics is defined as a systems based, physiologically robust reference architecture for designing and refining interactive human-computer interface systems in ways that increase operational throughput of information. The term physio-informatics will be used in this dissertation to denote informatic systems that are either biologically or physiologically based (primarily neurologic, i.e., neuro-informatic) information systems, and/or informatic systems that are designed to support interaction (dynamic exchange of information) with such systems.

The intent of this work is to develop a systems based, physiologically robust, reference architecture for designing and refining interactive human-computer interface systems in ways which increase operational throughput of information. Extending the perceptual dimensionality of information presented to the human and enhancing the expressional capacity of the human to convey intent to the informatic system achieves this increased throughput.

The conceptual, theoretical and analytical basis for establishing a general systems-based reference architecture for physio-informatic systems necessarily crosses many disciplines. It must be emphasized from the onset that the following discussion of the derivation and development of a general model (AKA reference architecture) for describing "meaningful" information flow between humans and informatic systems is a broad topic area that covers many scientific disciplines, engineering techniques and a continually expanding array of technologies. These technologies include but are not limited to physiology, physics, mathematics, philosophy, general systems, bio-cybernetics systems, cognitive neuroscience, perceptual psycho-physics, perceptual state space modulation, bio-sensors, quantitative human performance, expressional interface systems, physio-informatics, intelligent interface-metrics, user tracking interface systems, distributed tele-robotic controllers and intermental networking.

To take a general perspective, this effort is an attempt at integrating the areas of human scientific endeavor mentioned above in a manner that will not require that future researchers in physio-informatics master all of them before they can contribute meaningfully to the process of optimizing the coupling between humans and informatic systems in an interactive interface system. Thus, the intent of this effort is to establish a general conceptual framework (a reference architecture) which can be used as a guiding heuristic tool when confronted with the challenge of designing and developing interactive interface systems for human computer interaction, specifically ones that extend perceptual dimensionality and facilitate enhanced expressivity.

Interactive Human-Computer Interface Systems

In the various traditional models of human computer interaction it is customary to think in terms of inputs and outputs; input from the computer to the human and output from the human to the computer or input from the human to the computer and output of the computer to the human. This dissertation will develop a systems model for interactive human-computer interface systems that is more representative of reality than traditional models in that it is consistent with the phenomenological aspects. This more accurate systems model is a physiologically based reference architecture for designing and developing interactive human computer interface systems to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information.

THE PHYSIOLOGIC BASIS OF A REFERENCE ARCHITECTURE FOR DESIGNING INTERACTIVE HUMAN-COMPUTER INTERFACE SYSTEMS

Context and Initial Motivation

The capacity of computers to receive, process, and transmit massive amounts of information is continually increasing. Attempts to develop new human-computer interface technologies have given us devices such as gloves, motion trackers, 3-D sound and graphics. Such devices greatly enhance our ability to interact with an increasing flow of information. Interactive interface technologies currently emerging include devices for directly sensing bio-electric signals (from eye, muscle and brain activity) as inputs and rendering information in ways that take advantage of the psycho-physiologic signal processing of the human nervous system (perceptual psychophysics). The next steps in human-computer interface technology will tailor the technology to the physiology, producing a more biologically responsive interactive interface.

Interactive Information Technology

Interactive information technology is any technology that augments our ability to create, express, retrieve, analyze, process, communicate, and/or experience information in an interactive mode. Biocybernetics optimizes the interactive interface, promising a technology that can profoundly improve the quality of life of real people today. The next paradigm of interface technology is based on new theories of human-computer interaction, which are physiologically and cognitively oriented. This emerging paradigm of human computer interaction incorporates multi-sensory rendering technologies giving sustained perceptual effects, and natural user interface devices which simultaneously measure multiple physiological parameters and use them as inputs. Biologically optimized interactive information technology has the potential to better facilitate effective communication. This increase in effectiveness will improve communication between humans and computers as well as assisting human to human communication ("enhanced expressivity").

Biocybernetic Controller

Interactive interface technology renders content-specific information onto multiple human sensory systems, giving a sustained perceptual effect, while monitoring human response in the form of physiometric gestures, speech, eye movements, and various other inputs. Such quantitative measurement of activity during purposeful tasks allows us to quantitatively characterize individual cognitive styles. This capability promises to be a powerful tool for characterizing the complex nature of normal and impaired human performance. The systems of the future will monitor a user's actions, learn from them, and adapt by varying aspects of the system's configuration to optimize performance. By immersion of external senses and iterative interaction with biosignal triggered events, complex tasks are more readily achieved. This paradigm shift from mass communication and information technologies is providing an exciting opportunity to facilitate the rapid exchange of relevant information, thereby increasing the individual productivity of persons involved in the information industry. Areas such as computer-supported cooperative work, knowledge engineering, expert systems, interactive attentional training, and adaptive task analysis will be changed fundamentally by this increase in informatic ability.

The psycho-social implications of this technologically mediated human-computer and human-human communication are quite profound. If people are provided with the knowledge and technology to empower them to make a positive difference, this could foster the promotion of an attitude of social responsibility starting with the use of this technology, and may be a profound step forward in modern social development. Applications that can obviously improve quality of life, such as applications in medicine; education, recreation and communication would become a personal, and hopefully a social, priority.

FIRST PRINCIPLES OF PHYSIO-INFORMATIC SYSTEMS

HISTORY

An overview of the field of interactive human-computer interface systems

From the earliest electronic computing machines until the late 70's, the ability of the computer to respond to a task given it by a human was, for the most part, limited by computational power. Back when each logical connection was "hard wired" by an army of technicians until the time of punch cards, the concept of interaction with a computer had a very limited and specific interpretation. A great advance was made when the computer had a typewriter-like mechanism that allowed computers to be programmed through "terminals". As the speed of the computer increased along with its capacity to respond iteratively to the human, it began to become apparent that the humans' ability to convey intent to the computer and the computer's capacity to display results would become a limiting factor in the interaction between humans and computers. Innovators at Xerox PARC, MIT, NASA, DARPA and other computer research facilities began to rethink the concept of how humans and computers would be able to communicate more effectively. The first significant breakthrough to make it out of the lab was the "WIMP" (Windows, Icons, Mouse, Pointer) graphical user interface, also referred to as the "GUI" (pronounced gooey). In the late 70's with the commercial release of Apple's personal computer, with its GUI, to the general public, the interaction with computers began to change to a more human-centered design.

In the mid 80's computer systems used by industry were becoming fast enough, and display technology and was becoming sophisticated enough, to be able to "render" (calculate and display) graphic images for engineers in computer aided drafting and design jobs, who were then able to manipulate these rendered images with ever increasing speed and resolution. At the same time new techniques were being developed by scientists to enable them to "visualize" (display) a graphic image that was the result of a very complex set of calculations. These new areas of "CAD/CAM" (computer aided design and computer aided manufacture) and scientific visualization continued to evolve with faster and faster computation and ever increasing quality of graphic images.

In the late 80's it was recognized that the compute power and graphic display techniques presented new opportunities to develop interactive technologies. While it is true that many advances were made with human factors in "man-machine" interfaces throughout the late 70's and the 80's, this work dealt with the physicality of the environment and information displays, and much of that work was done in the context of very specialized tasks. Specific kinds of work such as piloting fighter airplanes or space craft, controlling complex industrial processes such as nuclear power plants, or complex chemical processing plants were well studied and refined; however, these tasks differed from interactive human computer systems in that the humans were controlling some machine or physical system and were not primarily interacting with information as represented by computer systems. One exception to this would be the interaction with a computer simulation of a complex physical system. The primary efforts for researching and refining these systems were in the field of ergonomics, dealing with the energetics of the human interacting with their environment, and in cognitive science, the mental computation required to perform effectively in the environment.

Also in the late 80's a new concept began to take hold in the field of human computer interaction -- the idea of "immersive systems", where the computer systems began to encompass the humans senses and track the movements and position of the human in an effort to develop a synthetic environment within which the human could interact in a more natural way. These "virtual reality" systems sparked a brief but important revolution in the thinking and machinery of human computer interaction. From an evolutionary neuro-information processing perspective, this technology created a new potentiality for response to perceptual awareness: it canalizes not a single response to a single stimulus, but rather multiple responses to multiple stimuli born of a single though multi-dimensional sensorial perceptual state. The combination of these different rendering modalities correlates with somatotopic placement to achieve and demonstrate spatial coding of the rendered information. Optimizing the human-computer interface will rely on the knowledge base of physiology and neuroscience; that is, the more we know about the way we acquire information physiologically the more we know the optimum way for a human to interact with intelligent information systems. The next paradigm will see the "thinning" of the human-computer interface to a biological "sheer" as the interface will map very closely to the human body.

Physiologically Oriented Interface Design

Knowledge of sensory physiology and perceptual psychophysics is now being used to optimize our future interactions with the computer. By increasing the number and variation of simultaneous sensory inputs, we can make the body an integral part of the information system, "a sensorial combinetric integrator". We can then identify the optimal perceptual state space parameters in which information can best be rendered, that is, what types of information are best rendered to each specific sense modality. Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile modalities. Then by using the natural bioelectric energy as a signal source for input -- electroencephalography, electroocculography, and electromyography (brain, eye and muscle) -- we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multi-modal feedback and closed-loop interactions.

The following discussion is concerned with developing a "reference architecture" (a formalized conceptual framework for thinking) for designing physiologically robust interactive human computer interface systems. The purpose of the reference architecture will be to provide insight into the various components of the system in the context of how they might affect the flow of information as information is passed through them. The primary focus will be to consider the flow of information between the human and the computer in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during the intentional interaction of a conscious human and a computer system. An exchange of information between an experienced perceptual state and an external physical state is mediated by a biologic or physiologic information transporter system.

The biological system can be described accurately as a multi modal, multi scale, concurrent, hetero-purpose, poly-dyno-morphic simul-tasking information processor.

FIRST PRINCIPLES OF PHYSIO-INFORMATIC SYSTEMS

THEORY

For this discussion we will assume that:

-- Interface systems which support human-computer interaction can be modeled as systems where information flows between various components in a specific manner.

-- Information can be mapped and represented as a specific state space parameter set.

-- The mind exists as the super set of all perceptual states.

-- The anthroscopic scale, the natural scale of perceptibility and expressivity of an individual human, is from meters to millimeters, from decades to deci-seconds.

-- In the universe of discourse, mind is a phenomenon occurring at an anthroscopic scale.

-- The phenomena of interest, (perception and expression), occur at the anthroscopic scale.

Additional Assumptions

-- Time is perceived as a unidirectional vector.

-- The nervous system is the primary information infrastructure for humans.

-- The nervous system supports the transduction transmission representation and response to information in the environment.

-- Human perception and expression are mediated, for the most part, by the nervous system.

Hypothesis

An understanding of human neuro-physiology allows for the exploitation of predictable adaptive capabilities. The assertion is that the information flow between external sources and direct experience is biased, restrained, constrained, limited, enhanced and/or facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.

Physioinfometrics

Physioinfometrics is defined as the quantitative measure of the information carrying capacity of a physiologic system.

Physiologically mediated information is exchanged between the external environment and experiential awareness. The fundamental nature of the nervous system (neuro info-matrix) determines its operational capacity. Both physicality and physiology contribute to the set of bio-physical restraints. The physicality of the nervous system constrains the perception of space, time, mass and energy. The physiology of the human nervous system restrains perception by the computational limits of the system. The complexity, functionality and capacity of the intra-activity of the nervous system sustain perception. Therefore, the form and function of the nervous system influence various parameters of perception and expression. The nervous system is the biologic structure considered most likely to be responsible for mediating information flow within the human body.

The basic ideas leading to the primary foundations for this thinking can be seen as coming from the following areas.

Operational Philosophy

We operate through action directed goals in the pursuit of new knowledge, starting with logical analysis of observed phenomena and proceeding to the point of discerning an operational utility of continuing the pursuit in the current mode of analysis or changing modes to seek a more fruitful mode of investigating the phenomena (Oppenheimer).

In other words, this is a philosophy of scientific investigation that constantly seeks to validate the current mode of analysis for a given set of observed phenomena to maintain constant progress in the discovery process of new knowledge.

General Systems Theory

General systems theory is a useful framework for developing complex models for investigating complex systems, like those of physio-informatics. Particularly useful are concepts of systems models and principles such as hierarchical order, progressive differentiation and feedback that can be defined, characterized and elaborated on with set and graph theory, and which state explicit conditions for membership and orders of relationship.

The "open systems" approach to a general systems theory by Von Bertalanffy in the late 1930's was instigated by a perceived need to break out of the "closed systems" model that implicitly separates the system from its environment, as it was leading to incorrect conclusions. His idea was that biological systems necessarily must be considered open systems, where both information and energy are in continuous flow between the system and the environment. His initial formulation of a general system was an attempt to derive principles that were valid for open systems.

A system can be defined as an object consisting of a set of complex objects or relationships, each of which are in some way associated with other objects within the system in a way that some quantities within those objects are associated with quantities of other objects within the same system (Von Bertalanffy).

Information

The base elements with which information is constructed are "differences". A difference can be interpreted as either an ontological fact or as an abstract matter. Information can therefore be defined as "a difference that makes a difference" (Bateson), or "a difference with a non zero significance" (Warner).

The aspects of information theory concerning transmission effects on information across physical structures are considered to be important in physio-informatic systems, but are tempered by the fact that biological systems do not adhere to the negentropy formulation of Shannon.

Cybernetics

Cybernetics is the theoretical model of feedback governed systems whose present state influences in some way the probabilities of any future state occurring in the system. Operators that are invoked in the system are a result of past or current states. This establishes that there is a relationship between operators and states beyond the "transformational function" of operators on states.

Definitions

A state of any system is defined by the set values that describe the condition of the system in any given point in time (the value of all the state vectors). A system will have a state space that represents and/or contains all possible states of that system.

State Space, States and Operators

For systems whose quantities are in continuous flux a special kind of set called a "state space" can be constructed which has as its elements (set members) an n-tuple of values that are the values of the quantities at a given instant. At any given instant the system is said to have a "state" that is determined by the values of each of the "parameters" at that instant (Ashby, Zadeh).

For a given system whose states are not temporally static within a given state-space it can be asserted that a transformational function has been performed on the system, determining the next state the system will be in. Such a transformational function is called an operator. Thus it is correct to say that an operator acts on an initial state parameter value and produces a new state parameter value.

In an open system it can be asserted that the evolution of the states in time i.e. the "state space trajectory" is influenced by both the current state of the system (internal factors) and the processes of the environment (external factors) which are acting on the system.

States Operators and Information State Spaces

A state space of a physio-informatic system can be described as a set of information-based states which behave in a particular manner. A useful mathematical modeling system for mapping the flow of information in any physio-informatic system has been established --the "Expression Space Notational System". The following discussion is concerned with the development and elaboration of Neurocosmology and the Expression Space Notational System.

Please keep in mind that the expression space is a universe of discourse within which the physical and mental realms are complimentary subspaces.

"NEUROCOSMOLOGY"

To make the initial assumption that all information that is perceived about the external environment is filtered, mediated and/or biased by the nervous system is known as the "Neurocosmological Principle" of epistemology. Neurocosmology operates at an anthroscopic scale and has an anthrocentric perspective.

Neurocosmology incorporates a notational system which enhances the ability to see differences. It has as its prime focus the flux and flow of information between a perceiving human being and the environment.

Some properties of neurocosmology include:

• A superior descriptive process

• A powerful system of methods and models

• A superior conceptual frame work within which to map information flux

• Derivational pathways from first principles to operational systems

• Intrinsic heuristics

Assertion

The nervous system's capacity to transduce, transmit, characterize, experience and respond to information of environmental conditions is the limit of the know-ability of the environment. The anthroscopic neurophysiologic info-matrix supports and provides the basic infrastructure for continuous intentional and willful interaction with the environment. Biologically mediated information exchange couples perception to the environment. The physio-infometrics of the neural info-matrix determines the rate of information flow (through-flux). The exchange of that information can be abstracted; furthermore, the flow of information can be parameterized by temporal, spatial, ergo-dyno-morphic flux . Any state of information is characterized by a specific state-space parameter set.

Theoretical construct

A descriptive mathematical model that can map the transformation of information as it is exchanged between various components of the interactive human-computer interface system is presented below. This model, which generalizes phenomenological aspects, has a notational system which exploits interdisciplinary interaction and a languaging system which can classify emerging observations.

Neurocosmology can be described as an anthroscopic epistemology which is biased by neural systems. Neurocosmology is a member of the expression space and you define it with perceptual parameters. A notational system has been derived which represents the flow of information between the environment and the neurally mediated experience of consciousness. This formal descriptive notational system will enable the creation of "most probable maps" of information flow between humans and their environment.

The whole universe of discourse will be characterized by the term "E-space" (expression space). To discuss the exchange of information in the context of neurocosmology it is necessary to be in the E-space universe.

A set can be defined --Expression space is characterizable.

Definition: ANY expressible information is in Expression space

An expression of any experiencable information is a member of E-space

ANY state of information which is experienced and expressed.

The minimum element of resolution is the existence/experience of a difference.

To have a non-zero information content a noticeable difference with non-zero significance needs to occur.

The minimum element of information is a significant difference.

IT is a difference between ANY 1 and ANY other

IT is the information trajectory which relates ANY state of information

IT is an expression space state path

It should be noted that the following formalisms will be loosely followed: 1) Set theory, where membership to a given set is determined by some formal means; and 2) Graph theory, where the position and relationships are determined by intrinsic values. Given the dynamic nature of physio-informatic systems, a more general system of states and operators will be pervasive.

In the course of systematic reflection upon those fundamental essences, which appear to be ontologically distinct primary percepts, one may construct a sound and verifiable epistemology with a triad of categorical types. That is, if one is to distinguish the entirety of perceptible reality as perceived into categories which are characterized phenomenologically, rather than by the material constituents with which they are constructed, one finds that there are three principle realms which may act in concert and in various combinations to account for the whole. These three categories are Experiential ("mind"), Biophysical ("life") and the Physical ("matter") (Figure 1).

Figure 1. Three principle realms of the E-space

Physical (matter)

Biophysical (life)

Consider a complex system that has as its primary components three fundamentally distinct sets of parameter values, or three "state spaces". A state space is a set whose members are defined by an n-tuple of values which correspond to the parameter values of the system. We may define one set as a set of all parameter values of a computational system. Another set is a set of directly measurable parameter values of a physiologic system. The third set is a set of parameter values which are directly experienced by a conscious human. Each set is distinguishable from the other in that the computational system and the physiological system are physically separable and the physiological and perceptual state spaces are phenomenological. It is acknowledged that the basis of perceptual experience is most probably supported by a specific set of physiologically distinct systems. It is beyond the scope of this dissertation (and frankly unnecessary) to develop a robust explanation or model for the specifically and/or physically distinguishable aspects of conscious experience (the mind) and assumed neural matrix (the body) that is thought to at least co-occur with those conscious experiences. Suffice it to say that a user in the context of an experiential interaction with information does not routinely confuse the two.

Expression-space (E-space) This is the whole universe of discourse. To discuss the exchange of information in the context of neuro-cosmology it is necessary to be in the E-space universe.

Mind subspace This is the subspace of E-space which contains all the information states directly perceived by the mind. An instance of conscious self perception requires Mind-space. This is the realm of perceived thoughts and ideas. While there is no known limits to the level of complexity which can be obtained by mind-space information space, a certain level of structure is assumed to exist and as such would limit it.

C-term [] C-term is the mind-space operator. C-term is the operation which changes one mind-state to another. There is an implication of conscious attention in a C-term operation. When one instance of conscious perception intentionally and willfully affects the generation of a specific type of mind-state then this is considered a C-term operation

Biological subspace This is the subspace of E-space which contains all the information states which are directly associated with biological processes. This real would include all the bio-physical (physiological) phenomena that is associated with life--sustaining functions and the functions which transpose, convert and or transduce information between the physical and mental realm.

B-term [] B-term is the generic bio-subspace operator. B-term changes information states which are directly linked to the biological / physiologic information processing functions. Any process which is achieved by a living organism is a B-term type function.

Space-time This is the subspace of E-space which contains all the information about the physical universe except that which is directly related to biological processes.

Z-term [] Z term is a generic space-time operator. This is a function which accounts for all the physical (non-biological) changes in information states in the physical subspace. Z-term functions are those which the physicists and other physical scientists study. They are things like electricity, magnetism, gravity, thermal dynamics, etc. In general, anything with just physical properties is a Z-term function.

Now with these terms in mind the next step is to specify the format or syntax of a valid expression in Neurocosmology notation. The general form of a neurocosmology expression is in the form of a commutativity diagram, and it looks something like this:

All instances of information from the physical subspace to mind subspace will be in this form, where PHI is the object as it exists in space-time, and Z-term is the information of the object being transmitted through physical substance via light, sound, smell, etc. PHX is in this instance the composite of all the intermediate biologic information state spaces necessary for information transduction (sensation). This was comprised of stimuli, transmission of sensory output and neurological processing and coding in the brain. B-term is the process of sending the final product of sensation to the mind for the purpose of perception. It is an axiom of neurocosmology that this expression in the form presented above is a necessary and sufficient condition in order for an instance of perceptual experience to have an information content assimilated from an external object that exists in the space-time realm (there is a world external to our senses). All the information we have about the outside world was obtained in this way.

It may be easier to use the following approximation when applying Neurocosmology:

• PSI = the self perceiving mind; perceptual space; contents of awareness.

• PHX = the living body; biological space; what chemistry refers to as "organic" and all things related to it.

• PHI = objective, material, space-time reality; physical; what physics refers to as space, time, mass, energy.

Thus ANY information that can come into being is one of these categories or combinations of the categories.

The following is a list of all the major state-operator-state expressions required to map information flow through the physio-informatic system (Figure 2).

Figure 2. State-Operator-State Expressions for mapping information flow.

Neurocosmology states that perceptual states are basically a result of brain states (Figure 3).

Figure 3.Perceptual states arise from brain states.

However, the state of perception at any given time is specified and categorized as PSI state, because when perceiving a uniform, coherent perception at ANY given time, an instant of perception is all we ever have. Information coming from the outside, causing neural structures to fire on the sensory cortex, is sensation. The brain correlates that sensation with all its other relevant internal functioning and all layers of psychosocial elements and the gestalt of all brain function in consciousness equals perception.

ANY contains a subspace of possible expressions; everything I could possibly perceive is herein contained. Rather than matter and energy we're referring to perceptual state-space. The perceptual state is always related to some biologically parameterized state of the brain. The term for these biologically parameterized states of the brain is PHX, referring to all those entities and processes discussed within the life sciences.

Suppose PSI wishes to perform an action in the outside world. By means of a cognitive operator, PSI initiates the required parameters of PHX via a biological operator, to perform this action: throw a ball, for example. This is a PSI to PHX information transection resulting in a PHI operation: the ball flying. So, information from the PSI subspace transferred information (via these various operators) to the PHX subspace which in turn transferred information to the PHI subspace. The principle which derives from this dynamic is that whenever a PSI subspace has influenced a PHI subspace some PHX subspace has mediated the transfer of information (Figure 4).

Figure 4. PHX mediated information transfer.

Figure 5. KHI: A PSI-PHX-PHI rendered state.

The above figure (Figure 5) is an illustration of KHI; a state of PHI whose form can only be accounted for by positing the willful action of PSI via PHX acting on unaltered PHI.

The above diagram is the biocybernetic loop. It illustrates in a very complete way all the elements of how a human interacts with a computer. If there is a mind, then no matter what the particulars of a disability or restriction, we should be able to find a way to interface that mind with a communications technology.

An earlier representation of this process was illustrated in the following diagram (Figure 6).

Figure 6. Diagram of the cognitive cybernetic feedback loop.

Within the notational system described, the necessary and sufficient conditions for mapping information flow through the physio-informatic system is covered by the following:

• A Mind (symbol: PSI) as the beginning of all expression and the end of all perception; the foundation of the whole process.

• An Organism/Life (symbol: PHX) system facilitating that perception and expression via sensory and motor actions biological/physiological factors).

• An Interface system which is a two-fold element. An example of a very high level technology which applies this thinking is "Bot Masters" human-robotics interface system (www.pulsar.org)

• Expressional system (symbol: SNS) by which some mechanism senses and/or captures some motor action as input from the human (e.g., a mouse click, facial movement).

• Perception system (symbol: SKY) by which some mechanism renders the results of the expression back to the human (e.g., screen, audio screen reader for the blind, etc.)

• SKY: that product of KHI having been manipulated by PSI so as to change PSI.

• Synectic notational languaging structure in which we have a generic metaphoric ability to talk about anything; its goal is to is to allow you to linguistically navigate to any far reaching PSI space that you wish to explore.

Now that we have come so far, we can move into a higher state of abstraction connecting more information.

For example, sensory physiology is talking about the PHX specific state spaces which are involved in transducing and transmitting information to the central nervous system, which is part of the PHX system. The following three fundamental state spaces are required to describe the complete system:

• Information that is described by the state of an external computational system.

• Information that is described by a the state of a physiologic system.

• Information that is described by the state of perceptible components of a conscious experience.

Each of these systems can be considered to be a fundamental source of information. It is also presumed that there is an ordered flow of information between these three fundamental state spaces. As information is exchanged between these state spaces there are direction-specific boundary-crossing transfer functions which restrict, bias, interfere and/or otherwise constrain and restrain the functions' capacity and fidelity. From the perspective of the physiologic system there is information exchange with a persistent external information system and an emergent internal information system. Therefore it is considered a basic tenant for this model that the body (comprised of all its information processing physiologic systems) mediates the exchange of information between the computer and conscious experience. This at first glance may seem somewhat obvious to the casual observer. However, it is this physiologic mediation of information (with all of its specific processes) which is the basis for the development of this reference architecture for developing and optimizing interactive human computer interface systems.

Simply put, the three coupled state spaces of the interactive human computer interface system being proposed may be described (from the perspective of a human user) as being comprised of an external computational system, the imperceptible physiologic processes which mediate the exchange of information with that system, and conscious experience and the perceptual qualities of experienced information:

• Information, which is externally generated -- PHI (Physics)

• Information, which is biologically/physiologically mediated -- PHX (Life)

• Information, which is directly experienced -- PSI (Mind ; perception and intention of expression)

The following discussion covers efforts to demonstrate Neurocosmology's capacity to increase perceptual dimensionality.

In addition to introducing new models of mapping information flow, Neurocosmology identified new mathematical models and representation systems that would give insight to complex phenomena. These new graphical methods of mathematical analysis bring fresh insight into biological processes.

The following images represent the nature of this "new math":

Fractal geometry (geometry of fractional or non-integer dimensions) was useful in helping to analyze complex systems and their states. Of particular interest was the work of David J. Warner, Steven H. Price, and S. Jeffrey Sale, which begin to explore the use of the deterministic iterative model of mathematical analysis to gain insight on what happens when you alter the various parameters of the base expression to be iterated. What was discovered was that one could see complex yet identifiable changes in the rendered geometry, which demonstrated the basic premise of fractal geometry that initial conditions of seed values and the basic algorithms led to comprehensible maps.

Figure 7. Fractal images.

Early work by Price, Sale and Warner showed that the basic fractals, which were common in the popular media and scientific literature (Figure 7), were useful tools for gaining insight into the nature of iterative systems. These images show that a simple change of the basic algorithm can led to very ordered systems which are predicted by the change in the base formula. The basic fractal formula is z^2+c, but a progression to z^n +c where n is 0-4 shows that the number of symmetries varies as n-1 but the individual patterns are fairly constant (that is, they look to be of the same class).

Figure 8. Fractal representation of iterative systems.

The above images (Figure 8) also demonstrate the nature of iterative, deterministic functions as they also show that a difference can be characterized both visually and quantitatively.

The basic concept distilled from this observation and exploration was that for a given initial condition a unique result would occur and that this method could be used to begin to quantitatively characterize the specific complex dynamics which were intrinsic in the physiological data. When this new tool was applied to physiology, we began to gain a higher level of insight to the complex state dynamics of physiological systems. These complex structures are important for the development of a robust physiological system in which states of information can be inferred from their unique dynamics, and which state changes could be measured by various instruments. To begin to move forward in the development of a theoretical model of physio-informatics it was necessary to demonstrate as true the assumption that one can uniquely characterize a measured physiological state space dynamic and to link it to a specific information condition of a physiological system. To establish this relationship between the geometric, the quantitative and the physiologic the following experiment was done.

FIRST PRINCIPLES OF PHYSIO-INFORMATIC SYSTEMS

METHODS AND DISCUSSION

The following discussions show early explorations into the physio-informatic methods discussed above. First, these new mathematical insights were applied to the EEG which is a very complex system.

1. Chaotropic Dynamical Analysis of the EEG

The mechanisms generating normal and abnormal rhythms in the brain are poorly understood but are usually attributed to a combination of sinusoidal oscillations and stochastic noise. Quantitative analysis of the EEG has emphasized application of the FET and statistical analysis of the resulting power spectra. It is now possible to perform chaotropic dynamical analysis of the EEG. Phase portraits are obtained by imbedding the EEG time series in a multidimensional space using various time lags. Phase portraits can be rendered graphically in 2 and 4 dimensions. Lyapanov exponents, fractal dimensions and Poincare sections can also be obtained. Inspection of phase portraits during 3/sec spike and wave suggests the presence of a low dimensional state space (Figure 9). EEG during normal eyes open condition suggests the presence of a high dimensional state space. In deterministic systems, low dimensional state spaces have low information content and limited response capability. High dimensional state spaces have rich response repertoire (Figure 10). Chaotropic dynamical analysis of the EEG thus provides a powerful theoretical structure within which to interpret normal and abnormal findings. Chaotropic dynamical analysis is an important new approach to quantitative investigation of electro physiologic measures such as EEG and MEG.

Figure 9. Generated Phase Portraits used for Chaotropic dynamical analysis of the EEG.

Figure 10. Comparisons of state space dimensionality and corresponding information content.

The Compressed Dimensional Array: a new topographic technique for EEG analysis.

The mechanisms generating the EEG are poorly understood but are thought to involve non-linear deterministic dynamics. The complexity parameter is an important measure of dimensionality but displays usually do not permit parameter comparisons between many EEG channels over time. The complexity parameter is obtained by embedding the time series in progressively higher dimensions until a scaling property emerges. This dimension is then selected for the Compressed Dimensional Array (CDA). The complexity parameter is then calculated for two second epochs and arrayed in a single display on an graphics workstation so as to appear as a topographic contour in which elevation represents the complexity parameter. This permits a real-time interaction with this array and is a convenient method of dimensional analysis. Areas of low dimensionality appear as easily recognized valleys. The CDA provides a new method of visualizing dimensionality of the EEG and reveals subtle features of clinical and scientific interest.

The use of these new graphical techniques, which enabled the researchers to gain a greater degree of insight into the phenomena, was also applied to the problem of extending the perceptual dimensionality of research data. Initially, EEG data was used to test the utility of this exploratory mode; however, it was not long before other data sets were evaluated with these methods.

The following series of images shows the progression from the standard methods of analysis to a visually rich technique (it is asserted that this is consistent with the reference architecture being developed in this thesis). The fundamental idea here is to use humans' natural ability to perceive differences in space, color, and structure to help elucidate the various features embedded in the data.

Traditional EEG data is of this nature:

From the images below it can be seen that this new "spatial temporal" isosurface technique enables the detection of structural components in data that was previously seen as not having any intrinsic structure:

This technique, developed for exploring and characterizing the electro physiological data, can be shown to be of value for displaying the data in a way in which the various different modes of EEG seen clinically can be easily classified. The images below illustrate that point in that they show various conditions seen clinically, all in a very "perceptible" form (all images show evoked potentials, anesthesia states, eyes open relaxed normal).

As mentioned above, this technique which was initially applied to the EEG was extended to the ECG. It is interesting to note about the nature of both these methods is that they cause a new mode of thinking and a series of questions about the former formal methods (for example, where electrodes should be placed for best results).

It was necessary to establish that the method being used was providing meaningful and reproducible results. Having established the relationship between the dynamics of a physiological system and a quantitative and graphical representation system, the next effort was to gain an understanding of more specific (experimentally controlled) dynamic physiologically mediated information. One of the purposes of this level of effort is to establish a relationship between the physio-dynamics (as measured by various instrumentation) and the meaningful, which can be utilized in both a research/exploratory setting and also for a finer grain on clinical assessment. In the effort to develop a series of meaningful representations, it became necessary to develop a new method of representation which was able to convey complex dynamic transitions which varied in both space and time.

2. The VPL Data Glove® as an Instrument for

Quantitative Motion Analysis

Having established dynamical analysis methods of measuring direct bioelectric signals, research effort was then turned to implementing the array of new interface devices which were being developed for virtual reality. Several quantitative human performance assessment tools for clinical, educational and vocational applications have been developed and refined.

Movement related potentials (MRPs) are useful in the investigation of motor physiology. Measurement of MRPs requires accurate determination of movement onset. New techniques have been developed for precise quantitative measurement of complex movements of individual fingers and the hand. Joint position, angulations, movement onset, acceleration, and velocity can all be obtained through the use of the VPL DataGlove®. The closely fitting lightweight Lycra glove does not restrict movement, change biomechanics or alter moments. Fiber optic sensors on the dorsum of the hand and each digit dynamically measure angulations (with 1 degree resolution) of p joints, the PIP joint of the thumb and the PIP joints of the four fingers. The position of the hand in three dimensions is measured using a Polhemus tracking system with 3.3 mm accuracy for x, y, and z coordinates and 0.85 degree accuracy for pitch, yaw, and roll. Data from each of the sensors is sampled at 60 cps, and can be rendered graphically in real time or stored in a file. Movement onset, acceleration, velocity, and amplitude can be displayed, and complex relationships between joints can be studied during arbitrary motor tasks. Data from healthy individuals during a range of motor tasks will be demonstrated. Precise quantitative measurement of hand and finger movement will be an important contribution to neurophysiologic studies of motor disorders.

3. The VPL DataGlove® as a Tool for Rehabilitation and

Communication

Rehabilitation of hand function following a stroke may be enhanced through the use of biofeedback. The VPL DataGlove® provides a rich biofeedback environment permitting the patient to interact with an anatomically accurate computer graphics representation of the hand. The glove is made of comfortable light-weight Lycra and is easily pulled onto the hand. Fiber optic sensors on the dorsum of the glove precisely measure hand and finger position. The glove is not restrictive and does not interfere with movement. Data from each of the sensors is rendered graphically in real time on a Macintosh computer screen, giving the patient visual feedback. The data can also be saved for quantitative assessment of improvement. This allows the patient to set goals and measure progress. Tasks can be customized for each patient's disability and changed to enhance patient interest and effort. The glove has gesture-to-speech capabilities, permitting patients with hearing or speech impairment to communicate through computer generated phonemic speech. The glove may also be used as a computer interface permitting disabled individuals to control a wide range of external devices. Use of the glove in occupational therapy and gesture-to-speech will be demonstrated.

4. The DataGlove® for Precise Quantitative Measurement

of Upper Motor Neuron (UMN) Function in

Amyotrophic Lateral Sclerosis

The response to treatment or determination of progression in diseases such as ALS. The Medical Research Council's clinical rating scale measures only strength, is insensitive to early changes and is ordinal. The Appel scale and Tufts Quantitative Neurologic Evaluation (TQNE) scale measure isometric strength, but inadequately assess UMN function. UMN function is best assessed by measurement of dexterity.

The VPL DataGlove® can precisely measure joint position, angulation, acceleration and velocity, permitting quantitation of motion of the digits, hand, wrist, elbow, and shoulder. The light-weight closely fitting Lycra DataGlove® does not restrict movement or change biomechanics. Fiber optic sensors dynamically measure angulation (with 1 degree resolution) of the hand and finger joints with high precision and in three dimensions, the position of the hand is measured 60 times per second and can be rendered graphically in real time or stored for later analysis. Movement onset, acceleration, velocity, and amplitude can be displayed. Data from healthy individuals and ALS patients will be demonstrated. Precise quantitative measurement of movement will be an important contribution to clinical trials in ALS.

5. Quantitative Analysis of Tremor and Chorea Using the

VPL DataGlove®

Analysis of chorea, myoclonus and tremor is often limited to direct observation or videotape recording. New techniques have been developed for precise quantitative measurement of finger and hand movement using the VPL DataGlove®. The comfortable light-weight Lycra glove is easily slipped onto the patient's hand and does not restrict movement, or change of mechanics. Fiber optic sensors on the dorsum of the hand and each digit dynamically measure angulations (with 1 degree resolution) of MP joints, the P joint of the thumb and the PIP joints of the four fingers. The position of the hand in three dimensions is measured using a Polhemus tracking system with 3.3 mm accuracy for x, y, and z coordinates and 0.85 degree accuracy for pitch yaw, and roll. Data from each of the sensors is sampled at 60 cps, and may be rendered graphically in real time or stored in a file. Movement onset, acceleration, velocity, and amplitude can be measured. Frequency profiles for tremor analysis can be obtained using the fast Fourier transformation. Complex relationships between joints during kinetic movements and tremor can be studied. Data from patients with various types of chorea and tremor will be presented. Precise quantitative measurement of movement will be an important contribution to assessment of tremor and chorea.

6. General Utility of Quantitative Assessment Using the

VPL DataGlove®

The validation of the DataGlove® (along with EMG) as a tool for quantitative assessment for Parkinson's tremor was done by the Department of Neurology, Loma Linda University, California in 1991. The following year the same technique was utilized to validate a new treatment modality elsewhere at Loma Linda. The glove was used by the Neurosurgery Department to validate the human performance of patients undergoing an experimental treatment which developed an intentional lesion in the brain of an awake patient. The patient was required to perform a series of motor tasks, and their performance was then used to refine the treatment in real time. The following images show that this method of quantitative analysis was of sufficient utility to be utilized in an invasive procedure in real time.

The data at the center shows the activity of forearm muscles in a flexion-extension task. The data clearly shows that Motor activity is more "normal" in the later tracing (i.e., after the lesion).

7. The VPL DataGlove® in the ICU

A patient from Neurology was admitted to the ICU with a stroke which left them mostly paralyzed. The DataGlove® was used as a tool to establish a communication channel which could reliably convey the intent of the individual

8. The VPL DataGlove® in the Case of Coma

A young girl who had a traumatic brain injury was emerging from a coma. There was a need to establish her cognitive level so as to optimize her therapy. As with the stroke case above, a glove was used as a communication tool which could be reproducibly manipulated by the girl to show that she was in fact aware and processing information. The interesting thing about this experiment was that it was the first experiment which established the use of a feedback system; the girl had to listen to the output of her actions to ascertain whether or not she had achieved a goal/target state.

9. The VPL DataGlove® for Objective Measurement

Having repeatedly demonstrated the utility of using emerging interface technology for the evaluation of clinical conditions, the next step was to validate the methodology by comparing it to the current "subjective" methods used by clinicians. A study which helped to establish the utility of these new transducers was that of a neurosurgeon developing an new experimental technique. He wished to determine via objective measurement the efficacy of his intervention. This new technology was able to help him evaluate his new clinical technique more objectively and to record his measurements.

10. Technology Demonstrations

The above cases are an illustration of the ability of the physio-informatics system to have a wide range of utility. As the testing expanded the complexity of the instrumentation was also increasing. We felt that we were not going to be able to explore all the medical and rehabilitative possibilities, and that we needed to get the word out so that more people could be using these technologies. We also wanted to find out about more emerging technologies, both hardware and software, that we could incorporate into the systems we were developing for clinical, diagnostic, and rehabilitative functions. These

are some of the reasons we initiated a series of technology demonstrations to establish the wide range of applicability of this method and technology.

11. Controlling Virtual Objects for Rehabilitation

This series of exploratory adventures led to development of a system of hardware, software, and analytical methods that were first described in the neurorehabilitation workstation paper.

The nature of diversity encountered when dealing with individuals with various disabilities or interface needs became evident as the experimental nature of our systems begin to extend to larger and larger numbers. While it was the case that the systems used to date were for the most part off-the-shelf interface devices which had been developed for other uses, it was also observed that a wider application of these techniques was possible only if the system could be individualized and customized to meet the requirements of the actual users.

Given the variability of the capacity of the human information processing it was determined that a very modular, component based system was necessary. The need to reflect the complexity of the human systems and the ability to adapt to different channels for input and output was necessary. The ability to address individual data channels which contained the information coming from or going to the individuals was seen as the most fundamental issue to be addressed.

Another very important finding was that the ability for individuals to actually control something in their own environment was a very motivating activity and that there was an increase on volitional activity on the part of the individuals to engage their environment.

The following case will illustrate this, as well as our process of using the reference architecture and iteratively refining the interface system.

12. Ashley: the Migration from BioMuse and Meme to

TNG-1 and Neat

Ashley's disability was the result of a birth accident. She is classified as C1 with some brain stem involvement (see pictures); she can move her head and she is vent dependent. When we first started working with Ashley, her way of interacting with the world was with the head stick (which she was good at and proud of). Her grandmother wanted us to experiment to see if we could identify some other avenues for Ashley to start interacting with computers and other devices.

One of the first things we did was to take something she already knew how to use well (the head stick) and adapt it to an existing interface (the pen mouse). By understanding the technology, we were able to very quickly enable Ashley to interact with the computer. However, this was still quite limited and we felt more could be done.

What we really wanted to do with Ashley was to have her start using the BioMuse system to control the computer. BioMuse allowed direct muscle (bioelectric) control in a virtual environment. Over time we adapted a series of interfaces which detected bioelectric activity in very specific muscles of her face and used those as independent data channels. The idea was, since Ashley had control of her facial muscles, we would use those as her "fingers" to give her differential control into the computer. While Ken Kashuahara of ABC's America Agenda (see photographs) was here, we showed that Ashley was able to navigate around in a virtual environment using only her facial muscles. This was only the second time she had ever even seen this technology.

We had to build software called Neat DOS for having different muscle signals cause different outputs (gesture recognition?). Then some of the engineers I was working with built a remote control car that had a camera (radio shack remote control car and radio shack transmitter) and Ashley with a set of VR glasses and a TV seeing what camera was seeing was able to drive this car out onto her back porch and to start playing with her niece and nephew. And so this car actually became an extension of Ashley's intentional actions with the world. And this is where we got into biocontrolled telepresence where a direct interface from muscle activity is used to control a tele-robot interacting in a fairly complex environment (i.e. with her niece and nephew).

It was very interesting to watch her play as she would chase them around and run into them and you could really tell by looking at the car that she understood the paradigm of what was going on.

Another extension for Ashley was that at the time we had access to a surgical robot that was for laproscopic surgery for positioning the camera and we bypassed its interfaces and we created a system that she with her facial muscles, which she had been refining, we were able to put a paint brush at the end of the surgical robot, give her some paint and allow her to use her face to use the surgical robot to create art. Biocontrolled tele-robotic arm capability.

Ashley was a good test case as she had a great family environment, personable, excited, easy to work with.

So we extended that one step further to the helicopter. We took a virtual IO pair of glasses coupled with Ashley's ability to move her head left, right and up and down and there was a camera controller on a helicopter which when she would move her head it would move the camera positions around while it was flying.

Thus Ashley was one of the first biocybernauts in controlled unmanned aerial vehicle payload controller which we are currently under contract with DARPA to develop for the military.

So the idea was that with Ashley and the concept of the flow of information we were able to take her from where she was and look at her brain function, input information into her, allow her to process it and then output something into the world. And Ashley then became, because of the way we were presenting information and the way we were able to acquire information was one of our test cases for the model of perceptual cybernetics and physioinformatics.

The main focus of these case examples was to show the utility of this reference architecture. Especially to show the capacity for enhanced perceptibility and expressability that may be achieved through an intelligent application of this physiologically robust interface systems model.

Using this reference architecture one can demonstrate interactive environments that combine new ways to render complex information with advanced computer to human input devices. Content specific information can be rendered onto multiple human sensory systems giving a sustained perceptual effect, while monitoring and measuring human response in the form of physiometric gestures, speech, eye movements, and various other inputs.

Summary

The Physio-informatic systems architecture has been successfully implemented in our efforts which:

• Assessed the limits of traditional input devices such as mouse, joystick and keyboards to determine when interface complexity precludes their use as primary inputs.

• Researched and evaluated the users' ability to integrate several input systems so as to have a multiplicity of simultaneous interaction options.

• Researched system capacity for filtering and combining data streams from the various human to computer input devices.

• Researched system capacity for developing user defined gestures for controlling various parameters of the interface.

• Researched the functional integration of relevant human-to-computer input devices into the system.

• Researched various multisensory rendering systems integrated into the system.

• Researched an integrated system of human to computer input devices with the multisensory rendering systems.

• Researched an interactive, experiential environment optimized for intelligent interaction with information.

We were able to achieve increased throughput by extending the perceptual dimensionality of information presented to the human and enhancing the expressional capacity of the human to convey intent to the informatic system. Quantitative human performance assessment tools for clinical, educational and vocational applications have been developed and refined, and interactive systems for the disabled which empower them to participate more actively in their own environment have been prototyped and investigated.

The following appendices show further utility of PHYSIO-INFORMATIC SYSTEMS.

FIRST PRINCIPLES OF PHYSIO-INFORMATIC SYSTEMS

APPENDICES

APPENDIX 1 -- NeatTools

What is NeatTools?

NeatTools is a powerful visual programming environment that allows users with disabilities to control and communicate with a computer. It operates in conjunction with hardware devices created especially for this purpose. A disabled individual generates some deliberate movement under his or her control, perhaps moving a cheek muscle; this movement is then detected by the device, which transmits signals conveying the information to the computer. NeatTools then translates this information into some form that the computer can interpret and generates some meaningful output , perhaps a mouse click or a cursor move. The software thus allows disabled individuals to use whatever physical capabilities are available to them in order to interact with a computer to improve their quality of life. NeatTools can permit quadriplegics to type, draw, or play games; to use the World Wide Web and e-mail; to control devices in their environment such as lights, stereos, or TVs; and more generally, to interact with others. Capabilities that the able-bodied take for granted are made available to the disabled through this sophisticated computer program. Individuals who have previously had to depend on others to do nearly everything for them can gain some control and independence with the help of NeatTools.

Who Developed NeatTools, and When?

The story of NeatTools goes back to 1990. At Loma Linda University in California, Dr. Douglas Will led the Neurology Research Group, which Dave Warner joined soon after becoming an MD/PhD student there in 1988. The group researched ways that a human/computer combination might offer valuable diagnostic information, or might explain more about how the human brain functions. The group identified and experimented with the newest developments in technology for this project. They acquired a special lycra fabric glove with optical fibers which they used to investigate the effects and treatments of neurological diseases. A corporation called VPL Research had created the glove, called the DataGlove®, and donated the $10,000 device to the group for exploration of its medical potential. Dr. Will's group determined that because the glove could detect small changes in hand or finger position it could measure hand function in patients with Parkinson's disease or other motor disorders. The glove could also help physicians to diagnose whether a patient had Parkinson's or whether that patient suffered from a similar disorder, such as essential tremors. Through the precise quantitative feedback, the glove could also aid physicians in understanding the effects of medication on the functioning of Parkinson's patients. It could also be used in rehabilitation, to help patients develop skills in particular motions.

For such a glove to communicate with a computer, an intermediary device was needed to translate the glove's electrical impulses into signals that a computer could understand. The device chosen by the group for this purpose was an workstation called BioMuse, which had just been developed by BioControl Systems in Palo Alto, California. Although its cost was approximately $20,000, the Neurological Research Group was one of four locations given free use of the equipment for medical and humanitarian purposes. BioMuse used electromyographic (EMG) sensors to detect voltage differences that arise when muscles are flexed. It then transmitted these voltage measurements to a computer where the BioMuse software converted the voltage differences into music. BioMuse was one of the first devices to allow this type of bi-directional feedback between a patient and a computer.

At the end of 1991, the Neurological Research Group was approached to see if they could help a hospitalized patient, a baby named Crystal Earwood. At the time, Crystal was 18 months; she had been paralyzed from the neck down in a car accident. She needed stimulation, and she needed some way to interact with the outside world. Dave Warner, still a medical student and a member of the Neurological Research Group, took up the challenge to find out what the technology could do for this patient.

Dave found that 18-month-old Crystal could control a computer by moving her eyes. BioMuse could transmit the voltage differences from her eye motions to the computer. Her eyes became the equivalent of hands, transmitting commands to the computer. Instead of generating music, the BioMuse system was modified by the group to give a graphical output. Crystal could then interact with displays on a computer screen, effectively demonstrating the potential value of biocybernetic technology for the severely disabled.

But despite all the promise, one serious hurdle remained. Since Bio-Muse cost around $20,000 and the group had only one unit to use for research and development, the equipment could not be left with Crystal, or with any of the 30 subsequent patients it was used with. For most of them, the BioMuse technology proved highly effective. For some, it greatly improved opportunities for communication and interaction. For others, the experience of generating music significantly increased patients' motivation to exercise weak limbs. But the high cost limited BioMuse's accessibility to the disabled patients who most needed it. Two patients were able to afford to buy their own equipment; the others were not so fortunate.

MAKING AFFORDABLE TECHNOLOGY

The Needed Hardware

The first breakthrough in this impasse came in 1995, when Salomo Murtonen, the Finnish inventor of the Sound Chair, came to America to volunteer for this project. A self-taught electronics engineer, Salomo committed himself to create the equivalent of the Bio-Muse device at low cost. At first he worked for next to nothing, since the group had no substantial source of funds. Salomo created a four-channel EMG interface that could take any signal derived from muscle movements into the computer. The device was named TNG-1 (Thing 1, from The Cat in the Hat by Dr. Seuss); TNG is also short for Totally Neat Gadget. Salomo produced TNG-1 with Radio Shack parts for a cost of $200, 1% of the cost of a Bio-Muse workstation.

Creating the Software

In 1994, the group had begun to work with a seven-year-old girl named Ashley Hughes. As a result of a broken neck during birth, Ashley is a C1 quadriplegic, paralyzed from the neck down, dependent on a respirator for breathing. She could move facial muscles, and TNG-1 could transmit the EMG signals from her facial movements to her 286 computer. Software was then needed to make those signals meaningful to the computer, and to display them on the screen. Joh Johansen wrote a program called BioEnvironmental Control (BEC) to make the EMG gesture signals meaningful to the computer, and to convert these to graphical outputs. BEC was designed for Ashley's facial capabilities; it allowed her to express herself in rich and complex ways, using her body as a way to control a computer. With these powerful technologies, Ashley was able to play computer games, drive a remote-controlled car, experience her world remotely through a camera and microphone mounted on a Styrofoam structure named Cindy Cyberspace, and interact with others in her environment.

Now the group had created both affordable hardware and software. But though TNG-1 was inexpensive, it was not free of problems. For one thing, the electrodes that detected the muscle movement were not stable. Setting up TNG-1 was not easy for the families; it could not just be left with them. Further development was needed to make the technology more stable and easier for family members to use.

Making the Technology Easier to Use

With the development of the TNG-1 interface device and the BEC software in California, the capability to interact with a computer was now affordable for the disabled. However, TNG-1 was not easy to operate. TNG-1 depended on the use of EMG sensors attached to the faces of quadriplegic patients, but whether mounted on cheeks or foreheads, the EMG sensors would not work for very long. They didn't stick to the face well, coming loose easily with repeated movement. To get around the problem posed by dependence on EMG sensors, Salomo went to work to create a more flexible interface device, that could receive a variety of types of sensory signals. The next upgrade, TNG-2, could detect changes in light signals resulting from cheek motions. Such a motion would distort the light path to a light receptor and thus create a signal. The lights did not have to be attached to the patients' skin, as did the EMG sensor; the light sensors could be mounted from a hat, a helmet, or most recently, from a pair of glasses. But light was only one possible type of signal detectable by TNG-2, which was constructed to receive up to four general-purpose analog inputs. The use of light signals had advantages over the EMG approach, but raised new problems. Because photocells change their signal levels when the brightness level changes in a room, the use of light signals required frequent calibration, often beyond the capacities of a disabled individual's family. The group then experimented with signal sources other than light by using Hall Effect transducers to detect changes in magnetic field and by using pressure transducers. Other approaches include using bend sensors or tilt sensors. For Ashley Hughes, for instance, the group was able to use a tilt sensor along with a pointing stick attached to her head. With this combination, she could use a screen keyboard to type.

New and Better TNGs

TNG-1 and TNG-2 could each convey four channels of information to the NeatTools software. This meant that the system was limited to information from, say, four muscles, or from four light detectors. Alternatively the user would have to depend on two TNG devices at once, requiring two available ports on the computer. With the creation of TNG-3, 16 channels became available (8 analog and 8 digital). As of January 2000, the group was testing a working prototype of TNG-4, which will have eight analog and 16 to 20 digital lines, each of which can serve as input or output. This increased capability will mean that the computer's parallel port can be left alone for other functions, such as connecting a printer or a zip drive. TNG-3 and TNG-4 have been developed by Edward Lipson and Paul Gelling at Syracuse University, New York.

Neat Becomes Really Neat Becomes NeatTools

As computer hardware technology has been advanced, the group has simultaneously been working on new iterations of the software, to take advantage of the new hardware's capabilities. The basic conception and architecture for the software, from the original DOS version to the latest NeatTools version, was laid out by Dave Warner. In 1993, the original DOS version and the first Windows version of the BEC software (renamed "Neat") was written by Joh Johansen in California for the MS-DOS, a common operating system, for personal computers ("PC"s). There were many limitations to DOS, which affected the capabilities of Neat. For instance, DOS does not support standard TCP/IP protocol, so the DOS version was limited in connectivity. DOS does not support multiple processes occurring at once, so the DOS Neat program was slow. DOS does not support multimedia, so neither could Neat.

In 1996, a version of this software was created for a Windows 95 PC environment; in late 1996, work began on another version of Neat based on a Java-like environment. "NeatTools" is suitable for virtually any computer system: it can run on Windows 95 or 98 or Windows NT; it can run on Unix, Irix, or Linux; and it will be able to run on Macs upon the availability of multithreaded operating systems. for Macs. NeatTools can interface with a much broader range of devices than previously. Its capabilities include both Internet connectivity and multimedia sound. A user can simultaneously develop, edit, and execute programs in NeatTools.

Neat Tools was created by a Computer Science Ph.D. student at Syracuse, New York named Yuh-Jye Chang, and formed the basis for his Ph.D. dissertation. Yuh-Jye defended his dissertation work in 1999, and is now at Bell Labs in New Jersey. The group has been especially fortunate in having both programmers. Before taking on the Neat Tools project, Yuh-Jye had won the 1996 Java Cup International Award for an earlier Java-based graduate project: the Visible Human Viewer program.

One of the major breakthroughs in Yuh-Jye's creation of NeatTools was the development of a Windows mouse driver, which would allow a disabled individual to move a cursor in any direction with cheek or eye motions. This complex program allowed a user to move the mouse and to simulate a mouse click. Before this, the group had to get hold of the source code for each program a disabled user would employ, and rewrite the code so the disabled individual could work with the program. With commercial software applications like Microsoft Word® or Windows-type programs, there is no way to acquire the source code, so it was quite a boon to have the general capability to simulate mouse events. A subsequent program created by Edward Lipson allows the user to move the cursor via a custom joystick mechanism. The software allows for fine calibrations, since different quadriplegic users have differing kinds and ranges of motion, as well as differing facial shapes and sizes. The NeatTools software and some related application programs can be downloaded at no cost from http://pulsar.org/NT/index.html.

What Makes NeatTools so Neat

The software had to be adaptable to the widest possible range of industry or in-house devices, and allow as many input channels as possible for users of limited physical capabilities. To allow maximum flexibility and modification for the needs of individual disabled users, the software is based on modules. NeatTools now consists of approximately 200 different modules, not counting the alphanumeric characters, and it is relatively easy to add modules.

The program offers visual programming with a highly user-friendly graphical interface. A user can create a simple NeatTools data-flow network by dragging a few modules to the desktop and connecting them. Typically, a module has inputs on the left and outputs on the right, and control inputs on top. A user can modify the properties of a module by a right mouse click. There are keyboard modules, modules for serial and parallel ports, modules for Internet sockets, calibrator modules, graphical display modules, modules for arithmetic and logic operations, multimedia sound modules, etc.

Because he was aiming for speed, a high level of performance, and platform flexibility, Yuh Jye chose to use the C++ program language instead of Java. When he began in 1997, the Java technology was not mature, and even today is not fast enough for intensive real-time tasks such as compression or decompression of video data. NeatTools consists of over 50,000 lines of C++ code. The program adopts the advantages of Java, especially its ability to work on different types of computers, by utilizing a Java-like Cross-Platform API (Application Programming Interface). This API is a very thin layer which provides an interface to the user's computer operating system and to the C++ core of Neat Tools. The API unhooks an application, such as Netscape or Word®, from dependency on the computer's operating system or on Windows. Thus the architecture of NeatTools consists of three layers: 1) the computer's operating system and the C++ programming language; 2) the Java-like cross-platform API; and 3) the NeatTools application layer. This combination allows NeatTools to incorporate the speed and efficiency allowed by the high performance levels of C++ along with Java's ability to run on different platforms. The thin layer of the API also provided the ability to synchronize many operations at once; such functionality would have been much more difficult to create in C++ alone. NeatTools provides the advantages of Java without using Java.

HARDWARE:

TNGs and Widgits

Neurocosmology influenced the design philosophy of the microcontroller technology, resulting in the following products:

• TNG-1 -- EMG

• TNG-2 -- 4 analog input channels

• TNG-3-- 16 input channels (first production version)

• TNG-4 -- 32 channels bi-directional

• Widgits -- sensors and transducers

These representative applications are described in detail at www.pulsar.org

APPENDIX 2 -- Anthrotronics (Human Instrumentation Systems) and Physiologically Oriented Interface Design

Anthrotronic systems designed for interactive information exchange continue to evolve. Applying the first principles of physio-informatics facilitates design innovation for operational refinement of the evolving interface system,. This enables the demonstration of an integrated system of human to computer input devices with the multisensory rendering systems for an interactive, experiential environment optimized for intelligent interaction with information. Research areas and methodology required to fully understand the quantifiable information flux capacity of a physiologic system requires knowledge in these following areas: neuro-physiologic restraints and limits of the computer to human linkages; psycho-physiologic capacity for optimal cognitive function within an extended perceptual environment; psycho-motor function for simultaneous interaction with multiple human to computer devices; functional integration of relevant human-to-computer input devices into the system; and multisensory rendering systems integrated into the system.

The reference architecture of physio-informatics has the necessary features to map the flow of information in an interactive human computer interface system, as well as he necessary complexity to be able to account for the physiological issues in an interactive human computer interface system.

The development of hardware and software systems based on principles derived from this reference architecture, and implementation of such systems in real world settings, along with the combination of applied physio-informatic principles with an operational notational system, creates a research tool capable of mapping the time evolution of information propagation through a perceptual cybernetic system. The next focus will be to develop a more generalized capacity to address the interface issues of human-computer interaction, and optimize the technology to the physiology.

Physiologically Oriented Interface Design incorporates this next paradigm of interface technology. It is based on new theories of human-computer interaction which are physiologically and cognitively oriented, and describes a biologically responsive interactive interface. Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. We will develop these interface techniques and technologies consistent with basic neuroscience issues, among which are modality, duration, intensity, distribution, frequency, spatial displacement, contrast, inhibition, threshold, adaptation, transduction, conductance and transmission.

To integrate a set of advanced human-to-computer input devices into a single interface system and demonstrate data fusion to enable meaningful correlations across various input modalities will significantly enhance progress toward these ends. We propose to develop an interactive environment incorporating new ways to render complex information to the user by optimizing the interface system to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information. Viewing the entire body as a perceptual and expressional technology opens up possibilities for exploiting the heretofore untapped richness and greater volumetric potential of its informatic capacities.

Our group has been researching, prototyping, and demonstrating the implementation of a data analysis subsystem designed to enhance the ways that relevant data may then be rendered optimally to the operators' sensory modalities, utilizing such techniques as linear and nonlinear multivariate analysis tools for the processing of multiple data sets. These sets include graphical analysis (phase portraits, compressed arrays, recurrence maps, etc.) and sound editing (mixing, filtering ). The human-computer interface was designed to optimize the salience and content of data sets; for example, some data which conventionally would be displayed visually might be processed to be perceived in a tactile or auditory manner. Using the DataSuit®, an extension of the DataGlove® technology previously mentioned, we can enable synesthetic interaction with and communication of data. Synesthetic environments rendered with this technology make it possible to feel the sound, see the pressure and ultimately be able to reconfigure the rendering parameters of the interface based on the specific elements of a situation. Seeing colors may be more appropriate in one context whereas hearing them may be more suitable for another; many factors will determine the tailoring of rendered data (which data will be shunted to which renderer). Novel interface controllers will prove of value essential here.

Our experiments into this area: