FIRST PRINCIPLES OF PHYSIO
INFORMATIC SYSTEMS
The conceptual, theoretical
and experimental basis for a general systems reference architecture for
Physio-informatic systems
Physio-informatics
is a new
systems model for linking human physiologic systems to information systems
in the most general way. This general systems model has been derived through an
ever evolving series of experiments and explorations.
The
conceptual, theoretical and analytical basis for establishing a general systems
based reference architecture for
Physio-informatic systems necessarily crosses many disciplines. It must be
emphasized from the onset that the
following discussion of the derivation and
development of a general model (aka.. reference architecture ) for
describing “meaningful” information
flow between humans and informatic systems is a broad topic area which covers
many scientific disciplines, engineering techniques and a continually expanding
array of technologies. Including but not limited to Physiology, Physics,
Mathematics, Philosophy, General Systems, Bio-Cybernetics Systems, Cognitive
Neuroscience, Perceptual Psycho-Physics, Perceptual State Space Modulation,
Bio-Sensors, Quantitative Human Performance, Expressional Interface Systems,
Physio-Informatics, Intelligent Interface-Metrics, User Tracking Interface
Systems, Distributed Tele-Robotic Controllers and Intermental Networking.
A
general perspective of this effort is that it is an attempt at integrating these
areas of human scientific endeavor (as mentioned above) in a manner which will
not require that future researchers in Physio-Informatics master all of them before they can
contribute meaningfully to the process of optimizing the coupling between humans
and informatic systems in an interactive interface system. Thus the intent of
this effort is to establish a general conceptual framework (a reference
architecture) which can be used as a guiding heuristic tool when
confronted with the challenge of designing and developing interactive
interface systems for human computer interaction. Specifically one which extends
perceptual dimensionality and facilitates enhanced expressivity.
Physioinformatics
A systems based, physiologically robust, reference architecture for designing and refining interactive human-computer interface systems in ways which increase operational throughput of information.
The
term“ physio-informatics” will be used in this dissertation to denote
informatic systems which are either biologically/physiologically based
(primarily neurologic i.e. neuro informatic) information systems and/or
informatic systems which are designed to support interaction (dynamic exchange
of information) with such systems
The
intent of this work is to develop a systems based, physiologically robust,
reference architecture for designing and refining interactive human-computer
interface systems in ways which increase operational throughput of information.
Extending the perceptual dimensionality of information presented to the human
and enhancing the expressional capacity of the human to convey intent to the
informatic system achieve this increased throughput.
Interactive
Human-Computer Interface Systems
In
the various traditional models of human computer it is customary to think in
terms of inputs and outputs. Input from the computer to the human and out from
the human to the computer or input from the human to the computer and output of
the computer to the human. The purpose of this dissertation is to develop a
systems model for interactive human-computer interface systems which is thought
to be more representative of reality than traditional models in that it is
consistent with the phenomenological aspects. That is the development of a
physiologic based reference architecture for designing and developing
interactive human computer interface systems to match the human nervous
system's ability to transduce, transmit, and render to consciousness the
necessary information to interact intelligently with information.
"The
physiologic basis of a reference architecture for designing interactive
human-computer interface systems"
Context And
Initial Motivation
The
capacity of computers to receive, process, and transmit massive amounts of
information is continually increasing. Current attempts to develop new
human-computer interface technologies have given us devices such as gloves,
motion trackers,3-D sound and graphics. Such devices greatly enhance our
ability to interact with this increasing flow of information. Interactive
interface technologies emerging from the next paradigm of human-computer
interaction are directly sensing bio-electric signals (from eye, muscle and
brain activity) as inputs and rendering information in ways that take advantage
of psycho-physiologic signal processing of the human nervous system (perceptual
psychophysics). The next paradigm of human-computer interface will optimize the
technology to the physiology -- a biologically responsive interactive
interface.
INTERACTIVE INFORMATION TECHNOLOGY
Interactive
information technology is any technology which augments our ability to create /
express / retrieve / analyze / process / communicate / experience information
in an interactive mode. Biocybernetics optimizes the interactive interface,
promising a technology that can profoundly improve the quality of life of real
people today. The next paradigm of interface technology is based on new
theories of human-computer interaction, which are physiologically and cognitively
oriented. This emerging paradigm of human computer interaction incorporates
multi-sense rendering technologies, giving sustained perceptual effects, and
natural user interface devices which measure multiple physiological parameters
simultaneously and use them as inputs. Biologically optimized interactive
information technology has the potential to facilitate effective communication.
This increase in effectiveness will impact both human-computer and human-human
communication, "enhanced expressivity".
"BIOCYBERNETIC CONTROLLER"
Interactive
interface technology renders content specific information onto multiple human
sensory systems giving a sustained perceptual effect, while monitoring human
response, in the form of physiometric gestures, speech, eye movements and
various other inputs. Such quantitative
measurement of activity during purposeful tasks allows us to quantitatively
characterize individual cognitive styles. This capability promises to be a
powerful tool for characterizing the complex nature of normal and impaired
human performance. The systems of the future will monitor a user's actions,
learn from them, and adapt by varying aspects of the system's configuration to
optimize performance. By immersion of external senses and iterative interaction
with biosignal triggered events complex tasks are more readily achieved. This
paradigm shift of mass communication and information technologies is providing
an exciting opportunity to facilitate the rapid exchange of relevant
information thereby increasing the individual productivity of persons involved
in the information industry. Areas such as computer-supported cooperative work,
knowledge engineering, expert systems, interactive attentional training, and
adaptive task analysis will be changed fundamentally by this increase in
informatic ability. The psycho-social implications of this technologically
mediated human-computer and human-human communication are quite profound. Providing the knowledge and technology required
to empower people to make a positive difference with information technology
could foster the development an attitude of social responsibility towards the
usage of this technology and may be a profound step forward in modern social
development. Applications which are intended to improve quality of life, such
as, applications in medicine; education, recreation and communication must
become a social priority.
An overview of the
field of interactive human-computer interface systems
In
the time between those early days and the late 70’s the ability for the
computer to respond to a task given it by a human was, for the most part,
limited by computation power. From the early days of computer programming where
each logical connection was “hard wired” by an army of technicians to the time
of punch cards, the concept of interaction with a computer had a very limited
and specific interpretation. A great advance was made when the computer had a
“typewriter” like mechanism which allowed computers to be programmed through
“terminals”. As the speed of the computer increased along with its capacity to
respond to the human, it began to become apparent that the humans ability to
convey intent to the computer and the computers capacity to display results of
its calculations to the human would become a limiting factor in the
“interaction” between humans and computers.
Innovators at Xerox PARC, MIT, NASA, DARPA and other computer research
facilities began to rethink the concept of how humans and computers would be
able to communicate more effectively. (in this context the term “communicate”
is used to mean the ability to intentionally exchange meaningful
information).The first significant breakthrough to make it out of the lab was
the “WIMP” interface (Windows, Icons, Mouse, Pointer) graphical user interface,
referred to in the nerd zone as the GUI (pronounced gooey). In the late 70’s
with the commercial release of Apple’s“ personal” computer, with it’s GUI, to
the general public …….blah..blah
In
the mid 80’s computer systems used by industry were becoming fast enough, and
display technology and was becoming sophisticated enough to be able to “render”
graphic images for engineers in computer aided drafting and design jobs to be
able to begin to manipulate these rendered images with ever increasing speed
and resolution. At the same time new techniques were being developed by
scientists to enable them to “visualize” a graphic image that was the result of
a very complex set calculations. These new areas of CAD/CAM and scientific
visualization continued to evolve with faster and faster computation and ever
increasing quality of graphic images.
In
the late 80’s it was recognized that the compute power and graphic display
techniques
While
it is true that there was much work was done in the human factors of “Man-Machine”
interfaces throughout the late 70’s and the 80’s, this work dealt with the
physicality of the environment and information displays, and much of that work
was done in the context of very specialized tasks. Tasks for specifics kinds of
work such as piloting fighter airplanes or space craft, controlling complex
industrial processes such as nuclear power plants, or complex chemical
processing plants were well studied and refined. However these task differ from
interactive human computer systems in that the humans are controlling some
machine or physical system and were not primarily interacting with information
as represented by computer systems. (one exception to this would be the
interaction with a computer simulation of a complex physical system). The primary
efforts for researching and refining these systems were in the field of
ergonomics, dealing with the energetics of the human interacting with their
environment, and cognitive science, the mental computation required to perform
effectively in the environment.
Also
in the late 80’s a new concept began to take hold in the field of human
computer interaction, the concept of “immerse systems” where the computer
systems began to encompass the humans senses and track the movements and
position of the human in an effort to develop a synthetic environment with in
which the human could interact in a more natural way. These virtual reality
systems sparked a brief but important revolution in the thinking and gadgetry
of human computer interaction. From an evolutionary neuro-information
processing perspective this technology creates a new potentiality for response
to perceptual awareness: it canalizes not a single response to a single
stimulus, but rather multiple responses to multiple stimuli born of a single though
multi-dimensional sensorial perceptual state. the combination of these
different rendering modalities with somatotopic placement, in order to achieve
and demonstrate spatial coding of the rendered information. Optimizing the
human computer interface will rely on the knowledge base of physiology and
neuroscience, that is, the more we know about the way we acquire information
physiologically the more we know the optimum way for a human to interact with
intelligent information systems. The next paradigm will see the
"THINNING" of the human-computer interface to a biological sheer as
the interface will map very close to the human body.
PHYSIOLOGICALLY ORIENTED INTERFACE DESIGN
Knowledge
of sensory physiology and perceptual psychophysics is being used to optimize
our future interactions with the computer. By increasing the number and
variation of simultaneous sensory inputs, we can make the body an integral part
of the information system, "a sensorial combinetric integrator". We
can then identify the optimal perceptual state space parameters in which
information can best be rendered. That is what types of information are best
rendered to each specific sense modality, "a sense specific optimization
of rendered information. Research in human sensory physiology, specifically
sensory transduction mechanisms, shows us that there are designs in our nervous
systems optimized for feature extraction of spatially rendered data, temporally
rendered data, and textures. Models of information processing based on the capacity
of these neurophysiological structures to process information will help our
efforts to enhance perception of complex relationships by integrating visual,
binaural, and tactile modalities. Then by using the natural bioelectric energy
as a signal source for input; electroencephalography, electroocculography, and
electromyography (brain, eye and muscle) we can generate highly interactive
systems in which these biological signals initiate specific events. Such a
real-time analysis enables multi-modal feedback and closed-loop interactions.
The
following discussion is concerned with developing a “reference architecture” (a
formalized conceptual framework for thinking) for designing physiologically
robust interactive human computer interface systems. The purpose of the
reference architecture will be to provide insight into the various components
of the system in the context of how they might affect the flow of information
as information is passed through them The primary focus will be to consider the
flow of information between the human and the computer in a sustained,
iterative, experiential interaction In the context of this dissertation it will
be assumed that the intent of developing this reference architecture is to map
the information flow during/caused by the intentional /volitional interaction
with information between a conscious human and a computer system An exchange of
information between the an experienced perceptual state and an external
physical state is mediated by a biologic / physiologic information transporter
system This system is multi modal – multi scale – concurrent hetero-purpose
poly-dyno- morphic simul-tasking
For
this discussion we will assume that interface systems which support Human
computer interaction can be modeled as a system where information flows between
various components of the system in a specific manner
Theoretical position –
Information can be mapped and represented as a specific state space parameter set.
Universe of
discourse “Mind happens at an anthroscopic scale.”
The
phenomena of interest, (perception and expression), occurs at the anthroscopic
scale.
The
anthroscopic scale, the natural scale of perceptibility and expressivity of an
individual human, is “From meters to millimeters, from decades to
deci-seconds.”
Assumptions
Time
is perceived as a unidirectional vector.
The
nervous system is the primary information infrastructure for humans.
The
nervous system supports the transduction transmission representation and
response to information in the environment.
Basic
principles.
Human
perception and expression is mediated, for the most part, by the nervous
system.
Hypothesis
An
understanding of the human neuro physiology allows for exploitation of
predictable adaptive capabilities. The assertion is that the information flow
between external sources and direct experience is
biased/restrained/constrained/limited/enhanced/facilitated in understandable
and predictable ways by the physiological mechanisms of human information
processing.
Physio
info metrics
--- the quantitative measure of the information carrying capacity of a
physiologic system.
Physiologically
mediated information is exchanged between external environment and experiential
awareness. The fundamental nature of the nervous system (neuro info matrix)
determines its operational capacity. Both the physicality and the physiology
contribute to the set of bio-physical restraints. The physicality of the
nervous system constrains the perception of space, time, mass and energy.
Physiology of the human nervous system restrains perception by computational
limits of the system. The complexity, functionality and capacity of the
intra-activity of the nervous system sustains perception. ERGO - The form and
function of the nervous system influence various parameters of perception and
expression. That is to say that nervous system is the biologic structure that
is considered most likely to be responsible for mediating information flow
within the human body
The
Basic ideas leading to the primary foundations for this thinking can be seen as
coming from the following areas
Operational Philosophy
-Action
directed goals in the pursuit of new knowledge - which start with logical
analysis of observed phenomena and
proceed to the point of discerning an operational utility of continuing the
pursuit in the current mode of analysis or changing modes to seek a more
fruitful mode of investigating the phenomena. (Oppenhiemer)
In
other words it is a philosophy of scientific investigation which constantly
seeks to validate the current mode of analysis for a given set of observed
phenomena so as to maintain constant progress in the discovery process of new
knowledge.
General Systems Theory
General
systems theory is a useful framework for developing complex models for
investigating complex systems, like those of Physio-Informatics, is in as far
as it has certain concepts of systems models and principles such as
hierarchical order, progressive differentiation and feedback that can be defined and characterized and elaborated
on with set and graph theory which state explicitly conditions for membership
and orders of relationship.
The
“open systems” approach to a general systems theory by von Bertalanffy in the late 1930’s was instigated by a
perceived need to break out of the “closed systems” model which implicitly
separates the system from its environment, as it would lead to incorrect
conclusions. His concept was that biological systems necessarily must be
considered as being open systems where both information and energy is in
continuous flow between the system and the environment. His initial formulation
of a general system was an attempt to derive principles which were valid for
open systems.
A
system can be defined as an object consisting of a set of complex objects or relationships, each of which are in
some way associated with other objects with in the system in a way that some
quantities (parameters) with in those objects are associated with quantities
(parameters) of other objects within the same system.
(
von Bertalanffy)
Information
The
base elements with which information is constructed is “difference”. A difference can be interpreted as either an
ontological fact or as an abstract matter.
Information can be defined as a difference which makes a
difference. (Bateson 1970) Or a difference with a non zero significance
(Warner)
The
relevant aspects of Information Theory concerning the transmission effects
on information across physical
structures, are considered to be important in physio-informatic systems, but
are tempered by the fact that biological systems do not adhere to the neg-entropy formulation of Shannon
Cybernetics
Also
of significant importance is theory of Cybernetics, the theoretical model of
feedback governed systems whose present state influences in some way the
probabilities of any future state occurring in the system. It is interesting to
note that the operators which are invoked on the system are a result of past or
currents states. This is important to establish that there is a relationship
between operators and states beyond the “transformational function” of
operators on states.
Definitions
A
state of any system is defined by the set values which describe the condition
of the system in any given point in time (the value of all the state vectors).
A system will have a state space which represents/contains all possible states
of that system.
State Space, States and Operators
For
those systems whose quantities are in continuous flux a special kind of set
called a “State Space” can be constructed which has as its elements (set
members) an n-tuple of values which are the values of the quantities at a given
instant. At any given instant the system is said to have a “state” which is
determined by the values of each of the “parameters” at that instant. (Ashby,
Zadeh)
For a given system whose States are not static (in time) within a given state-space it can be asserted that a transformational function has been performed on the system which determines the “next” state the system will be in. Such a transformational function is called an operator. Thus it is correct to say that an operator acts on an initial state parameter value and produces a new state parameter value.
In
an open system is can be asserted that the “evolution of the states in time”
i.e. the “state space trajectory” can be considered to be influenced by both
the current state of the system (internal factors) and the processes of the
environment (external factors) which are acting on the system.
States Operators and Information State-Spaces
A strange but useful mathematical modeling system for elaborating this has been established. A state space of a physio-informatic system can be described as a set of information-based states which behave in a particular manner.
The initial assumption is that all the information that is perceived about the external environment is filtered-mediated-biased by the nervous system. This is known as the “Neuro Cosmological Principle” of epistemology, which has an “Anthro-scopic scale” and an “Anthro-centric perspective”.
Expression Space - is a universe of discourse within which the physical and mental realms are complimentary subspaces.
The following discussion is
concerned with the development and elaboration of
The notation system for
mapping the flow of information in any physio-informatic system
The formal term for this is
“Expression Space Notational System”
“NEUROCOSMOLOGY”
A notational system which enhances the ability to see
differences
ANY information which can be
expressed is a member of Expression space.
Neuro-cosmology is a
notational system which has as its prime focus the flux and flow of information between a perceiving human being an
the environment. As such Neuro-cosmology has :
-A
superior descriptive process
-A
powerful system of methods and models
-A
superior conceptual frame work within
which to map info flux
-Derivational
pathways from first principles to operational systems
-Intrinsic heuristics
Assertion
The
nervous systems capacity to transduce, transmit, characterize, experience and
respond to information of environmental conditions limits the know-ability of
the environment. The anthroscopic neurophysiologic info matrix
supports/provides the basic infrastructure for the continuous
intentional/willful interaction with the environment. Biologically mediated
information exchange couples perception to the environment. The physio info
metrics of the neural info matrix determines the through flux. The exchange of
that information can be abstracted. The flow of information can be
parameterized by temporal, spatial, ergo-dyno-morphic flux. Any state of
information is characterized by a specific state-space parameter set.
Theoretical
construct
A
descriptive mathematical model that can map the transformation of information
as it is exchanged between various components of the interactive human-computer
interface system is presented below. This model which most generalizes the
phenomenological aspects has a notational system which exploits
interdisciplinary interaction and a languaging system which can classify
emerging observations.
.
Neurocosmology-
Anthroscopic epistemology is biased by Neural systems
A notational system has been derived which
represents the flow of information between the environment and the neurally
mediated experience of consciousness. This formal descriptive notational system
will enable the creation of“ most probable maps” of information flow between
humans and their environment.
|
A set can be defined
--Expression space is characterizable
Definition ANY expressible
information is in Expression space
|
|
An expression of any
experiencable information is a member of E-space ANY state of information which is experienced
and expressed |
|
The minimum element of
resolution is a the existence/experience of a difference
To have a non zero
information content a noticeable difference with non zero significance needs
to occur |
|
The minimum element of
information is a significant difference
IT is a difference between
ANY 1 and ANY other IT is the information
trajectory which relates ANY state of
information IT is an expression space
state path |
It
should be noted that the following formalisms will be loosely followed.
Set theory – where membership to a given set is
determined by some formal means
Graph
theory – where the position and relationships are determined by intrinsic
values
Given
the dynamic nature of physio-informatic
systems
A
more general system of States and Operators will be pervasive.
In
the course of systematic reflection upon those fundamental essences which
appear to be ontologically distinct primary percepts, one may construct a sound
and verifiable epistemology with a triad of categorical types. That is, if one
is to separate the "WORLD" as perceived into categories which are
characterized
phenomenologically rather than by the material constituents
with which it is constructed, one finds that there are three principle realms
which may act in concert and in various combinations to account for the whole
of perceptible "reality". These three categories are Experiential,
Biophysical and the Physical, more commonly known as MIND, LIFE and MATTER.
Consider
a complex system that has as its primary components 3 fundamentally distinct
sets of parameter values. Or three “state spaces” a state space is a set whose
members are defined by an n-tuple of values which correspond to the parameter
values of One set is a set of parameter values of a computational system.
Another set is a set of directly measurable parameter values of a physiologic system.
The third set is a set of parameter values which are directly experienced by a
conscious human. Each set is distinguishable from the other in that the
computational system and the physiological system are physically separable and
the physiological and perceptual state space are phenomenological. It is
acknowledged that the basis of perceptual experience is most probably supported
by a specific set of physiologically distinct systems. It is beyond the scope
of this dissertation (and frankly unnecessary) to develop a robust
explanation/model for the specifically/physically distinguishable aspects of
conscious experience (the mind) and assumed neural matrix (the body) that is
thought to at least co-occur with those conscious experiences. Suffice it to say
that a user in the context of an experiential interaction with information does
not routinely confuse the two.
|
Expression-space
|
|
Mind subspace
C-term [] C-term is the
mind-space operator. C-term is the
operation which changes one mind-state to another. There is an implication of conscious attention in a C-term
operation. When one instance of
couscous perception intentionally and willfully affects the generation of a
specific type of mind-state then this is considered a C-term operation |
|
Biological subspace
B-term [] B-term is the generic bio-subspace operator. B-term changes information states which
are directly linked to the biological
/ physiologic information processing functions. Any process which is achieved by a living
organism is a B-term type function. |
|
Space-time
Z-term [] Z term is a
generic space-time operator. This is
a function which accounts for all the physical (non-biological) changes in
information states in the physical subspace.
Z-term functions are those which the physicists and other physical
scientists study. They are things
like electricity, magnetism, gravity, thermal dynamics, etc. In general, anything with just physical
properties is a Z-term function. |
This
is the subspace of E-space which contains all the information states directly
perceived by the mind.
Now with these terms in mind (pardon the
pun) the next step is to specify the format or syntax of a valid expression in
Neurocosmology notation. The general
form of a neuro-cosmology expression is in the form of a commutatively diagram,
and it looks something like this:
All
instances of information from the physical subspace to the mind subspace will
be in this form, where PHI is the
object as it exists in space-time, Z-term is the information of the object
being transmitted through physical substance via light, sound, smell etc. PHX is in this instance the composite of all
the intermediate biologic information-state spaces necessary for information transduction
(i.e., sensation). This was comprised
of no less than stimuli, transmission of sensory output and neurological
processing and coding in the brain.
B-term is the process of sending (transmitting) the final product of
sensation to the mind for the purpose of perception. It is an axiom of neuro-cosmology that this expression in the
form presented above is both a necessary and sufficient condition for an
instance of perceptional experience to have in it an
information
content which was assimilated from an external object which did in fact exist
in the space-time realm (there is a world external to our senses!). That is to say that all the information we
have about the outside world was obtained in this way. If it hasn't occurred to you yet it may be
easier to use the following approximation when doing Neurocosmology
PSI
= the self perceiving mind--perceptual space: contents of awareness
PHX
= the living body -biological space: what chemistry refers to as
"organic" and all things related to it.
PHI
= objective, material, space-time reality --physical: what physics refers to as
space, time, mass, energy
Thus
any information that can come into being is one of 3 categories or combinations
of the 3.
The following is a list of all the major
state-operator-state expressions required to map information flow through the
physioinformatic system.
Neuro
cosmology states that basically that
perceptual states are a result of a brain state. However perceiving a uniform, coherent perception at
any given time, the state of perception, at ANY given time is called PSI, an
instant of perception which is all we ever have. Information coming from the
outside, causing neuro structures to fire on the sensory cortex is sensation.
The brain correlates that sensation with all its other relevant internal
functioning and all layers of psychosocial elements and the gestalt of all
brain function in consciousness equals perception.
ANY
contains a subspace of possible expressions; everything I could possibly
perceive is herein contained. Rather than matter and energy we're referring to
perceptual state-space. The perceptual state is always related to some
biologically parameterized state of the brain. The term for these biologically
parameterized states of the brain is PHX. This refers to all those entities and
processes discussed within the life sciences.
So PSI wishes to perform an action in the
outside world. By means of a cognitive operator PSI initiates the required
parameters of PHX, via a biological operator, to throw a ball, for example: a
PSI to PHX information transection resulting in a PHI operation: the ball
flying. So, information from the PSI subspace transferred information (via
these various operators) to the PHX
subspace which in turn transferred information to the PHI subspace. The
principle which derives from this dynamic is that whenever a PSI subspace has influenced
a PHI subspace some PHX subspace has mediated the transfer of information.
Neurocosmology comes from the expression
space and you define it with perceptual parameters. KHI- state of PHI whose form can only be accounted for by
positing the willful action of PSI via PHX acting on raw PHI, etc. KHI
The
diagram you see here is the biocybernetic loop. It is a very good way for
understanding all the elements of how a human interacts with a computer. If there is a mind then no matter what the particulars
of a disability, what have you, there is a way to interface that mind with a
communications technology.
Thus
it follows that with in the notational system described the necessary and
sufficient conditions for mapping the information flow through the
physio-informatic system is covered by the following:
A Mind (symbol: PSI) as the beginning of
all expression and the end of all perception; the foundation of the whole
process.
An Organism/Life (symbol: PHX) system
facilitating that perception and expression via sensory and motor actions (biological/physiological
factors).
An Interface system which is a two-fold
element (for a very high level technology which applies this thinking: Bot Masters)
Expressional (symbol: SNS): some
mechanism senses/captures some motor action as input from the human (e.g., mouse click, facial
movement).
Perception symbol: SKY): some mechanism
which renders results of the expression back to the human (e.g., screen, audio screen reader for the blind, etc.)
SKY-that product of KHI having been
manipulated by PSI so as to change PSI.
Synectic languaging structure in which we
have a generic metaphor to talk about anything; the goal of this notational
languaging system is to is to allow you to linguistically navigate to any far
reaching PSI space that you want to be involved in.
Move
into a higher state of abstraction connecting more information.
e.g., Sensory physiology is talking about the
PHX specific state spaces which are involved in transducing and transmitting
information to the central nervous system, which is part of the PHX system we
think is highly related to.
These
three fundamental state spaces are required to describe the complete system.
Information
that is described by the state of an external computational system
Information
that is described by a the state of a physiologic system
Information
that is described by the state of perceptible components of a conscious
experience
Each
of these systems is considered to be a fundamental source of information. It is
also presumed that there is an ordered flow of information between these 3
fundamental state spaces. As information is exchanged between these three
state-spaces there are direction specific boundary crossing transfer functions
which restrict/bias/interfere and/or otherwise constrain/restrain the
capacity/fidelity. From the perspective of the physiologic system there is an
information exchange with a persistent external information system and an
emergent internal information system. Thus it is considered a basic tenant for
this model that the body (comprised of all its information processing
physiologic systems) mediates the exchange of information between the computer
and conscious experience. This at first glance may seam some what obvious to
the casual observer. However it is this physiologic mediation of information
(with all of its specific processes)which is the basis for the development of
this reference architecture for developing and optimizing interactive human
computer interface systems..
Simply
put the 3 coupled state spaces of the interactive human computer interface
system being proposed may be described (from the perspective of a conscious
human user of this interactive interface system) as being comprised of/by an
external computational system, the imperceptible physiologic processes which
mediate the exchange of information with that system and conscious experience
and the perceptual qualities of experienced information
Information,
which is externally generated - Phi –Physics
Information,
which is biologically/physiologically mediated - Phx – Life
Information,
which is directly experienced - Psi – Mind (Perception and Intention of
Expression)
The
following discussion covers the efforts to demonstrate the capacity to increase
perceptual dimensionality.
In addition to the new models of mapping information flow was the identification of new mathematical models and representation systems that would give insight to complex phenomena. . New graphical methods of mathematical analysis were being developed which would bring insight into biology.
The following images represent the nature of this “new math”
The development of fractal (fractional dimension ..or non
integer dimension) geometry which was used in several ways to help develop an
understanding of complex systems and their states. Of particular interest was
the work of Price, Sale and Warner
which begin to explore the use of the deterministic iterative model of
mathematical analysis to gain insight on what happens when you alter the
various parameters of the base expression to be iterated. What was discovered
was that one could see complex yet identifiable changes in the rendered
geometry which demonstrated the basic premise of fractal geometry that initial
conditions of seed values and the basic algorithm 
Early
work by David J. Warner, Steven H. Price, S. Jeffrey Sale, in the above
images,
showed that the basic fractals which were
common in the popular media and scientific literature were useful tools for
gaining insight as to the nature of iterative systems. The images above show
that a simple change on the basic algorithm can led to very ordered systems
which are predicted by the change in the base formula. Ie z^2+c gives the basic
fractal but a progression to z^n +c ..where n is 0-4 shows that the number of
symmetries varies as n-1 but the individual patterns are fairly constant (that
is they look to be of the same class)
The
above images demonstrate the nature of iterative, deterministic functions they
also show that a “difference “ can be characterized both visually and
quantitatively.
The basic concept which was utilized from this observation/exploration was that for a given initial condition a unique result would occur and that this method could be used to begin to quantitatively characterize the specific complex dynamics which were intrinsic in the physiological data. This new tool was then applied to physiology to begin to explore the utility of using this description to gain a higher level of insight to the complex state dynamics of the physiological systems. This was important for the development of a robust physiological system which states of information can be inferred form the unique dynamics which could be measured by various instruments. To begin to move forward in the development of a theoretical model of physioinformatics it was necessary to demonstrate that the assumption about being able to uniquely characterize a measured physiological state space dynamic and to link it to a specific information condition of a physiological system.. To establish a relationship between the geometric, the quantitative and the physiologic the following experiment was done
The
following discussions show an early exploration into these methods
These
new mathematical insights were applied to the EEG which is a very complex
system.
CHACTROPIC
DYNAMICAL ANALYSTS OF THE EEC
The
mechanisms generating normal and abnormal rhythms in the brain are poorly
understood but are usually attributed to a combination of sinusoidal
oscillations and stochastic noise. Quantitative analysis of the EEC has
emphasized application of the FET and statistical analysis of the resulting
power spectra. It is now possible to perform chaotropic dynamical analysis of
the EEC. Phase portraits are obtained
by imbedding the EEG time series in a multidimensional space using various time
lags. Phase portraits can be rendered graphically in
2 and 4 dimensions. Lyapanov
exponents, fractal dimensions and Poincare sections can also be obtained. Inspection of phase portraits during 3/sec
spike and wave suggests the presence of a low dimensional state space. EEC during normal eyes open condition suggests
the presence of a high dimensional state space. In deterministic systems, low dimensional state spaces have low
information content and limited response capability. High dimensional
state spaces have
rich response repertoire. chaotropic dynamical analysis of
the EEC provides a powerful theoretical structure within which to interpret
normal and abnormal findings. chaotropic dynamical analysis is an important new
approach to quantitative investigation of electro physiologic measures such as
EEC and MEG.
The Compressed Dimensional Array: a new topographic technique for EEC analysis
The mechanisms generating
the EEC are poorly understood but are
thought to involve non-linear deterministic dynamics. The complexity parameter is an important mea sure of
dimensionality but displays usually do not permit of this parameter comparisons
between many EEC channels over
time. The complexity parameter is
obtained by embedding the time series in progressively higher dimensions until
a scaling property emerges. This dimension is then selected for the Compressed
Dimensional Array (CDA) - The
complexity parameter is then calculated for two second epochs and arrayed in a single
display on an graphics workstation so as to appear as a topographic contour in
which elevation represents the complexity parameter permits a Real-Lime
interaction with this array convenient Areas
method of dimensional analysis.
of low dimensionality appear as easily The CDA provides a new recognized valleys. method of visualizing
dimensionality of the EEC and reveals subtle features of clinical and
scientific interest.
The
use of new graphical techniques which enabled the researcher to gain a greater
degree of insight into the phenomena was applied to the problem of extending
the perceptual dimensionality of research data. Initially EEG data was used to test the utility
of this exploratory mode, it wasn’t
long however that other data sets were evaluated with these methods.
The
following series of images shows the progression from the standard methods of
analysis to a visually rich technique. (it will be asserted that this is
consistent with the reference architecture being developed in this thesis.) The
fundamental idea here is to use the humans natural ability to perceive
differences in space, color, and structure to help elucidate the various
features embedded in the data.
Traditional EEG data is of this nature
From
the images below it can be seen that
this “spatial temporal” iso surface technique enables the detection of
structural components in data that was normally seen as not having any
intrinsic structure.
The
technique developed for exploring d characterizing the electro physiological
data can be shown to be of value for displaying the data in a way in which the
various different modes of EEG seen clinically can be easily classified . The
images below illustrate that point in that they show the various conditions
seen clinically, evoked potentials, anesthesia states, eyes open relaxed
normal. All in a very “perceptible” form.
As
mentioned above the technique which was initially applied to the EEG was
extended to the ECG. It is interesting to note that the nature of these methods
is that they cause a new mode of thinking and a series of questions about the
former formal methods. (like electrode placement for best results)
It
was necessary to establish the fact that the method being used was in fact
providing meaningful and reproducible results
Having
established the relationship between the dynamics of a physiological system and
a quantitative and graphical representation system the next efforts were to
gain an understanding of more specific (experimentally controlled ) dynamic physiologically
mediated information The purpose of this level of effort is several fold. To
establish a relationship between the physio-dynamics as measured by various
instrumentation and meaningful which can be utilized in both a
research/exploratory setting and also for a finer grain on clinical assessment.
In the efforts to develop a series of meaningful representations if became
necessary to develop a new method of representation which was able to convey
complex dynamic transitions which varied in both space and time
Having established the dynamical analysis
methods of measuring direct bioelectric signals the research efforts were then
turned to implementing the array of new interface devices which were being
developed for virtual reality.
.
-Quantitative human performance assessment tools for clinical, educational and
vocational applications have been developed and refined
THE VPL DATA GLOVE As AN INSTRUMENT
FOR QUANTITATIVE MOTION
ANALYSIS
Movement related
potentials (Mrps) are
useful in the investigation of motor physiology.
Measurement of MRPs requires accurate determination of movement onset. New techniques have been developed for
precise quantitative measurement of complex movements of individual fingers and
the hand. Joint position, angulations, movement onset, acceleration, and
velocity are obtained through the use of the VPL Data Glove. The closely fitting light-weight lycra glove
does not restrict movement, change biomechanics or alter moments. Fiber optic sensors on the dorsum of the
hand and each digit dynamically measure angulations (with 1 degree resolution)
of p joints, the p Ip joint of the thumb and the PIP joints of the four
fingers. The position of the hand in
three dimensions is measured using a polhemus tracking system with 3.3 mm
accuracy for x, y, and z coordinates and 0.85 degree accuracy for pitch, yaw,
and roll. Data from each of the
sensors is sampled at 60 cps, and rendered graphically in real time or
stored in a file. Movement onset,
acceleration, velocity, and amplitude can be displayed. complex relationships between joints can be
studied during arbitrary motor tasks.
Data from healthy individuals during a range of motor tasks
will be demonstrated. Precise quantitative
measurement of hand and finger movement will be an important contribution to
neurophysiologic studies of Motor disorders.
THE VPL DATA GIDVE As A TOOL FOR Rehabilitation AND Communication
Rehabilitation of hand function following a stroke
may be enhanced through the use of biofeedback. The VPL Data Glove provides a rich
biofeedback environment permitting the patient to interact with an anatomically
accurate computer graphics representation of the hand. The glove is made of
comfortable light-weight lycra and is easily pulled onto the hand. fiber optic sensors on the dorsum of the
glove precisely measure hand and finger position. The glove is not restrictive and does not interfere with
movement. Data from each of the
sensors is rendered ;graphically in real time on a Macintosh computer screen,
giving the patient '$usual feedback.
The data can also be saved for quantitative assessment of improvement.
This allows the patient to set goals and measure progress. Tasks can be
customized for each patient's disability and changed to enhance patient
interest and effort. The glove has
gesture-to-speech capabilities, permitting patients with hearing or speech
impairment to communicate through computer generated phonemic speech. The glove
may also be used as a computer interface permitting disabled individuals to
control a wide range of external devices. Use of the glove in occupational
therapy and gesture-to-speech will be demonstrated.
THE DATA GLOVE For PRECISE QUANTITATIVE
MEASUREMENT OP UPPER MOTOR NEURON (UMN) FUNCTION IN ANYOTROPHIC LATERAL
SCLEROSIS
The
response to treatment or determination of progression in diseases such as
ALS. The Medical Research Council
(KEtO) rating scale measures only strength, is insensitive to early changes and
is ordinal. The Appel scale and Tufts
Quantitative Neurologic Evaluation (TQNE) scale measure isometric strength, but
inadequately assess UMN function. UMN
function is best assessed by measurement of dexterity.
The
VPL Data Glove can precisely measure joint position, angulation, acceleration
and velocity, permitting quantitation of motion of the digits, hand, wrist, elbow, and shoulder. The light-weight closely fitting lycra Data
Glove does not restrict movement or change biomechanics. Fiber optic sensors dynamically measure angulation
(with 1 degree resolution) of the hand and finger joints. with high precision and in three dimensions,
the position of the hand is measured 60 times per second and can be rendered
graphically in real time or stored for later analysis. Movement onset, acceleration, velocity, and
amplitude can be displayed. Data from
healthy individuals and ALS patients will be demonstrated. Precise quantitative measurement of movement
will be an important contribution to clinical
trials in ALS.
.QUANTITATIVE ANALYSIS OF TREMOR AND CHOREA USING
ThE VPL DATA GLOVE
Analysis
is of chorea, myoclonus and tremor is often limited to direct 0bservation or
videotape recording. New techniques have been developed for precise
quantitative measurement of finger and hand movement using the VPL ata
Glove. The comfortable light-weight
lycra glove is easily slipped onto the patient's hand and does not restrict
movement, or change of mechanics. Fiber optic sensors on the dorsum of the
hand and each digit 4 dynamically measure angulations (with 1 degree
resolution) of MP joints, the P joint of the thumb and the PIP joints of the
four fingers. The position
I
the hand in three dimensions is measured using a Polhemus tracking system with
3.3 mm accuracy for x, y, and z coordinates and Q.85 degree accuracy for
pitch yaw, and roll. Data from each of the sensors is sampled at
60 cpa, and rendered graphically in real time or stored in a file. Movement onset, acceleration,
velocity, and amplitude can be
measured. Frequency profiles for tremor analysis can be obtained using the fast
Fourier transformation. Complex
relationships between joints during
kinetic movements and tremor can be studied. Data from patients with various types of chorea and tremor will
be presented. Precise quantitative measurement of movement will be an important
contribution to assessment of tremor and chorea.
Having
repeatedly demonstrated the utility of using emerging interface technology for
the evaluation of clinical conditions it became necessary to validate the
methodology by comparing it to the current “subjective” methods used by
clinicians.
The
following study was done to help establish the utility of these new transducers
Having
established the clinical validity of the technology and technique
A
significant case of where the use of this new technique was actually utilized
to validate a new clinical technique was when a neuro surgeon was developing an
experimental technique and was in need
of an objective measurement of the
efficacy of his intervention.
It is interesting to note a
bit of irony at this point.
The
validation of the data glove (along with EMG) as a tool for quantitative
assessment for Parkinson’s tremor was done by the dept of neurology . The following year the same technique was utilized to validate
a new treatment modality. The glove was used by the neuro surgery dept to validate
the human performance of patients undergoing an experimental treatment which developed an intentional lesion in the
brain of an awake patient. The patient
was required to perform a series of motor tasks, and their performance was then
used to refine the treatment in real time. The following images show that this
method of quantitative analysis was of sufficient utility to be utilized in an
invasive procedure in real time.
The
data at the center shows the activity of forearm muscles in a “flexion
extension” task. The data clearly shows that
Motor activity is “more” normal in the later tracing (ie after the
lesion).
Again this is an illustration of the ability of the
physio-informatics system to have a wide range of utility.
Glove
in icu
A
patient from the neurology service was admitted to the icu with a stroke which
left them mostly paralyzed the data
glove was used as a tool to establish a communication channel which could be
reliably interpreted as actually conveying the intent of the individual
Glove
in coma
A
young girl who had a traumatic brain injury
was emerging from a coma. There was a need to establish her cognitive
level so as to optimize her therapy. As with the stroke case a glove was used
as a communication tool which could be reproducibly manipulated by the girl to
show that she was in fact aware and processing information. The interesting
thing about this experiment was that it was the first experiment which
established the use of a feedback system. That is that the girl ha to listen to
the output of her actions to ascertain whether or not she had achieved a
goal/target state
A
series of technology demonstrations was then initiated to establish the wide range of applicability
of this method and technology
As
the testing expanded the complexity of the instrumentation was also increasing
They
are listed below
Controlling virtual objects for rehab
This
series of exploratory adventures led to development of a system of hardware,
software, and analytical methods which were first described in the NRW paper
Another
very important finding was that the ability for individuals to actually control
something in their own environment was a very motivating activity and that
there was an increase on volitional activity on the part of the individuals to
engage their environment
The
nature of diversity encountered when dealing with individuals with various
disabilities or interface needs became evident as the experimental nature of
these systems begin to extend to larger and larger numbers.
While
it was the case that the systems used to date were for the most part off the
shelf interface devices which had been developed for other uses, it was also
observed that a wider application of these techniques was possible only if the
system could be individualized and customized to meet the requirements of the
actual users.
Given
the variability of the capacity of the human information processing it was
determined that a very modular, component based system was necessary. The need
to “reflect the complexity of the human systems and the ability to adapt to
different channels for input and output was necessary.
The
ability to address individual data channels which contained the information coming from or going to the
individuals was seen as the most fundamental issue to be addressed
The
following case illustrates the process
of using the ref arc and iteratively refining the “interface system”
Ashley
– the migration from the bio-muse and meme
--- to tng1 and neat
ASHLEY
In
a birth accident, a C1 with some brain stem involvement [pictures] she can move
here head, she is vent dependent. When we first started hanging out with
Ashley, her way of interacting with the world was with the head stick (which
she was good at and proud of). Her grandmother wanted us to experiment to see
if we could identify some other avenues for Ashley to start interacting with
computers and other things.
One
of the first things we did was to take things she already knew (stick) and to
adapt it to an existing interface (pen mouse). Just very quickly, by
understanding the technology we were able to turn what Ashley actually could do
into something where she could interact with the computer. But really that was
limited function.
What
we really wanted to do with Ashley was to start exploring the biomuse at that
time to control the computer. This is some photographs when Ken Kashuahara of
ABCs Americal Agenda cameout and we showed that we were able to plug in Ashley,
only the second time she had ever even seen this technology) and using her
facial muscles she was able to navigate around in a virtual environment. Direct
muscle (bioelectric) control in a Virtual environment. That was done by over
time adapting a series of interfaces to her face which picked off very specific
muscles from her face and to use those as independent data channels. The idea
was that as Ashley had control of her face we would use that as her sort of her
"fingers" to give here differential control into the computer.
So
here’s a system [photo] looking back over head we had to build software called
Neat Dos for having different muscle signals cause different outputs (gesture
recognition?). Then some of the engineers I was working with built a remote
control car that had a camera (radio shack remote control car and radio shack
transmitter) and Ashley with a set of VR glasses and a TV seeing what camera
was seeing was able to drive this car out onto her back porch and to start
playing with her niece and nephew. And so this car actually became an extension
of Ashley's intentional actions with the world. And this is where we got into
Biocontrolled telepresence where a direct interface from muscle activity is
used to control a telerobot interacting in a fairly complex environment (ie
with her niece and nephew). It was very interesting to watch her play as she
would chase them around and run into them and you could really tell by looking
at the car that she understood the paradigm of what was going on.
Another
extension for Ashley was that at the time we had access to a surgical robot
that was for laproscopic surgery for positioning the camera and we bypassed its
interfaces and we created a system that she with her facial muscles, which she
had been refining, we were able to put a paint brush at the end of the surgical
robot, give her some paint and allow her to use her face to use the surgical
robot to create art. Biocontrolled telerobotic arm capability.
Ashley
was a good test case as she had a great family environment, personable,
excited, easy to work with.
So
we extended that one step further to the helicopter. We took a virtual IO pair
of glasses coupled with Ashley's ability to move her head left, right and up
and down and there was a camera controller on a helicopter which when she would
move her head it would move the camera positions around while it was flying. SO
Ashley was one of the first biocybernauts in controlled unmanned aerial vehicle
payload controller which we are currently under contract with DARPA to develop
for the military.
So
the idea was that with Ashley and the concept of the flow of information we
were able to take her from where she was and look at her brain function, input
information into her, allow her to process it and then output something into
the world. And Ashley then became, because of the way we were presenting
information and the way we were able to acquire information was one of our test
cases for the model of perceptual psybernetics and.
The
main focus of the case examples will be to show the utility of this reference
architecture. Especially to show the capacity for enhanced perceptibility and
expressability that may be achieved through an intelligent application of this
physiologically robust interface systems model.
NeatTools
What
is NeatTools?
NeatTools
is a powerful visual programming environment that allows users with
disabilities to control and communicate with a computer. It operates in
conjunction with hardware devices created specially for this purpose. A
disabled individual generates some deliberate movement under her
control--perhaps moving a cheek muscle; this movement is then detected by the
device, which transmits signals conveying the information to the computer.
NeatTools then translates this information into some form that the computer can
interpret, and generates some meaningful output – perhaps a mouse click, or a
cursor move. The software thus allows disabled individuals to use whatever
physical capabilities are available to them in order to interact with a
computer, to improve their quality of life. NeatTools can permit quadriplegics
to type, draw, or play games; to use the World Wide Web and e-mail; to control
devices in their environment such as lights, stereo, or TV; and more generally,
to interact with others. Capabilities that the able-bodied take for granted are
made available to the disabled through this sophisticated computer program.
Individuals who have previously had to depend on others to do nearly everything
for them can gain some control and independence with the help of NeatTools.
Who
Developed NeatTools, and When?
The
story of NeatTools goes back to 1990.At Loma Linda University in California ,
Dr. Doug Will led a Neurology Research Group, which Dave Warner joined soon
after becoming an MD/PhD. student there in 1988.The group researched ways that
a human/computer combination might offer valuable diagnostic information, or
might explain more about how the human brain functions. The group identified
and experimented with the newest developments in technology for this project.
In 1990, they acquired a special glove with optical fibers in the lycra fabric,
which they used to investigate the effects and treatments of neurological
diseases. A corporation called VPL Research had created the glove, called the
Data Glove, and donated the $10,000 device to the lab for exploration of the
medical potential. Dr. Will’s lab determined that the glove could detect small
changes in hand or finger position; it could thus measure hand function in
patients with Parkinson’s disease or other motor disorders. The glove could
also help physicians to diagnose whether a patient had Parkinson’s or whether
that patient suffered from a similar disorder, such as essential tremors.
Through the precise quantitative feedback, the glove could also aid physicians
in understanding the effects of medication on the functioning of Parkinson’s
patients. And it could be used in rehabilitation, to help patients develop
skills in particular motions.
For such a glove to communicate with a computer, an intermediary device is needed to translate the glove’s electrical signals to signals that a computer could understand. The lab used the glove in conjunction with a workstation called BioMuse. This instrument had just been developed by a Palo Alto company named BioControl Systems; though the cost was approximately $20,000, the Neurological Research Group was one of four locations given free use of the equipment for medical and humanitarian purposes. BioMuse used electromyography (EMG) sensors to detect voltage differences that arise when muscles are flexed. It then transmitted these voltage measurements to a computer, and the software program converted the voltage differences into music. BioMuse was one of the first devices to allow any such bi-directional feedback between a patient and a computer.
At
the end of 1991, the Neurological Research Group was approached to see if they
could help a hospitalized patient – a baby named Crystal Earwood. At the time,
Crystal was 18 months; she had been paralyzed from the neck down in a car
accident. She needed stimulation, and she needed some way to interact with the
outside world. Dave Warner, still a medical student and a member of the
Neurological Research Group, took up the challenge to find out what the
technology could do for this patient.
Dave
found that 18-month-old Crystal could control a computer by moving her eyes.
BioMuse could transmit the voltage differences from her eye motions to the
computer. Her eyes became the equivalent of hands, transmitting commands to the
computer. Instead of generating music, the BioMuse system was modified by the
lab to give a graphical output. Crystal could then interact with displays on a
computer screen, effectively demonstrating the potential value of biocybernetic
technology for the severely disabled.
But
despite all the promise, one serious hurdle remained. Since Bio-Muse cost
around $20,000 and the lab had only one unit to use for research and
development, the equipment could not be left with Crystal, or with any of the
30 subsequent patients it was used with. For most of them, the BioMuse
technology proved highly effective. For some, it greatly improved opportunities
for communication and interaction. For others, the experience of generating
music significantly increased patients’ motivation to exercise weak limbs. But
the high cost limited BioMuse’s accessibility to the disabled patients who most
needed it. Two patients were able to afford to buy their own equipment; the
others were not so fortunate.
Making Affordable
Technology
The Needed Hardware
The
first breakthrough in this impasse came in 1995, when Salomo Murtonen, the
Finnish inventor of the Sound Chair, came to America to volunteer for the
project. A self-taught electronics engineer, Salomo committed himself to create
the equivalent of the Bio-Muse device at low cost. At first, he worked for next
to nothing, since the group had no substantial source of funds to pay its
volunteers. Salomo created a four-channel EMG interface that could take any
signal derived from muscle movements into the computer. The device was named
TNG-1 (Thing 1, from The Cat in the Hat by Dr. Seuss); TNG is short for
Totally Neat Gadget. Salomo produced TNG-1 with Radio Shack parts for a cost of
$200, far less than the cost of Bio-Muse.
Creating the Software
In
1994, the group had begun to work with a seven-year-old girl named Ashley
Hughes. As a result of a broken neck during birth, Ashley is a C1 quadriplegic,
paralyzed from the neck down, dependent on a respirator for breathing. She
could move facial muscles, and TNG-1 could transmit the EMG signals from her
facial movements to her 286computer.But then software was needed to make those
signals meaningful to the computer, and to display them on the screen. Joh
Johansen wrote a program called BioEnvironmental Control (BEC) to make the EMG
gesture signals usable for the computer, and to convert these to graphical
outputs. BEC was designed for Ashley’s facial capabilities; it allowed her to
express herself in rich and complex ways, using her body as a way to control a
computer. With these powerful technologies, Ashley played computer games, drove
a remote-controlled car, experienced her world remotely through a camera and
microphone mounted on a Styrofoam structure named Cindy Cyberspace, and
interacted with others in her environment. Now the group had created both
affordable hardware and software. But though TN-1 was inexpensive, it was not
free of problems. For one thing, the electrodes that detected the muscle
movement were not stable; and setting up TNG-1 was not easy for the families –
it could not just be left with them. Further development was needed, to make
the technology more stable and easier for family members to use.
Making the
Technology Easier to Use
With
the development of the TNG-1 interface device and the BEC software in
California, the capability to interact with a computer was now affordable for
the disabled. But TNG-1 depended on the use of EMG sensors attached to the
faces of quadriplegic patients, where they had some muscle control. Whether
they were mounted on cheeks, or foreheads, the EMG sensors would not work for
very long. They didn’t stick to the face well, coming loose easily with
repeated movement. To get around the problem posed by dependence on EMG
sensors, Salomo went to work to create a more flexible interface device, that
could receive a variety of types of sensory signals.TNG-2 could detect changes
in light signals resulting from cheek motions, since such a motion would
distort the light path to a light receptor, and thus create a signal. The
lights did not have to be attached to the patients’ skin, as did the EMG
sensor; the light sensors could be mounted from a hat, a helmet, or most
recently, from a pair of glasses. But light was only one possible type of
signal detectable by TNG-2, which was constructed to receive up to four
general-purpose analog inputs. The use of light signals had advantages over the
EMG approach, but raised new problems. Because photocells change their signal
levels when the brightness level changes in a room, the use of light signals
required frequent calibration, often beyond the capacities of a disabled
individual’s family. The group then experimented with signal sources other than
light – by using Hall Effect transducers to detect changes in magnetic field
and by using pressure transducers. Other approaches include using bend sensors
or tilt sensors. For Ashley Hughes, for instance, the group had been able to
use a tilt sensor on her head along with a pointing stick attached to her head.
With this combination, she could use a screen keyboard to type.
New and Better TNGs
TNG-1
and TNG-2 could each convey four channels of information to NeatTools. In real
terms, this meant that the system was limited to information from, say, four
muscles, or from four light detectors. Or the user would have to depend on two
TNG devices at once – thus requiring two available ports on the computer. With
the creation of TNG-3, 16 channels became available – 8 analog and 8 digital.
As of January 2000, the group is now testing a working prototype of TNG-4,
which has eight analog and 16 to 20 digital lines, each of which can serve as
input or output. This increased capability will mean that the parallel port can
be left alone for other functions, such as connecting a printer or a zip
drive.TNG-3 and TNG-4 have been developed by Edward Lipson and Paul Gelling at
Syracuse University
Neat becomes Really Neat becomes NeatTools
Similarly,
as computer technology has advanced, the group has been working on creating new
iterations of the software, to take advantage of the capabilities that the
newest developments of technology will allow. In 1993, the BEC software was
renamed Neat; it was written for MS-DOS operating system, then common for PC’s.
In 1996, a version of this software was created for a Windows 95 PC
environment; and in late 1996, work began on another version based on a
Java-like environment. This is the current version called NeatTools. It is
suitable for virtually any computer system – it can run on Windows 95 or 98 or
Windows NT; it can run on Unix, Irix, or Linux; it will be able to run on Macs
upon the availability of multithreaded operating systems. It can interface with
a much broader range of devices. Its capabilities include both Internet
connectivity and multimedia sound. A user can simultaneously develop, edit, and
execute programs in Neat Tools.
The
original DOS version of Neat and the first Windows version were created by Joh
Johansen in California. But there were many limitations to DOS, which affected
the capability of Neat. For instance, DOS does not support TCP/IP, so the DOS
version was limited in connectivity. DOS does not support multiple processes
occurring at once, so the DOS Neat program was slow. DOS does not support
multimedia. NeatTools was created by a Computer Science Ph.D. student at
Syracuse, named Yuh-Jye Chang; this program formed the basis for his Ph.D.
dissertation. Yuh-Jye defended his dissertation work in 1999, and is now at
Bell Labs in New Jersey. The group has been especially fortunate in having both
programmers. Before taking on the Neat Tools project, Yuh-Jye won the 1996 Java
Cup International Award for an earlier Java-based graduate project – the
Visible Human Viewer program. One of the major breakthroughs in Yuh-Jye’s creation
of NeatTools was the development of a Windows mouse driver, which would allow a
disabled individual to move a cursor in any direction with cheek or eye
motions. This complex program allowed a user to move the mouse, and to simulate
a mouse click. Before this, the group had to get hold of the source code for
each program a disabled user would employ, and rewrite the code so the disabled
individual could work with the program. With commercial software applications
like Microsoft Word, there’s no way to acquire the source code. So it was quite
a boon to have the general capability, in any Windows-type computer program, to
simulate mouse events. A subsequent program created by Edward Lipson allows the
user to move the cursor via a custom joystick mechanism. The software allows
for fine calibrations, since different quadriplegic users have differing kinds
and ranges of motion, as well as differing facial shapes and sizes. The
NeatTools software and some related application programs can be downloaded at no
cost from http://pulsar.org/NT/index.html.
The
basic conception and architecture for the software – from the original DOS
version to the latest NeatTools version – was laid out by Dave Warner. The
software had to be adaptable to the widest possible range of industry or
in-house devices, and to allow as many input channels as possible for users of
limited physical capabilities. To allow maximum flexibility and modification
for the needs of individual disabled users, the software is based on modules.
NeatTools now consists of approximately 200 different modules, not counting the
alphanumeric characters, and it is relatively easy to add modules. The program
offers visual programming, with a highly user-friendly graphical interface. A
user can create a simple NeatTools data-flow network by dragging a few modules
to the desktop and connecting them. Typically, a module has inputs on the left
and outputs on the right, and control inputs on top. A user can modify the properties
of a module by a right mouse click. There are keyboard modules, modules for
serial and parallel ports, modules for Internet sockets, calibrator modules,
graphical display modules, modules for arithmetic and logic operations,
multimedia sound modules, etc.
Because
he was aiming for speed, a high level of performance, and platform flexibility,
Yuh Jye chose to use the C++ program language instead of Java .When he began in
1997, the Java technology was not mature, and even today is not fast enough for
intensive real-time tasks such as compression or decompression of video data.
NeatTools consists of over 50,000 lines of C++ code. The program adopts the
advantages of Java – especially its ability to work on different types of
computers, by utilizing a Java-like Cross-Platform API (Application Programming
Interface) – a very thin layer which provides an interface to the user’s
computer operating system, and to the C++ core of Neat Tools. The API unhooks
an application, such as Netscape or Word, from dependency on the computer’s
operating system or on Windows. Thus the architecture of NeatTools consists of
three layers:(1) the computer’s operating system, and the C++ programming
language (2) the Java-like cross-platform API; and (3) the NeatTools application
layer. This combination allows NeatTools to combine the best of both worlds
.NeatTools incorporates the speed and efficiency allowed by the high
performance levels of C++, along with Java’s ability to run on different
platforms. The thin layer of the API also helped provide the ability to
synchronize many operations at once; such functionality would have been much
more difficult to create in C++ alone. NeatTools provides the advantages of
Java without using Java.
Hardware:
TNGs
and Widgits
Intro--
design philosophy; microcontroller technology; ...
TNG-1
-- EMG
TNG-2
-- 4 analog input channels
TNG-3--
16 input channels (first production version)
TNG-4
-- ~32 channels bi-directional
Widgits
-- sensors and transducers
Representative
Applications
Anthrotronics – Human Instrumentation
Systems
Anthrotronic
systems designed for interactive information exchange continue to evolve
Applying
first principles of physio info metrics facilitates design innovation for
operational refinement of the evolving interface system, thus enabling the
demonstration of an integrated system of human to computer input devices with
the multisensory rendering systems for an interactive, experiential environment
optimized for intelligent interaction with information. Research areas and
methodology required to Grok (– to intentionally seek to comprehend at a
profound level) the quantifiable information flux capacity of a physiologic
system necessary knowledge in these following areas:
Neuro-physiologic
restraints and limits of the computer to human linkages, Psycho-physiologic
capacity for optimal cognitive function within an extended perceptual
environment, Psycho-motor function for simultaneous interaction with multiple
human to computer devices, Functional integration of relevant human-to-computer
input devices into the system, Multisensory rendering systems integrated into
the system.
The
development of hardware and software systems based on principles derived from
the reference architecture, and implementation of such systems in real world
settings
The
combination of applied physio info metric principles with an operational
notational system creates a research tool capable of mapping the time evolution
of information propagation through a perceptual cybernetic system. Focus will
be to develop a more generalized capacity to address the interface issues of
human-computer interaction requirements.
Physiologically
Oriented Interface Design: the next paradigm of human-computer interface will
optimize the technology to the physiology -- a biologically responsive
interactive interface. paradigm of interface technology is based on new
theories of human-computer interaction which are physiologically and
cognitively oriented.
Research
in human sensory physiology, specifically sensory transduction mechanisms,
shows us that there are designs in our nervous systems optimized for feature
extraction of spatially rendered data, temporally rendered data, and textures.
We will develop these interface techniques and technologies consistent with the
basic neuroscience issues of modality, duration, intensity, distribution,
frequency, spatial displacement, contrast, inhibition, threshold, adaptation,
transduction, conductance and transmission (to name a few)
The
ability to achieve the integration of a set of advanced human-to-computer input
devices into a single interface system and demonstrate data fusion to enable
meaningful correlations across various input modalities will significantly
enhance progress toward this end. Viewing the entire body as a perceptual and
expressional technology opens up possibilities for exploiting the heretofore
untapped richness and greater volumetric potential of its informatic
capacities. Hence, one could propose to develop an interactive environment
incorporating new ways to render complex information to the user by optimizing
the interface system to match the human nervous system’s ability to transduce,
transmit, and render to consciousness the necessary information.
Efforts
have been to research, prototype, and demonstrate the implementation of a data
analysis subsystem designed to enhance the ways that relevant data may then be
rendered optimally to the operators sensory modalities, utilizing such
techniques as linear and nonlinear multivariate analysis tools for the
processing of multiple data sets in a variety of ways, including graphical
analysis (phase portraits, compressed arrays, recurrence maps, etc.) and sound
editing (mixing, filtering ). Interface was designed so as to optimize the
salience and content of data sets. Some data which conventionally would be
displayed visually might be processed so as to be perceived (in the suit) in a
tactile or auditory manner Synesthetic environments rendered with this
technology make it possible to feel the sound, see the pressure and ultimately
be able to reconfigure the rendering parameters of the interface based on the
specific elements of a situation. Seeing colors may be more appropriate in one
context whereas hearing them may be more suitable for another; many factors
will determine the tailoring of rendered data: which data will be shunted to
which renderer. Novel interface controllers are essential here.
Using
this reference architecture one can demonstrate interactive environments that
combine new ways to render complex information with advanced computer to human
input devices, such that it renders content specific information onto multiple
human sensory systems giving a sustained perceptual effect, while monitoring
human response in the form of physiometric gestures, speech, eye movements, and
various other inputs, as well as providing for the measurement of same.
A
series of experiments we conducted which
--- assessed the limits of usability of
traditional input devices such as mouse, joystick and keyboards to determine
when interface complexity precludes their use as primary inputs.
---
researched and evaluated the users ability to integrate several input systems
so as to have a multiplicity of simultaneous interaction options.
---
researched the capacity for filtering and combining data streams from the
various human to computer input devices.
---
researched the capacity for developing user defined gestures for controlling
various parameters of the interface system.
---
researched the functional integration of relevant human-to-computer input
devices into the system.
---
researched various multisensory rendering systems integrated into the system.
---
researched an integrated system of human to computer input devices with the
multisensory rendering systems.
---
researched an interactive, experiential environment optimized for intelligent
interaction with information.
This
increased throughput is achieved by extending the perceptual dimensionality of
information presented to the human and enhancing the expressional capacity of
the human to convey intent to the informatic system. -Quantitative human
performance assessment tools for clinical, educational and vocational
applications have been developed and refined -Interactive systems for the
disabled which empower them to participate more actively in their own
environment
The
reference architecture presented has the necessary features to map the flow of
information in an interactive human computer interface system. The reference
architecture has the necessary complexity to be able to account for the
physiological issues in an interactive human computer interface system.
A
system of hardware, software and methodology was created
Universal Interfacing
System for Interactive Technologies in Telemedicine, Disabilities,
Rehabilitation, and Education
Edward Lipson1,
David Warner2, and Yuh-Jye Chang3
1Department of Physics
and 1-3Northeast Parallel Architectures Center, Syracuse University,
Syracuse, NY 13244; and 1,2MindTel LLC, 2-212 Center for Science and
Technology, 111 College Place, Syracuse, NY 13244
A modular
hardware and software system for human-computer interaction is described that
allows for flexible, affordable interfacing of people, computers, and
instruments. The approach is illustrated with an application in the
disabilities area. Other application areas are outlined.
- Introduction
Emerging methods for human-computer interaction [HCI; 1] offer revolutionary opportunities to advance healthcare and quality of life, particularly as the power, functionality, and affordability of computers continues to soar. In particular, the advent of wearable computers calls for new types of interfaces, since the users are typically not desk-bound. Further, for people with disabilities who are unable to use a keyboard and/or mouse, the need for alternative interfaces is compelling. Clinical environments can enjoy improved efficiencies and outcomes, as new ways evolve to interface patients, caregivers, and instruments to computers and networks.
Our group has been developing powerful, low-cost technologies combining modular software and hardware that accommodate expressional gestures and perceptual modalities as essential parts of the interface. These systems allow for adaptive rapid prototyping in which practically any input to the computer can be mapped to appropriate actions and outputs.
- Methods
The NeatTools visual-programming environment allows rapid prototyping and implementation of HCI and other dataflow applications, in conjunction with custom sensors, mounting hardware, computer interface boxes (TNGs), and clinical/scientific instruments.
- NeatTools
Software
NeatTools constitutes a visual-programming and runtime environment that produces fine-grain dataflow networks for data acquisition and processing, gesture recognition, external device control, virtual world control, remote collaboration, and perceptual modulation. The design goals of NeatTools have been to make it simple, object-oriented, network-ready, robust, secure, architecture neutral, portable, high-performance, multithreaded, and dynamic. The program and representative applications are downloadable from www.pulsar.org. NeatTools can readily accommodate custom interface devices, or commercial devices including clinical instruments. Figure 1 shows two simple NeatTools programs. For a full-fledged application program, see the section below on the JoyMouse Network.
NeatTools is written in C++ but built on top of a thin-layer Java-like cross-platform C++ application programming interface (API), which operates presently on Windows 95/NT, Unix (Sun), Irix (SGI), and Linux. In due course, Macintosh will be supported, once its multitasking, multithreaded operating system is released (note that it can run provisionally on a Mac-based PC-simulator, such as Connectix Virtual PCÔ ), along with appropriate C++ development tools.
Currently, NeatTools includes serial, parallel, and joystick port interfaces; multimedia sound; MIDI (Musical Instrument Device Interface) controls; recording and playback; Internet connectivity (sockets, telephony, etc.); various display modalities including for time signals; time generation functions; mathematical and logic functions (including a state machine module); character generation; and a visual relational database system including multimedia functionality. Keyboard and mouse events can be received or generated via Keyboard and Mouse modules. This allows, among other things, the user to control a graphical user interface by alternative input devices that in effect simulate keyboard and mouse events. Data types in NeatTools include integer, real, string, block, byte array, MIDI event, and audio or video streams. NeatTools allows the visual programmer to package a dataflow network inside a container module that constitutes a reusable "complex module" with simple overt appearance. This procedure can be iterated to accommodate several layers of hidden complexity.
NeatTools modules provide multithreaded, real-time support. Editing and execution are active concurrently, without need for compilation steps. This generally accelerates system design, and facilitates rapid prototyping and debugging. To construct a dataflow network, the user drags and drops modules (objects) from toolboxes to the desktop and then interconnects them with input/output and control/parametric lines. Properties of the desktop and many of the modules are set via a right-mouse-click. In this way, users are in effect developing elaborate interface programs without having to know C++ or the fundamental structure of NeatTools, or indeed having to write any textual program code at all. On the other hand, the system is open, so that experienced programmers can develop external modules by following instructions in an online developer’s kit. External modules can be loaded into the system at runtime, or arranged to preload automatically. The NeatTools executable development program, while massive in terms of source code (~40,000 lines of C++), is compact; the downloadable compressed archive file is about 600 kilobytes in size, so it easily fits on a diskette along with a compressed archive (under 100 kilobytes) of representative "*.ntl" files.
- Interface
Devices
The system hardware consists of mounting components, sensors, serial interface boxes, computer, and optional output interfaces and devices. Our current electronic interface module (TNG-3; www.mindtel.com/mindtel/anywear.html) accommodates up to 8 analog and 8 digital (switch) sensors and streams the data at 19,200 bits per second to the serial port of a computer. Connections are made via standard stereo and mono plugs. The heart of TNG-3 is a programmable microcontroller integrated circuit [2], a type of computer-on-a-chip commonly used in industrial and office automation, and in automotive, communication, and consumer electronics under the general rubric of embedded control systems. The microcontroller in TNG-3 is programmed in assembly language for optimal performance. TNG-3 requires no batteries or wall transformer, as it derives 5 volt power for the onboard circuitry and sensors (requiring only modest power) by exploiting some of the unused serial-port lines—a technique commonly used to power a serial mouse on a PC.
- Sensors
Among the sensors we have used are switches, cadmium-sulfide (CdS) photocells, Hall Effect transducers (magnetic sensors), rotary and linear-displacement potentiometers, bend sensors, piezo film sensors, strain gauges, and custom electroconductive-plastic pressure sensors. Most of these sensors are inexpensive, some costing under a dollar and some costing but a few dollars. Certain types (Hall Effect and capacitive) require preamplifiers and/or signal processing electronics, which increase the cost, but not unduly.
- Results
The most substantive technical result of our work to date is the development of the NeatTools system along with the TNG interfaces and sensors, as described above. We have begun to apply the core technologies in a number of key application areas.
- Disabilities
Applications
For illustration, we describe the types of systems we developed for Eyal Sherman, a member of our team who, is a brainstem quadriplegic, unable to move his head or to vocalize. He is currently a senior at Nottingham High School in Syracuse. We have enabled Eyal to precisely control mouse motion, and thereby control graphical user interfaces, such as Windows 95. Eyal and his family have achieved independence in using this system; his mother is able to set up the hardware and software routinely in a matter of minutes.
The primary interface device is a chin joystick, extracted from an inexpensive game controller, mounted to a curved support rod, which is clamped in turn to the wheelchair headrest post, thereby allowing the device to be rotated away when not in use. To allow easy mounting and adjustment of sensors near Eyal’s expressive facial regions—mainly cheeks and forehead—an industrial designer on our team, Michael Konieczny, built lightweight adjustable mounts that attach to eyeglasses. Currently we are using small switches as the expressional sensors, but we have also used Hall Effect transducers (together with tiny rare-earth magnets) and photocells to detect facial gestures.
- JoyMouse
Network
An application program demonstrating the considerable power of NeatTools is the JoyMouse dataflow network (Fig. 2), which Eyal and other youngsters with quadriplegia have been using with good results. For details, manual, images, and downloads, see http://www.pulsar.org/neattools/edl/joymouse_docs/JoyMouseManual.html. This uses a modest fraction of the channel capacity of TNG-3 (2 of the 8 analog inputs; and currently 3 of the 8 digital inputs). The JoyMouse application uses advanced features of NeatTools including logic gates, multiplexers and demultiplexers, encoders and decoders, various timing and mathematical operations, and sockets (here in "localhost" mode so that two windows on the same platform can communicate). The network is shown here both in developer mode and in user mode, wherein editing is blocked and only essential regions of the network are visible.
Figure 2 includes a graph of the available relationships between mouse-cursor velocity and analog-joystick displacement. For all three functions, there is a dead band, or free-play zone, near the origin so that the mouse cursor is not subject to jitter when the joystick is physically at rest. The linear relation offers essentially proportional control. The nonlinear relations—quadratic (necessarily inverted for negative displacement) and cubic—offer fine control for up to about half-maximal displacement, and rapid travel with larger displacements. In most applications, the cubic function offers the best performance. Various parameters (pertaining to gain, resolution, etc.) can be set or modified using sliders while remaining in user mode.
The network also accommodates input from three switches: a) a left cheek switch for left-mouse-button; b) a variable-use right-cheek switch for right-mouse-button, enter-key or backspace-key; and c) a forehead switch to dynamically select action mode of the right-cheek switch. Alternatively, the switches could be replaced with analog sensors for which thresholds would be set with sliders within the JoyMouse program. Calibrator modules are included in the JoyMouse program, as in many other NeatTools programs, to automatically adjust to the signal range for analog inputs.
The network as shown can be minimized, once the Enable button has been activated, so that the operating system desktop becomes fully available to other application programs while the JoyMouse runs in background. An optional small satellite window (Fig. 2), a related NeatTools application, can remain visible to display the state of essential options that are under dynamic control of the user; this is made possible by using socket modules to communicate locally between the JoyMouse main window and satellite window. The user can toggle, for example, between mouse click and drag modes by a "smile" gesture (both cheek switches activated for 1 second).
By using this the JoyMouse in conjunction with low-cost commercial utility programs, including an onscreen keyboard (Fitaly™ from Textware Solutions, sometimes with their InstantText™ program for word/phrase prediction and abbreviation expansion), Eyal has been able to a) type text, b) generate speech, c) dial in to a server, d) invoke and use Web browsers and other application programs, e) compose and send e-mail messages, f) play video games alone or with others, g) operate remote controlled cars, h) draw sketches, and i) participate in science experiments and data analysis at school. Other people with severe disabilities have also successfully used the JoyMouse system and other applications with good success, for example children with cerebral palsy.
- Education,
Rehabilitation, Telemedicine, and Defense Applications
- Education
NeatTools has many possible applications and roles in the education arena. We have mentioned the use of NeatTools to allow students with disabilities to participate actively in science laboratory activities. More generally, NeatTools lends itself well to student projects in the classroom, laboratory, science fairs, etc. Moreover, NeatTools can be used for training and prototyping in an industrial or community college setting. Because NeatTools can accommodate diverse external modules, the environment can be adapted to a wide range of simulation applications, notably in medicine. With the increasing use of sophisticated technology in healthcare, environments like NeatTools can be expected to play an increasing role in practice and in training of healthcare practitioners. Medical students, interns, and residents can benefit from the rapid prototyping capability and flexibility of NeatTools. While prior programming experience is clearly of benefit for those who wish to write applications in NeatTools, it can serve, on the other hand, as a training ground for practitioners and others who want to get their feet wet in programming before learning conventional languages like C and C++. The immediacy of the results in this visual programming/runtime environment, without need to cycle through edit/compile/execute cycles is clearly an advantage.
In limited testing, we have observed that schoolchildren are often able to grasp the essentials of NeatTools programming quite rapidly. For example, at SIGGRAPH 98 in Orlando, a number of schoolchildren came to our exhibit in the sigKIDS area. Typically, after the first daytime session, they downloaded NeatTools at home the first evening, proceeded to develop applications of their own, and then returned to our site the following morning to continue their programming and obtain more advanced training. Some of the programs they wrote were quite remarkable.
- Rehabilitation
In the rehabilitation field, our devices have been used for monitoring range of motion, for example at an elbow or knee joint, during exercises, and other aspects of human performance. Our systems are currently in use at two rehabilitation centers, namely the Sister Kenny Institute at Abbott Northwestern Hospital in Minneapolis and at East Carolina University Medical Center. They are currently being implemented at the Extended Care Facility of Oneida City Hospitals in Oneida, NY in a context focused more specifically on monitoring of care of residents.
- Telemedicine
Development of external modules for digital signal processing, digital image processing, and a host of other advanced modalities will expand the scope of NeatTools for clinical applications, basic research, and education and training. Areas of telemedicine that we anticipate would be well served by NeatTools included telerehabilitation, teleradiology, and general remote patient monitoring including home healthcare, particularly for the elderly still living at home but in need of continual observation. NeatTools already includes a module for the Welch Allyn Vital Signs MonitorÔ . The Internet socket feature of NeatTools, in conjunction with its audio (and soon video) codec, recording, and database functions, already provide base functionality for telemedicine applications.
- Defense
Another new project area for our HCI technologies concerns landmine detection and related applications involving wearable computers and distributed robotics (our BotMasters project, funded by DARPA). NeatTools and interfaces like ours can facilitate the signal processing and alerts in such critical real-time environments. Given the scourge of 100 million landmines on our planet, often from conflicts settled long ago, we hope that our technology can help reduce this nightmare while affording maximal safety to those engaged in this dangerous task.
- Conclusion
Our work is based on a systems approach wherein we have developed modular HCI hardware and software that is customizable, scalable, and extensible. Although most of the core functionality is in place, NeatTools remains under development. Improvements in the visual interface for the end user are needed. Expanding and enhancing the documentation is now a major priority. Much of the functionality and design of our software and hardware has been introduced according to the real needs of users like Eyal, and this will continue as these systems evolve.
Dave
Warner MD
Medical
Neuroscientist
Dir.
Medical Intelligence
MindTel
davew@well.com
www.pulsar.org
Warner,
a medical neuroscientist, has an MD from Loma Linda University and is the
director of the Institute for Interventional Informatics and has gained
international recognition for pioneering new methods of physiologically based
human-computer interaction. Warner's research efforts have focused on advanced
instrumentation and new methods of analysis which can be applied to evaluating
various aspects of human function as it relates to human-computer interaction,
this effort was to identify methods and techniques which optimize information
flow between humans and computers. Warner's work has indicated an optimal
mapping of interactive interface technologies to the human nervous system's
capacity to transduce, assimilate and respond intelligently to information in
an integrative-multisensory interaction will fundamentally change the way that
humans interact with information systems. Application areas for this work
include quantitative assessment of human performance, augmentative
communication systems, environmental controls for the disabled, medical
communications and integrated interactive educational systems. Warner is
particularly active in technology transfer of aerospace and other defense
derived technologies to the fields of health care and education. Specific areas
of interest are: advanced instrumentation for the acquisition and analysis of
medically relevant biological signals; intelligent
informatic
systems which augment both the general flow of medical information and provide
decision support for the health care professional; public accesses health
information databases designed to empower the average citizen to become more
involved in their own health care; and advanced training technologies which
will adaptively optimize interactive educational systems to the capacity of the
user. Selected Publications are:
1.
Warner D, Rusovick R, Balch D (1998) The Globalization of Interventional
Informatics Through Internet Mediated Distributed Medical Intelligence, New
Medicine (in press)
2.
Warner D, Tichenor J.M, Balch D.C. (1996) Telemedicine and Distributed Medical
Intelligence, Telemedicine Journal 2: 295-301.
3.
Warner, D., Anderson, T., and Joh Johannsen. (1994). Bio-Cybernetics: A
Biologically Responsive Interactive Interface, in Medicine Meets Virtual
Reality II: Interactive Technology & Healthcare: Visionary Applications for
Simulation Visualization Robotics. (pp. 237-241). San Diego, CA, USA: Aligned
Management Associates.
4.
Warner, D., Sale, J., Price, S. and Will, D. (1992). Remapping the
Human-Computer Interface for Optimized Perceptualization of Medical
Information, in Proceedings of Medicine Meets Virtual Reality. San Diego, CA:
Aligned Management Associates.
5.
Warner, D., Sale, J. and Price, S. (1991). The Neurorehabilitation Workstation:
A Clinical Application for Machine-Resident Intelligence, in Proceedings of the
13th Annual International Conference of the IEEE Engineering in Medicine and
Biology Society. ( pp. 1266-1267). Los Alamos, CA: IEEE Computer
Society
Press.
Basic neuroscience
The following abstracts
demonstrate the application of dynamical analysis to physiological signals and
show that it is possible to characterize abnormal electrophysiological rhythms
as low dimensional attractors.
n Sale EJ, Warner DJ, Price S,
Will AD. Compressed complexity
parameter. Proceedings of the 2nd
International Brain Topography Conference., Toronto, Ontario. 1991
n
Warner
DJ, Price SH, Sale EJ, Will AD.
Chaotropic dynamical analysis of the EEG. Brain Topography. 1990.
n
Warner
DJ, Price SH, Sale EJ, Will AD.
Chaotropic Dynamical Analysis of the EEG. Electroencephalography and Clinical Neurophysiology. 1990.
n
Warner
D, Will AD. Dynamical analysis of EEG:
evidence for a low-dimensional attractor in absence epilepsy. Neurology. 1990 April;40(1):351.
The following abstract introduces the possibility of quantitatively
correlating movement related potentials recorded over the scalp with complex
motor tasks using human-computer interface technology
Warner
DJ, Will AD, Peterson GW, Price SH, Sale EJ, Turley SM. Quantitative motion analysis instrumentation
for movement related potentials.
Electroencephalography and Clinical Neurophysiology. 1991;79:29-30.
Clinical neuroscience
The
basic problem being addressed by the following abstracts is that clinical
research involving neurological disorders is severely limited by the inability
to objectively and quantitatively measure complex motor performance. Large double-blind randomized controlled
trials of novel therapies continue to rely on clinical rating scales that are
merely ordinal and subjective. In
addition, research on the basic neuroscience of motor control is greatly
impeded by the lack of quantitative measurement of motor performance.
n
Will
AD, Sale EJ, Price S, Warner DJ, Peterson GW.
Quantitative measurement of the “milkmaid” sign in Huntington’s
disease. Annals of Neurology.
1991;30:320
n
Warner
DJ, Will AD, Peterson GW, Price SH, Sale EJ.
The VPL data glove as an instrument for quantitative motion
analysis. Brain Topography. 1990.
n
Warner
DJ, Will AD, Peterson GW, Price SH, Sale EJ.
The VPL data glove as an instrument for quantitative motion
analysis. Brain Topography. 1990.
n Will AD, Warner DJ, Peterson
GW, Price SH, Sale EJ. Quantitative motion analysis of the hand using the data
glove. Muscle and Nerve. 1990.
n
Will
AD, Warner DJ, Peterson GW, Sale EJ, Price SH.
The data glove for precise quantitative measurement of upper motor
neuron (UMN) function in amyotrophic lateral sclerosis (ALS). Annals of Neurology. 1990;28:210.
n
Will
AD, Warner DJ, Peterson GW, Price SH, Sale EJ.
Quantitative analysis of tremor and chorea using the VPL data
glove. Annals of Neurology.
1990;28:299.
Therapeutic potential of
human computer interface
Warner
DJ, Will AD, Peterson GW, Price SH, Sale EJ.
The VPL data glove as a tool for hand rehabilitation and
communication. Annals of Neurology.
1990;28:272.