Up2me

What exists

The physical space…

The functional space

The operational space

The utilization dynamics

Spatial structural cyber social interaction dynamics

Track the flux

Flux is a quantitative flow of information

We instrument for fields,,, ,,emf

We instrument for activity of users

Ambient audio sensing

Tele controlled focal audio imaging

Infra and ultra

Voice stress analysis

Bi direction controls of audio field…..

Spatial damping and sonic enhancement

Bio-suit for tracking…

Personal multi person int transponders,,,

Bio signal

Gestural biometrics

Poly physiometric data streaming

Multi parameter poly physiometric

Web meme pull

Though the sheer volume of information that requires understanding by warfighters has increased dramatically, the complexity of that information and the associated processing by computers and humans has similarly increased. This leads to the conclusion that for any complex phenomenon, the large number of relevant input systems and techniques creates an interface problem with exponential proportions. It is therefore critical that technologies be researched and demonstrated for the representation and interpretation of multiple, diverse data sources, and for human interpretation of these data sources. Whereas filters and summarization (roll-up) might adequately accommodate the volume, no amount of such measures can help with the complexity. Consider the complexity of an operation represented as a polyhedron. The number of notional facets remains fixed regardless of how it’s turned. However not all facets are always available for attention. If a warfighter is faced with continuously understanding, and acting on as required the scope of activities represented by each facet, then a new paradigm for human perception of complex data and interaction with that complex information system is required. Research is therefore required to understand the physiologic carrying capacity of information in the human nervous system for sustained, volitional interaction within an experiential information environment. This research should lead to interface systems with interactive representations, with humans in some cases instrumented to enable optimal perception and sensibility of the experiential interaction with information. Traditional, display-centric HCI approaches are already insufficient to deal with the complexity and dynamicity of the information. They furthermore provide insufficient understanding to the human in order to accomplish the transformation from data through information to actionable knowledge.

Current research shows that physiologic mechanisms restrain the information carrying capacity of any particular channel; audio, for example. Information systems require the ability to remap cognitive and volitional expression to other channels in order to overcome the limitations of any particular channel. This use of multiple channels will enhance the sensibility, perceptibility, and express-ability of complex information and interactions in an experientially rich environment. This was the unrealized goal of virtual reality research.

However, what is really required are dynamic, contextually sensitive, interactive, representational systems for supporting time-critical decision making given highly complex information assessment environments. One approach may be to utilize unconventional spatial representations to more accurately convey the high dimensional nature of the operational environment; and to provide a perceptively (by humans) valid analog of the multi-directional (computer-to-human-to-computer, and human-to-human), dynamic information flow.

This paradigm of interface technology is based on new theories of human-computer interaction that are physiologically and cognitively oriented. This emerging paradigm of human computer interaction incorporates multi-sense rendering technologies, giving sustained perceptual effects, and natural user interface devices that measure multiple physiological parameters simultaneously and use them as inputs. Biologically optimized interactive information technology has the potential to facilitate effective communication. This increase in effectiveness will impact both human-computer and human-human communication through enhanced ability to express.

Optimizing the human computer interface will rely on the knowledge base of physiology and neuroscience. That is, the more we know about the way we acquire information physiologically the more we know the optimum way for a human to interact with intelligent information systems. Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. Models of information processing based on the capacity of these neurophysiologic structures to process information will help our efforts to enhance perception of complex relationships by integrating visual, binaural, and tactile modalities.

Background

The capacity of our sensor assets and information systems to receive, process, and transmit massive amounts of information is continually increasing. Current efforts to develop new human-computer interaction options to exploit these ever increasing information opportunities are ongoing. The development and experimental utilization of the ACDE is intended to greatly enhance our ability to intelligently interact with this increasing flow of information by creating an enriched experiential interactive information environment enabled with communication and computation resources.

In support of this, initiatives are ongoing to sense and integrate various user derived data streams, including bio-electric signals (from eye, muscle and brain activity) for interaction analysis

The interactive bio dynamics captured through user tracking will be used to characterize and exploit the advantage of psycho-physiologic signal processing of the human nervous system (perceptual psychophysics).

The emerging patterns observed during meaningful interaction will be used to characterize and exploit the human nervous system The next paradigm of human-computer interface will optimize the technology to the physiology -- a biologically responsive interactive interface for engaging “mind-in-the-loop” information fusion support for intelligence on demand

The next paradigm of human-computer interface will optimize the technology to the physiology -- a biologically responsive interactive interface for engaging “mind-in-the-loop” information fusion support for intelligence on demand

Human computer interaction in the ACDE incorporates technology which augments our ability to create / express / retrieve / analyze / process / communicate / experience information in an interactive mode The ACDE is designed to use multi-sense rendering technologies, giving sustained perceptual effects, and natural user interface devices to facilitate effective human to computer communication..

Various display interactive technologies will be implemented in the acde

Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures. The more we know about the way we acquire, react to or interact with information physiologically the more we know the optimum way for a human to interact with intelligent information systems

The environment can also be instrumented to measure multiple physical and physiological parameters simultaneously and use them for dynamic use tracking

Models of information processing based on the capacity of these neurophysiological structures to process information will help our efforts to enhance perception of complex relationships by integrating physical environmental and individual physiological modalities.

Then by using the natural bioelectric energy as a signal source for input; electroencephalography, electroocculography, and electromyography (brain, eye and muscle) we can generate highly interactive systems in which these biological signals initiate specific events. Such a real-time analysis enables multi-modal feedback and closed-loop interactions.

Interactive interface technology renders content specific information onto multiple human sensory systems giving a sustained perceptual effect, while monitoring human response, in the form of physiometric gestures, speech, eye movements and various other inputs.

Such quantitative measurement of activity during purposeful tasks allows us to quantitatively characterize individual cognitive styles.

This capability promises to be a powerful tool for characterizing the complex nature of normal and enhanced/“impaired” human performance.

The systems of the UTM will monitor a user's actions, learn from them, and adapt by varying aspects of the system's configuration to optimize performance.

The tracking of user dynamics in an enchanted interaction with communication and information technologies is providing an exciting opportunity to facilitate the rapid exchange of relevant information thereby increasing the individual productivity of persons involved in the information fusion services activities.

Areas such as computer-supported cooperative work, knowledge engineering, expert systems, interactive attentional training, and adaptive task analysis will be changed fundamentally by this increase in quantitative assessment of individual and group abilities meaningful informatic exchanges. In addition to the physical and performance dynamics of individuals the psycho-social aspects and interpersonal dynamics of this technologically mediated human-computer and human-human communication will be able to be studied in new ways with these tools. Providing the knowledge and technology required to empower people to make a positive difference with information technology could foster the development of new technology

A multi level perspective motivates the comprehensive nature of the environmental tracking matrix system, which integrates multiple data acquisition devices, interface technologies, advanced analytical techniques, and multi-sensory rendering capabilities. Emphasis is placed on environment-resident intelligence sensors embedded at several levels to monitor, track, and interact with the various participants /users of the system

a research methodology that applies techniques and resources from many disciplines and is constantly seeking to improve the measurement of human performance and the assessment of interaction efficacy

. Devices such as gloves to measure hand motion dynamics, surface EOG and EMG sensors for eye movement and muscle contraction, and lightweight pressure sensor arrays for gait analysis

A necessary feature of the NRW is the integration of a variety of data input devices into a single system to include EEG, EMG, EOG, ECG, dynamic bend sensors, pressure sensors, audio and video digitizers, etc. The resulting capacity for data fusion allows for meaningful correlations to be made across various performance modalities.

The devices and their hardware boards connect to an external module, and a high speed bus will route the data both to a central multi-tasking server and to the rendering subsystem for immediate feedback. The server should be intelligent enough to automatically implement a custom configuration of input device parameters, interface functionality, and relevant records based on the device(s) connected and the identity of the operator(s) and analysts (s) currently utilizing the environment for multi int pattern discovery and relationship recognition

The user interface will be based on new theories of human-computer interaction methodologies , computer-supported cooperative work, knowledge engineering, expert systems, and adaptive task analysis The system will monitor a user's actions, learn from them, and adapt by varying aspects of the system's configuration to optimize performance. Adaptable on-line knowledge-based support using text, graphics, and animated tutorials provide interactive learning/discovery and navigation.

comparative evaluation of a progressing or digressing state. The nature of the change in this state may often be quite subtle

User Classification

We have identified possible different types of users. These unique, task specific , dynamics of the user types help define discrete activity levels of user functionality.

- the primary users of the system, are responsible for data acquisition, data management, basic analysis, and topic-oriented interactive bio optimized feedback modes.

- Technicians are responsible for simple data acquisition.

- Operators will use the more comprehensive data analysis tools.

- Researchers will focus on the data analysis but their use of the system will be unconstrained. They will explore and develop custom analytical techniques.

- Analysts will primarily use the multi int allsource info fusion features of the system, usually in a supervised setting. –supervised vs un supervised all source fusion methods or dynamics

With multi-sensor data acquisition and advanced analytical characterization, the rendering capacity of the system becomes extremely vital

enhance perception of complex relationships by integrating multiple concurrent and spatially distributed environmental sensor nets….

modular infrastructure which will expedite the implementation of new technologies

are technologies which support an experiential interaction with in a computationally sustained environment

these technologies translate natural human actions of communication, such as speaking eye-movements and body gestures into computer commands , but they also render information to the human in multiple sensory modalities, that is spatialized audio, 3D graphics and various somato-sensory forms.

spatialize the displacement of multiple sound sources giving us simultaneous exposure to different dynamic data sets. In these spatialized environments we can shift our attentional focus from source to source for real time comparison of multiple sets of data

The Microscope of the Mind

The goal is to extend these environmental control systems into new methods of investigative research. Such as a test of basic cognitive functionality or the capacity to maintain attentional focus necessary to complete an iterative series of cognitive tasks. Data fusion of sensor data with user interaction parameters will allow meaningful correlation's to be made across various performance modalities. A goal of this application is to seek to identify a qualitative difference between the two performance/behavior states and then investigate various methods of quantifying that difference in a way that can be generalized.

It is postulated a difference will be seen in the modulation of some of the natural rhythms. It is also postulated that a cognitively induced modification would be consistent in an individual but would most likely be different between individuals. The psycho-social-behavioral nature of individuals factors into initial assessment of their cognitive function. Other indicators of cognitive function are short-intermediate-long term memory, sound judgment and the ability to identify similarities in related objects. Performance of these cognitive functions is a strong indicator of the biologic health of the brain. Poor performance is highly correlated with organic brain dysfunction.

of our efforts is in developing highly interactive, biocybernetic systems where biological signals can modify an environmental chambers' parameters allowing the user to bioelectrically interface with spatialized environments. We believe that such physiologically modulated environmental systems may have a health preserving function. Interfaces to control stimulation can adaptively utilize any biosignal. The result is the capacity to create a stimulus regime that accelerates relaxation and facilitates stress reduction.

. interface technologies by directly sensing as inputs and rendering information in ways that take

Proposal To:

University of Southern California

Information Sciences Institute - East

3811 North Fairfax Drive, Suite 200

Arlington, VA 22203

(703) 248-6166

By: Syracuse University

Dave Warner, M.D., Ph.D.

Department of Electrical Engineering and Computer Science

Date: August 13, 2002

For:

Design & Fabrication Services and Consulting Services for the ACDE projects.

Description of Services:

The Design and Fabrication of various human interface elements for the ACDE Projects.

Qualifications:

Syracuse University, (Dave Warner MD PhD as PI ) , is currently engaged in multiple efforts of the ongoing project and is familiar with the needs of the NRO and other sponsors of the ACDE

Period of Performance:

Design & Fabrication Deliverables below are due on September 30, 2002

And include as specified as Task 1 Below

1 Aeron Chair accessory kit

1 Command Chair

1 Monitor Arm

1 Knowledge Fountain

1 Pair Standing Hands

1 Pair Wall-Mounted Hands

Research and Reporting Deliverables are to be completed by Dec 31^st 2002

Introduction

ENVIRONMENTS AND INTERFACES FOR ACDE

The research, prototyping, development and demonstration of technologies to

support interpretation of data is a major challenge. For any complex

phenomenon, the large number of relevant input systems and techniques

creates an interface problem with exponential proportions. It is therefore

critical that technologies be researched and demonstrated for the representation and interpretation of multiple, diverse data sources, and for human interpretation of these data sources.

To achieve this goal the Sponsor has been tasked to support the prototyping and demonstration of advanced intelligence oriented technologies in the planning and conduct of U.S. National Security operations.

Background

The capacity of intelligence oriented sensor assets and interconnecting information systems is continually increasing. Current systems receive, process, and transmit massive amounts of information. Efforts to develop new human-computer interaction options to exploit these ever increasing information opportunities are ongoing. The development and experimental utilization of the ACDE is intended to greatly enhance our ability to intelligently interact with this increasing flow of information by creating an information enriched experiential interactive environment enabled with communication and computation resources.

Human computer interaction in the ACDE incorporates technology which augments the ability to create / express / retrieve / analyze / process / communicate / experience information in an interactive mode The ACDE is designed to use multi-sense rendering technologies, giving sustained perceptual effects, and natural user interface devices to facilitate effective human to computer communication.

The ability to achieve the integration of a set of advanced human-to-computer input devices into a single interface system and demonstrate data fusion to enable meaningful

correlations across various input modalities will significantly enhance progress toward this end.

INTENT of EFFORT

The contractor’s efforts are to ensure that the design and fabrication of initial experimental development units of various interactive display control systems and their support technologies will be optimally implemented in the acde in a modular and interchangeable manner.

SCOPE of EFFORT

The Contractor shall provide the systems, architecture, and design

engineering required to research, prototype, and demonstrate capabilities in

the following areas: the integration and fusion of information from

human-to-computer input devices and their application to problems such as innovative mechanisms to support interactive environments; data rendering ,analysis and design and development of experimental protocols.

REQUIREMENTS

4.1 PLANNING

.1 The contractor shall develop project plans to provide insight into the

various phases of the research.

4.2 INTEGRATION AND FUSION

4.2.1 The contractor shall provide the systems, architecture, and design

engineering required to research, prototype, and demonstrate the

integration of a set of advanced human-to-computer input devices into a

single interface system.

The Contractor will design fabricate and deliver the following experimental development units (see attached pdfs for more details)

Aeron Chair Accessory Kits

Accessory kits are modular components designed to attach to the Herman Miller Aeron Chair. These kits provide data connections for audio and video, as well as control functionality for the user. The Flying Hands control interface increases flexibility of

functionality and expressive capacity which allow the user to communicate intention more effectively.

Command Chair

The Command Chair is designed to integrate the human user at a neuro-cognitive level with advanced multi-sensory perceptualization systems. This provides for a much greater capacity to confront and manipulate information for specific purposes.

Standing

The standing hands are a self standing set of flying hands. Developed to maximize the hands as inputs for data rendering systems, these increase the capacity to manipulate multi-dimensional environments.

Knowledge Fountain

The knowledge fountain provides a rapid prototyping and experimentation platform for up to 4 simultaneous users. This provides a unified interface system comprised of a core infrastructure intended to optimize the integration of human intelligence through increased multi-sensory perceptualization and enhanced expressive capacity.

Monitor Arm

The monitor arm tower is designed to support a 24 inch lcd panel on an articulated arm. The tower's weight is carried by the raised floor, and the top is anchored to the ceiling grid. The articulated arm is capable of extended the lcd panel up to 6 feet from the

tower. Multiple arms can also be connected to the tower to create a grid of lcd panels.

Flying Hands

A set of multi modality hand based interface devices used to control multiple interaction parameters simultaneously

In addition to the physical interface systems ,,,,aka “experimental development units”

4.2.2 The contractor shall provide the systems, architecture, and design

engineering required to research, and recommend data fusion methodologies to enable meaningful correlations across various interaction modalities.

4.3. INTERFACING

4.3.1 The contractor shall provide the systems, architecture, and design

engineering required to research, and report methods for the integration

of comprehensive sets of visual, aural, tactile, proprioceptive, and

somatosensory rendering devices into a prototype system to demonstrate an

integrative experiential interaction with the complex data types.

4.3.2 The contractor shall provide the systems, architecture, and design

engineering required to research, prototype, and demonstrate advanced

interfacing technology protocol for the rendering of computer information onto

multiple human sensory systems to achieve and demonstrate a sustained

perceptual effect (i.e., a sensation with a context).

4.3.3 The contractor shall research, prototype, and demonstrate the combination

of these different rendering modalities with somatotopic placement, in order

to achieve and demonstrate spatial coding of the rendered information

4.3.4 The contractor shall research, and report recommendations on how the

implementation of vision, hearing, and touch technologies can allow for

simultaneous sensation of multiple independent and dynamic data sets that

can be integrated physiologically into a single perceptual state.

4.4. INTERACTIVE ENVIRONMENTS

4.4.1. The contractor shall research, new ways to render complex information with

advanced human-to-computer input devices well as providing technical guidance for the measurement of same.

4.5 DATA RENDERING AND ANALYSIS

4.5.1 The contractor shall research, interface systems to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information.

4.5.2 The contractor shall provide the systems, architecture, and design

engineering required to research, design and oversee the implementation of a data analysis subsystem designed to enhance the ways that relevant data may then be rendered optimally to the operators' sensory modalities

4.6 EXPERIMENTAL PROTOCOLS

4.6.1 The contractor shall provide the systems, architecture, and design

engineering required to research, prototype, and demonstrate an experimental

protocol to support the evaluation of systems for providing mission critical

support, including the interaction with appropriate agencies to ensure the

specific requirements of mission critical objectives are met.

4.7 DEMONSTRATIONS

4.7.1 The contractor shall provide support for demonstrations of the

developing functionality,

5.0 REPORTS, DATA AND DELIVERABLES.

Delivery of reports shall be submitted electronically.

Where devices are the deliverable they will be delivered to the site specified by the sponsor.

6.0 SPECIAL CONSIDERATIONS

6.1 The work for this contract will be performed primarily at the

contractor's off-site facilities. Associated efforts may be performed in Washington, DC area and other sites as necessary to complete each delivery order. The facilities shall contain sufficient space and equipment required to perform the tasks of this Statement of Work. The contractor's facilities shall include equipment necessary for preparation

and final reproducible copy of technical documentation, for making copies of

documentation, and for producing view graphs.

6.2 Travel is both required and expected, Travel will be estimated on similar effort histories

- contractor will provide travel and logistics support within budget constraints

--current similar efforts for travel under similar time constraints will be used to estimate

for budget purposes only

6.3 The Sponsor shall provide access to all government facilities with

schedules established by the government. The contractor is not guaranteed a

specific site or amount of computer time. The contractor shall adjust work

schedules to accommodate site and computer time availability and to minimize

efforts in excess of a normal work week.

7.0 LIMITATIONS, CONSTRAINTS, AND CONDITIONS

7.2 Work shall be performed at contractor's facility in Syracuse, NY, at

other government facilities and as required at other contractor offices and

facilities. The contractor shall provide facilities necessary for this effort

7.3 The contractor is authorized to subcontract with prior approval of the

Contracting Officer. All requirements of the Basic Contract shall apply to

Subcontracts .

7.4 Textual materials and other documentation shall be delivered by the

contractor in a format compatible with the MICROSOFT OFFICE environment.

Statement of Work

The Contractor will apply best efforts towards the following task groups

Task group 1 Design And Fabrication

Design and fabricate the experimental development units

Timely delivery of experimental development units as required by the sponsor

Task group 2 Implementation Testing And Evaluation

Perform best efforts toward supporting sponsors initial implementation and operation of experimental units. To include experimental design, interface testing, protocol development.

Task group 3 Initial Instrumentation And Experimental Methods

Apply best efforts towards the research design, development and testing of experimental methods of “use” tracking in an intelligence oriented interactive environment.

Costs are summarized as follows (see attached budget for details)

Task group1 Design And Fabrication (4.1 – 4.2.1 )

Task group 2 Implementation Testing And Evaluation (4.2.2 – 4.3.4)

Task group 3 Initial Instrumentation And Experimental Methods (4.4.1 – 4.71)

OPTIMIZING MIND IN THE LOOP INTELLIGENCE

Innovative Methods for Enhanced Perceptual Modulation of Human Interaction with Information in Perceptualization Environments

In recent years, the technological capacities of the National Reconnaissance Office to access and collect diverse information have increased substantially. These efforts are still insufficient to meet our emerging national security concerns, particularly in this time of growing concern over new forms warfare and the atrocious threat of biological and chemical weapons being used as a means of homeland assault. There is need for even greater capacities to process gathered intelligence. Given the complex and advancing nature of information gathering, there must be a more robust integration of the human user into a multipoint and immersive knowledge discovery matrix. Information gathering and manipulation technologies are limited in their ability to make a difference to the extent that the human user is limited in his ability to exploit that information. We propose to develop a system sensitive in the conceptual and methodological consideration of the humans extraordinary abilities for perception, cognition and expression. The perceptualization environment a multi-modal, multi-sensory, high performance communications tool for handling the escalating quantities and types of information the Office will confront in the near and distant future.

We will research and develop methodologies to not only present information to a user, but more importantly, to give the user tools which allow them to manipulate information in meaningful ways The representation and comprehensibility of information are the conceptual and technological challenges we seek to address with the perceptualization environment. This

environment is a unified interface system comprised of a core infrastructure and experimental methodology which optimizes the integration of human intelligence through increased multi-sensory perceptualization and enhanced expressive capacity. This tool will provide an order of magnitude increase in human-information interaction. We propose to research an approach to the problem that leverages sensory physiologic principles with an operational knowledge

of human-information system integration. The priority of our approach is to connect the human to the content via an interface which exploits the visual, auditory, and the tactile capacities of the human body for perception. This provides for a much greater capacity for perceptual interaction with information than the current interface model permits. Our approach is to develop a system which generates and modifies information in terms of the users perceptual and neuro-cognitive preferences to confront and manipulate information for specific, and at times highly contingent, purposes. We are after an emergent fusioning of information where the user will be able to rather spontaneously communicate their intention about the content at hand and its form. Human neuro-cognitive integration into the fusing and presenting of information opens the space for discernment of knowledge which would have remained invisible. When the human is more deeply woven into the processes whereby the machine makes decisions about data and representation, more powerful options for response are available. This involves, for example, the capacity to specify queries and the representations of the results thereof enabling a powerful capacity to share that information with another expert who will be sent much farther along in the process of action. Physicality & Functionality

MindTel has developed powerful and generic hardware/software solutions to handle core functional elements of this communications environment. We currently conceive of the perceptualization environment as a physical space, enabling an intelligence worker to simultaneously explore and enhance both the perceptual form in which information is being represented and the neuro-cognitive state of the human during perception of represented information. These Neuro-Cognitive State Enhancements which alter our perceptual cognitive states at calculated points during the design and execution of the interaction process will open new realms of information which would have remained totally invisible.

MindTels research in this area is to think in new ways about the perceptual basis of human interaction with vast and diverse data sources. What we are proposing is far outside the normative realm of how to construct interfaces and represent information. Thinking in these ways will require potential users to understand the challenge which confronts them in this time of intelligence gathering.

Without pushing the boundaries of what can count as an acceptable means for accessing information and making decisions based upon it, unrealized possibilities to help go unseen. We are aiming towards a new kind of perceptually based literacy to aid any process of high quantity, high performance interaction with information based on a desire for both a flexibility of functionality and massively enhanced perceptual dimensionality.

The Enchantment of Never Never Land

Instrumenting the ACDE for use tracking analysis

What exists

The physical space…

The functional space

The operational space

Spatial structural cyber social interaction dynamics

Instrument everything and everybody in as many ways as we can think of and “Track the flux ”

By quantitatively tracking the flow of information and utilization dynamics

of operational

We instrument the environment including looking for fields,,, ,,emf

We instrument for activity of users

Ambient audio sensing

Tele controlled focal audio imaging

Infra and ultra

Voice stress analysis

Bi direction controls of audio field…..

Spatial damping and sonic enhancement

Bio-suit for tracking…

Personal multi person int transponders,,,

Bio signal

Gestural biometrics

Poly physiometric data streaming

Multi parameter poly physiometric

A primary function of the ACDE is further research in the areas of information system infrastructure including networking technologies and architectures that support interaction between humans and information systems in distributed collaborative environments.

The ACDE was built to refine conceptual design, test innovative technology and component integration and to influence corresponding planned experimentation in the areas of human-information system interaction in distributed collaborative environments with multisensory interactive representations for command system applications.

Enchantment

The instrumentation of NNL

I an effort to monitor analyst interaction with information and understand the emergent behavior

Monitor human activity

The following

It is possible to instrument the people and the environment to determine the behavior and emergent dynamics of persons interacting with massive volumes of information…

This is done with the hope of determining what possible exploitations can be achieved

Tracking the behavior from multiple perspectives

The design and implementation of a system to develop understand and optimize to

Modes of tracking

Motion detection

Floor sensors

Audio sensors

Video

Proximity sensors

Instrumentation of the environment and the interaction tools them selves

This combined with the tracking of the information flow promises to yield a methodology to

· audio and video recordings of one's speech and behavior

· · GPS tracking of one's location

· · optical fibers, resistors, and potentiometers for things such as joint angulation, location, and orientation in space

addressing the need for rapid-prototyping hardware and software tools for versatile adaptable configuration of interface options.

Considerable focus should be placed on eye-tracking technology to exploit what limited eye movement and/or blink capability the subject has in order to implement a pseudo-Morse code communication program. EOG, optical-, or infrared eye-tracking systems must all be explored and exploited in this regard.

an input suit that can remotely monitor all bioelectric phenomena including muscle activity, heart rate, galvanic skin response, respiration rate, and brain activity;

that can identify the user for security and database collection purposes.

configuration allows flexibility in mapping

gesture recognition system using a sensor array

render complete stress, attention, and fatigue evaluation, as well as dynamic motion capture capabilities for real-time task performance evaluation.

data for quantitative analysis and storage. The stored information can then be structured as a database that will have detailed information on personnel performance for a specific complex task.

Automated, or command personnel monitored, comparison of the training procedure versus an ideal procedure will further help instruct the trainee.

Special purpose software will be developed for evaluating stress and attention factors. Follow-on stage software development will focus on human performance evaluation factors in both simulated and real task environments. BCS envisions using this technology to assess psychomotor performance for complex tasks. In addition, the data gathered during assessment can be used to create more effective training programs, individually tailored for the strengths and weaknesses of each trainee.

This optimization of human interaction with vast networked systems information assets incorporates multi-sense rendering technologies, giving sustained perceptual effects, and incorporates other natural user interface devices which measure multiple (physical and physiological) parameters simultaneously

increase in effectiveness will impact both human-computer and human-human communication through "enhanced expressivity” and extended perceptual dimensionality.

prototype and evaluate an integrative interface matrix that couples the data streams emerging from sensors and micro informatic technologies

Exploiting Natural systems of human information processing through developing “Physiologically oriented interface systems”.

Knowledge of sensory physiology and perceptual psychophysics is being used to optimize our future interactions with the information from sensor networks

Advanced research in the area of sensors and information systems continue to produce an ever emerging array of networked information and communication (info-com) technologies, which in turn produce an ever increasing volume of information which must be perceived and understood by a human operator. We propose to develop an interface to the network through which human communication of and interaction with this information is facilitated.

An understanding of the human neurophysiology allows for exploitation of predictable, adaptive capabilities. The assertion is that the information flow between external sources and direct experience is biased, restrained, constrained, limited, enhanced, and facilitated in understandable and predictable ways by the physiological mechanisms of human information processing.

what types of information are best rendered to each specific sense modality, a sense specific optimization of rendered information.

"reference architecture" (a formalized conceptual framework for research and technology development) for designing Physio-informatically robust interactive human computer interface systems

. The primary focus will be to consider the flow of information between the human and the com-system in a sustained, iterative, experiential interaction. The intent of developing this reference architecture is to map the information flow during/caused by the intentional /volitional interaction with information between a conscious human and an info-com system.

develop an interactive environment that combines new ways to render complex

information with the advanced human-to-computer input devices.

refine quantitative measurement of activity during purposeful tasks.

refine the interface system to match the human nervous system's ability to

transduce, transmit, and render to consciousness the necessary information to interact

intelligently with information. We will implement a data analysis subsystem designed to

enhance the ways that relevant data may then be rendered optimally to the operator's

sensory modalities. Linear and nonlinear multivariate analysis tools will be utilized

for the processing of multiple data sets in a variety of ways, including graphical analysis

(phase portraits, compressed arrays, recurrence maps, etc.) and sound editing (mixing, filtering). Automated detection of trends and correlations using fuzzy logic may be performed in the background or in a post-processing mode. The system will be designed so that the user may then be alerted by the system if it detects areas worthy of further investigation

We will design an experimental protocol to evaluate the Grok-Box system for

providing mission critical support. We will interact with the appropriate

agencies to ensure the specific requirements of this task to meet mission

critical objectives are met.

feedback regarding their performance,