NeatTools TNG, EVA and the Intermental Network: the Superglue for a Brain-Machine Interface

My interest in this effort lies mainly in designing the training system for subjects to master the BMI technology before going out into the ‘real world’ and applying their skills. More specifically, I'm interested in making a direct contribution to helping design the

1) perceptualization modules for designing tasks

2)the adaptive database management interface issues

3) strategies for incorporating an autonomic component into a distrubuted collaborative intermental network.

I'm assuming that because we know very little about how such a system is best implemented, and because it's likely that we're all underestimating the magnitude of the task, we need the most flexible affordable rapid-prototyping technologies available to piece together the other major components being pursued by the big (but overly specialized) minds elsewhere.

In the content below, I have not yet specified details of my role. I want to get my bearings, lay some ground rules, clarify some things, and then I'll brainstorm (later today after the Charger game) on just how I might help with the three prongs above.

The main reason for achieving a BMI is to significantly extend one's ability to communicate with and control one's environment, and perhaps even one's self. An added benefit will be to allow high fidelity tracking of a system of one or many individual's psychophysiological states and offer useful feedback to optimize the performance of the system (intermental network).

Typical brain-machine interfaces for the purpose of assisting the disabled consist of using a surface or subdural electrode array and training the subject to activate a small number of distinct cognitive states. Often as few as three different cognitive states suffice to give the subject a kind of 'Morse Code' with which to communicate. [14, 18] Another example would include adapting a prosthetic hand to the end of a severed limb which still has functional nerve endings and muscles subject to cognitive motor control. A third more esoteric example might include magnetic nano-coils distributed throughout the hypothalamus and pituitary region wirelessly linked to an embedded microchip to help regulate autonomic or endocrine behavior independent of the user's cognitive control. A fourth would consist of a synthetically-intelligent robotic system with a level of neural processing comparable to that of, say, a cricket's brain. A fifth would be to wire a cockroach's brain to a nano-hydrogen bomb (Bin Laden's working on that one at the moment, I believe). But I digress...

I’m assuming the typical brain-machine interface (if there is such a thing that pops into people's minds with the phrase) would consist of sensors measuring brain electromagnetic activity by acquiring EEG either directly from neural tissue (microvolt realm) or from the surface of the skin (millivolt realm), and EMG from both cortical and subcortical regions (10-6 Gauss realm). The practicality of using EMG ‘in the field’ is debatable, given the sensitive and expensive nature of current technology, so I will assume that EMG will be useful primarily in the learning process wherein the user’s ‘thumbprint’ is established and where the correlation of EEG with EMG provides useful insight into a subject’s neurophysiological behaviors. Data acquired from these sensors would be transduced into a useful form by some machine. Feedback to the system during the process would be essential for optimal control, but the nature of this feedback is perhaps the most important and least understood part of the system. (I'll try to focus on this in my next installment.)

top

Read-Write States

A distinction between 'read' and 'write' states must be made.

A 'read' state is a 'passive' measurement of one's current psychophysiological state with measurement tools such as electrode arrays and magnetic coils to measure bioelectromagnetic output from the cortex, scalp, and nerves, but also with more advanced tools such as PET and fMRI. (The degree to which the latter two are 'passive' is debatable).

A 'write' state typically includes the usual sensory input such as visual, auditory, vestibular, haptic (somatosensory), as well as direct cortical or neural stimulation typically with electrodes or magnetic coils. Micro- and nano-technologies may offer possibilities we must do our best to anticipate, such as micro- or nano-coils for intracellular electromagnetic inductive effects on a large distribution of neurons using histochemical specificity.

top

Distinguishing Between HCI, HMI, BCI and BMI

Because a brain-machine interface is a part of the larger 'human-machine interface', meaningful limits on what is meant by 'brain-machine interface' must be defined and used to distinguish a bmi from the more general 'human-machine interface'.

For example, 'read' states included in a human-machine interface but not included in a (typical) brain-machine interface would be things such as:

· audio and video recordings of one's speech and behavior

· GPS tracking of one's location

· optical fibers, resistors, and potentiometers for things such as joint angulation, location, and orientation in space

· electrode arrays to measure bioelectric output from the scalp, skin, and muscles

A brain-machine interface must also include access to data such as:

· brain atlases

· grand average and individual bioelectromagnetic data (EEG, MEG, EOG, EMG, fMRI, PET) to be referenced with one's own states

· data from neuro-mechanical simulations, biorobotics device performance, and neuro-mechanical prosthesis research

top

Focus of the Proposal

The proposal should focus mainly on NeatTools, TNG, EVA and the Intermental Network. NeatTools has a wide range of features that make it an appealing solution to a neglected part of a BMI. Hilight features such as the modularized data processing of arbitrary inputs, tracking, existing 3D viz capabilities plus the ability to hack an interface to Open Source viz tools such as OpenDX, and the fact that the NeatTools interface is inherently aware of the variability of user needs. We are addressing the need for rapid-prototyping hardware and software tools for versatile adaptable configuration of interface options. These technologies should serve as a kind of 'superglue' for piecing the other more obvious components of the system together (i.e. adaptive pattern recognition software, databases, sensor technologies, etc.).

If it doesn’t already, NeatTools must have an interface to both Java applets as well as servlets, and an XML parser.

It is likely each subject must undergo an extensive battery of ERP recordings in a variety of performance states simply to establish an individual 'thumbprint' for that particular cortical region. [21] This applies also to mapping the peripheral nervous system, as in such instances where a prosthetic device must adapt to the functional nerve endings of a partial limb. Such batteries must include but not be limited to spatial reasoning tasks in both 2D and 3D for aural and visual states. Cortical regions of interest would include whole brain integration of EEG as well as more specific surface arrays over known regions of activity such as the motor and pre-motor cortices, the pre-frontal region for pre-movement related 'bereitschaft' potential activity, the occipital region visual activity, the left temporal for language and the right temporal for spatial associativity tasks, and so on. Data acquired for these batteries must be archived in a metadata format and federated to participating research institutes. This calls for hi-speed connectivity and persistent archive management systems such as the Storage Resource Broker, SuMS, Globus and Legion, the Access Grid, and the soon to be realized Teragrid Facility.

During initial training, in order to rule out certain confounding problems that accompany the use of patients, subjects will be assumed to be fully functional, highly talented, and 'groomed' to manage the advanced prototype training BMI. Electrode arrays must be relatively easy to apply with reasonable cortical specificity. Subjects must be comfortable with the application of sensor technology just about anywhere we can put it! Subjects therefore will be expected to bathe and enema five times daily! Heh, heh, just kidding!

The application of surface electrodes offers greater practicality in the field compared with subdural micro-electrode arrays. Subdural electrodes offer higher fidelity data with greater specificity allowing much less computational demands on the system. In this regard, subjects will typically have their scalps shaved daily and ideally will be willing to subject themselves to some sort of 'flip-top' skull much like we had the pleasure of seeing in 'Hannibal' to afford extra convenience for deployment of subdural electrodes 'in the field'. To successfully manage the diversity of individual neural maps and to correlate them with realtime and archival physiological data presents unprecendented data management demands. Database management tools will require an adaptive interface as part of the brain-machine-brain loop.

Due to the variation in the computational nature of these techniques, data processing must be ongoing, performed both in realtime and offline using a variety of hardware from desktop workstations to commodity cluster technology and massively parallel systems, all managed over a high-speed grid computing facility (currently the Access Grid, soon the Teragrid facilitiy).

Traditional analytical methods of EEG such as frequency analysis and covariance measures combined with exciting new methods of analyzing and interpreting brain data such as Independent Components Analysis and Support Vector Machines suggest that the user/trainer system must have a range of options with which to configure the learning process.

top

Excerpts from pertinent articles:

NPACI Online article on Federating Brain Data:

The high degree of individual variability in cortical geography and functional organization compounds the complexity*.

In developing a database focused on the structure and function of invertebrate nervous systems, Jacobs, Miller, and other CCB members have also been able to construct a sophisticated interface for posing database queries. In the NPACI project, they are working with data from Ellisman's lab as well as their own and coordinating an investigation of database issues in neuroscience generally. "Our objective is to develop efficient schemas that can scale with the complexity of nervous systems in higher organisms and with the advances in data-intensive computing," Jacobs said. The CCB, the NCMIR lab, and labs at UCLA and Washington University have installed the SDSC Storage Resource Broker system developed in the Data-Intensive Computing Environments thrust area to facilitate the sharing and transmission of data, and they are developing a Neuroscience Data Interchange System.

Spatiotemporal dynamics of component processes in human working memory. Gevins A, Cutillo B. EEG Systems Laboratory and SAM Technology, San Francisco, CA 94107.

Working memory (WM), the ability to momentarily maintain information in an active state, is central to higher cognitive functions. The processes involved in WM operate on a sub-second timescale, and thus evoked potential measures have an appropriate temporal resolution for studying them. In the experiment reported here, evoked potential covariances (EPC) between scalp recording sites were computed for a task requiring maintenance of numeric information in WM; these EPCs were compared to those observed in a control task which had the same stimuli and responses but less of a WM requirement. EPC patterns differed between conditions prior to the stimulus, and in an interval spanning the P300 peak in the match detection trials which required response inhibition. The pattern of prestimulus EPCs was more complex and left-sided in the WM task, when memory codes were being maintained and responses contingent on those codes were being prepared. P300 peak latency was 140 msec shorter in the WM task, and the P300 EPC pattern was more anterior and left-sided. In contrast, EPC patterns did not differ during early stages of stimulus processing or during response execution. These results suggest that distinct EPC patterns associated with WM only occur during intervals in which the information in an active state is being utilized for task performance.

top

Data Management

In an effort to anchor these ideas in reality, let's consider a subject with a C1 spinal injury, a locked-in patient with limited eye movement and blink capability.

A typical short term goal would be to train the subject to control a computer interface for basic communication tasks (for example, TalkAssist from the Brain-User Interface Group at Georgia State). It must be assumed that the subject's abilities are relatively unknown, and for the sake of the vision, we'll also assume the subject is a candidate for multiple (3) electrodes implanted in the pre-motor (for pre-movement potentials), motor (to correlate with pmp), and parieto-occipital cortices (for eyes-open/eyes-closed alpha off/on binary switch mechanism). Three electrodes offer greater variability in control, offering the subject a range of options to avoid fatigue with a particular task. To date, no patient has had more than one electrode implanted in cortex.

A typical long term goal would be to control a communications device such as internet telephony with speech generation capabilities. This would require the subject to have command over an adaptive interface to a database of letters, words, phrases, and complete sentences. The interface must be capable of assembling these data into a logical structure for realtime communication over phone lines. The interface would also provide the ability to dial a phone number and initiate communication.

The added twist which other groups are most likely overlooking would be the need to embed the function calls between programs such as TalkAssist and the web into an XML framework for more generic web services support. Up-and-coming IP protocol standards such as Simple Object Access Protocol (SOAP), and other XML support standards such as the Web Services Description Language (WSDL), must be brought to bear on whatever interface technologies are provided to subjects. The web offers the greatest promise for unlimited communication, but without these standards, interface software for locked-in patients will continue to be prohibitively proprietary.

Technology integration

Considerable focus should be placed on eye-tracking technology to exploit what limited eye movement and/or blink capability the subject has in order to implement a pseudo-Morse code communication program. EOG, optical-, or infrared eye-tracking systems must all be explored and exploited in this regard. Portability/mobility is not a primary concern in this scenario with locked-in subjects, but may be a much higher priority with ambulatory subjects.

Perceptualization

top

References

1: Kostov A, Polak M. Parallel man-machine training in development of EEG-based cursor control. IEEE Trans Rehabil Eng. 2000 Jun;8(2):203-5.

2: Pfurtscheller G, Neuper C, Guger C, Harkam W, Ramoser H, Schlogl A, Obermaier B, Pregenzer M. Current trends in Graz Brain-Computer Interface (BCI) research. IEEE Trans Rehabil Eng. 2000 Jun;8(2):216-9.

3: Penny WD, Roberts SJ, Curran EA, Stokes MJ. EEG-based communication: a pattern recognition approach. IEEE Trans Rehabil Eng. 2000 Jun;8(2):214-5.

4: Robinson CJ. A commentary on brain-computer interfacing and its impact on rehabilitation science and clinical applicability. IEEE Trans Rehabil Eng. 2000 Jun;8(2):161-3. No abstract available.

5: Birbaumer N, Kubler A, Ghanayim N, Hinterberger T, Perelmouter J, Kaiser J, Iversen I, Kotchoubey B, Neumann N, Flor H. The thought translation device (TTD) for completely paralyzed patients. IEEE Trans Rehabil Eng. 2000 Jun;8(2):190-3.

6: Birch GE, Mason SG. Brain-computer interface research at the Neil Squire Foundation. IEEE Trans Rehabil Eng. 2000 Jun;8(2):193-5.

7: Pregenzer M, Pfurtscheller G. Frequency component selection for an EEG-based brain to computer interface. IEEE Trans Rehabil Eng. 1999 Dec;7(4):413-9.

8: Makeig S, Enghoff S, Jung TP, Sejnowski TJ. A natural basis for efficient brain-actuated control. IEEE Trans Rehabil Eng. 2000 Jun;8(2):208-11.

9: Middendorf M, McMillan G, Calhoun G, Jones KS. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Trans Rehabil Eng. 2000 Jun;8(2):211-4.

10: Bayliss JD, Ballard DH. A virtual reality testbed for brain-computer interface research. IEEE Trans Rehabil Eng. 2000 Jun;8(2):188-90.

11: Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Peckham PH, Schalk G, Donchin E, Quatrano LA, Robinson CJ, Vaughan TM. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng. 2000 Jun;8(2):164-73.

12: Levine SP, Huggins JE, BeMent SL, Kushwaha RK, Schuh LA, Rohde MM, Passaro EA, Ross DA, Elisevich KV, Smith BJ. A direct brain interface based on event-related potentials. IEEE Trans Rehabil Eng. 2000 Jun;8(2):180-5.

13: Babiloni F, Cincotti F, Lazzarini L, Millan J, Mourino J, Varsta M, Heikkonen J, Bianchi L, Marciani MG. Linear classification of low-resolution EEG patterns produced by imagined hand movements. IEEE Trans Rehabil Eng. 2000 Jun;8(2):186-8.

14: Kennedy PR, Bakay RA, Moore MM, Adams K, Goldwaithe J. Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng. 2000 Jun;8(2):198-202. http://www.cc.gatech.edu/computing/brainui/

15: Pfurtscheller G, Flotzinger D, Pregenzer M, Wolpaw JR, McFarland D. EEG-based brain computer interface (BCI). Search for optimal electrode positions and frequency components. Med Prog Technol. 1995-96;21(3):111-21.

16: Mason SG, Birch GE. A brain-controlled switch for asynchronous control applications. IEEE Trans Biomed Eng. 2000 Oct;47(10):1297-307.

17: Flotzinger D, Kalcher J, Pfurtscheller G. EEG classification by learning vector quantization. Biomed Tech (Berl). 1992 Dec;37(12):303-9.

18: Pineda JA, Allison BZ, Vankov A. The effects of self-movement, observation, and imagination on mu rhythms and readiness potentials (RP's): toward a brain-computer interface (BCI). IEEE Trans Rehabil Eng. 2000 Jun;8(2):219-22.

19: Miner LA, McFarland DJ, Wolpaw JR. Answering questions with an electroencephalogram-based brain-computer interface. Arch Phys Med Rehabil. 1998 Sep;79(9):1029-33.

20: Donchin E, Spencer KM, Wijesinghe R. The mental prosthesis: assessing the speed of a P300-based brain-computer interface. IEEE Trans Rehabil Eng. 2000 Jun;8(2):174-9.

21. Gevins, A.S. and Cutillo, B.A. (1993) Spatiotemporal dynamics of component processes in human working memory. Electroencephalography and Clinical Neurophysiology, 87, 128-143.

Others in Brain-Machine Interfaces

Andy Schwartz (Arizona State University)
Neural control of robotic arm
http://researchmag.asu.edu/stories/restoring.html

John Donoghue (Brown University)
Brain representation of movement
http://www.brainscience.brown.edu/departments/faculty/donoghue.html

Richard Andersen (Caltech)
Improved neuroelectrode systems
http://vis.caltech.edu/

Phillip Kennedy, Roy Bakay
(Emory University) Communication systems for paralyzed patients
http://www.emory.edu/WHSC/HSNEWS/releases/oct98/101398bakay.html