Innovative Claims:
1. We propose to develop an interactive head mounted display and
camera stereoscopic video and binaural audio communications system which
is worn by a user in the field and transmits said signals to a remote base
location. These signals are time-base corrected, multiplexed and video overlayed
(chroma-keyed) with hand gestures, text, photos, video or other vital data
and instantly transmitted back to the user in the field in real-time; thus
providing the user with an enhanced 3-D video reality and data to perform
task oriented operations. In addition, the receiving parties can view a
remote location and medical victims as accurately as if they were on the
scene and provide assistance via the wearer in the field. Deliverables:
1. Integrated head-mounted display, stereoscopic micro-camera and binaural
audio system.
2. Dual-channel video and dual-channel audio wearable microwave transmission
and reception system with a 2 mile radius for communication.
3. Portable base station with abilities to time-base correct (field-lock)
dual discrete video signals, field interlaced multiplexing, and chroma key
video overlay. Base staion will have capabilities to receive and transmit
signals via microwave to the user in the field and via satellite to anyone
anywhere having web-based connectivity.
Key Personnel:
1. Dr. Dave Warner, M.D., Ph.D.
2. Bill Rojas, Ph.D. - EVA Systems Developer
3. Mathew Carbone
4. Tim Murphy
5. David Balch
6. Patrick Keller
Summary:
The EVA System allows for a shared experience between the wearer and a person
or persons at a remote location or locations. By transmitting as true a
representation of reality as is electronically possible, this system provides
the wearer with an enhanced view of his current location, while providing
a remote person(s) with an accurate 3-D representation of a location at
a distance. In so doing, the wearer and the receiving parties can interact
“as one” to provide assistance which individually, because of knowledge
or distance, neither could accomplish. EVA System is adaptable to interface
with other medical and information gathering technologies, such as GPS,
and EKG, EEG, Pulse/Oxymetry or other vital signs which could be acquired
and transmitted as an integrated microwave signal along with stereo-video
and binaural audio. In conjunction with satellite and internet based connectivity,
EVA System can fulfill the goal of true Distributed Medical Intelligence.
EVA System:
1) Interface: The interface consists of a head mounted stereoscopic
display integrated with high resolution micro-video cameras and binaural
audio microphones which is worn by a user in the field. The acquired video
and audio signals are transmitted to a remote receiving station which is
manned by a person wearing a head mounted display who sees and hears the
sights and sounds of the user in the field. The person receiving the transmitted
signals can interact with the wearer by chroma-key overlay of hand gestures
thus visually pointing -out physical objects, specific body parts or anything
else in the wearers field of view. 1.1) Possible scenarios: A soldier
is dispatched into an open desolate terrain, plumes of smoke off in the
distance indicate over night fighting between opposing factions in the city’s
civil war has been intense. As a steady stream of the displaced, injured
and confused refugees parade by, our warrior captures the scene with a head-mounted
stereo-video camera and binaural audio systems and display (EVA). These
3-D images and localizing audio are transmitted back to Base Camp via wireless
microwave where field surgeons evaluate the situation. Suddenly, the soldier
sees the surgeons hand come into view in his HMD and is guided to a young
man bend over in pain, the soldier places his sensor glove on the victim
and begins to transmit medical data, temperature: 104, pulse; 127, blood-ox:
83. The surgeon directs the soldier to ask the victim a few questions in
a foreign language, which he quickly repeats. Fever, malaise, cough and
sore throat are diagnosed, the alert base camp team then directs the soldier
to secure a throat culture and asks the victim if he can draw a blood sample.
The sample is drawn and a drop of serum is quickly prepared on a glass slide.
The soldier pushes a button on his waist and the cameras on his head zooms
into the sample on the glass slide. The lymphocyte count appears reduced
@1000 cell/mm3. An urgent message is relayed from base camp and a specialist
in toxicology is located at ECU in North Carolina. Information is uplinked
to SAT/COM via LAN and specialists put on their Head Mounted Displays.
2) Information Communication: Information,
both video and audio, is communicated via microwave spread-spectrum transmitters
and receivers. This communication system forms the core of the EVA System
which can be modularly expanded to include Satellite up and down link, and
internet connectivity. Both the wearer and receiver persons will have real-time
audio and video interactivity with abilities to communicate in foreign languages,
or technically difficult areas such as medical biology, explosive demolitions,
etc.
3) Training: Training for personnel will be minimal since the human
visual and auditory system can acclimate to being immersed in a 3-D video
and audio environment quite readily.
Summary of the System:
1) Integrated HMD-C/M: Head mounted Display - Cameras and Mics
2) Microwave Transmitters and Receivers: Spread Spectrum
3) Base Station - TBC, Multiplexing, Chroma Key, Satellite
EVA, Experiential Video and Audio, is a 3-D video and audio interactive telepresence system. Using two micro cameras and microphones on a head mounted display, EVA collects and transmits signals to a base camp where the 3-D video sources are instantly time base corrected, multiplexed and overlayed with vital information and transmitted back to the wearer in the field in real time. This allows two persons in remote locations to share information and expedite assistance in the form of hand gestures, text or video and ultimately to "act as one".
Statement of Work:
Task 1 Research and develop a high resolution head mounted display
and camera/microphone system.
R&D:
1) HMD
2) Micro-Video cameras
3) Mini-Microphones
Task 2 - Obtain and integrate spread spectrum microwave transmitters
and receivers.
R&D:
1) 2 transmitters and 1 receiver on wearer
2) 2 receivers and 1 transmitter at base station
Task 3 - Obtain and integrate vital components of the base station.
R&D:
1) TBC - dual channel time base corrector
2) Multiplexer - field sequential
3) Chroma Key + video camera
4) Distribution Amplifier
5) Head Mounted Display
6) Satellite/Internet up/downlink
Task 4 - Research and develop a vest based power supply with rechargeable
lithium-ion batteries.
R&D:
1) Flat lithium -ion bateries.
2) Vest designs - ergonomic power supply
3) Alternative - Solar, Fuel Cell, Hydrogen Peroxide, etc.
Task 5 - Systems testing and endurance predictability.
R&D:
1) All weather ability - a) Rain b) Snow c) Dirt and Mud
2) Time of day - a) Day b) Night - lights, night vision, starlight, etc.
3) Distance and Inteference - a) 2 mile radius b) Microwave alignments c)
Jamming interference - field detector and ambient EMF levels
4) Satellite up/down link and Internet connectivity - a) Rapid deployment
b) Portability
Task 6 - Education and community outreach.
1) Educational: Robotic TeleMedicine
2) Space Exploration: NASA EVA EVA
3) Police Department:
4) Fire/Paramedic Department:
Integration Into the Real World: Operation Strong Angel - June 10 - 16 - Wiamea, Hawaii Humanitarian assistance portion of RIMPAC 2000 Reviewed by: * NAVY 3rd Fleet * DARPA * ECU Tel-Med * SPAWAR * Japanese Defense Navy * NASA Medical Systems
