The EEG system BrainScope consists of a special amplifier system for high quality signal detection in open field conditions during communicative situations. A high performance computer system which is capable of processing the huge amounts of data produced by a multi channel EEG record to gain information in real-time has also been developed. Algorithms for the recognition of events in single channels are implemented in the first level of the computer system. High performance 3D image processing algorithms are used in the second level, interpreting the sampled values of each channel as pixels of the image, from 256 to 2000 times per second.

The network of two or more Personal Computers (PCs) is co-ordinated through the computer system for presentation of EEG activity and control. Multi-media approaches to the application of psychological tests are possible through the user interface including tests in media of sound, words, pictures and moving pictures as videos. These tests can be arranged and carried out in computer-controlled sequences and modified by user interactions. Tools are also provided to allow the user to create his own tests. These methods are integrated into the powerful Graphic User Interface and use a Data Base & Knowledge Management System. Incorporated into this User Interface are state of the art EEGSYS algorithms from the NIMH (Washington / USA) for mappings, FFT, etc.

The BrainScope demonstrates the impacts and applications of the new strategy for EEG investigation in communicative situations between:

- patient and physician for subjective evaluation,

- patient and information technology for stimulation and acquisition of signals and reactions,

- physician and information technology for quantitative analysis of signals and reactions.

The major advantage of this new strategy is that the three processes can be carried out

in real-time.

It optimises the capacity of humans to interpret information with the capability of modern information technology to manipulate and process data. It therefore requires use by an experienced and trained user who can make accurate observations during the process of an investigation. The user can, for example, click on a significant EEG pattern (this makes it a further recognisable phenomenon through Fuzzy Logic) and correlate it with his own observations. The computer system recognises this EEG activity, i.e. it interprets this as a possible description of the state of the brain, sets a defined stimulus and recognises and evaluates the Event Related Potential (ERP) immediately.

F4: Recognition of Event Related Potential

ERPs are EEG-changes, related to a particular event (e.g. acoustic or visual stimulus or motor reactions) and give hints to the underlying information process.

In Fig 5 the recognition of Event Related Potentials (ERPs) in the EEG are presented as N1, P2, N2 and P3 components in a single trial without averaging.

In the middle part is presented the averaged ERP, using methods of on-line averaging of the ongoing test. The ongoing signal is presented in the right part.

You can recognise the equivalences and differences between the averaged ERP and the activities in the single trial. The red line marks the start of the stimulus (Reiz = stimulus) and the blue = reaction. The generated ERPs are marked by coloured lines:

magenta = N1, – in the EEG, the negative signal is above!

brown = P2,

grey = N2,

black =P3

On the left coloured maps are presented using the powerful 3D-Mapping of the NIMH / Washington. Normally, each sample generates one map. Because of space restrictions in the Figure, only the significant 3D maps for the P3 component are shown. The white crosses are the symbols for the ERP-component and describe the evolution of the appearance from the start up to the extreme value of the amplitude. These ERPs can be seen in the many channels EEG in the right side of in Fig. 5 by the vertical line for the P3-component. The description of the P3-component in the complex 3D-signal can be easily recognized and manipulated by data base mechanisms and statistics.

The user can select such important or interesting activities with the mouse and define templates for the recognition of the activities in the ongoing complex signal. These predefined templates are chosen from either the EEG display or the ERP display. The user can name this object, e.g. P3 and store the features in the DB&KMS.

The ongoing signal display or the on-line averaged ERP can be examined stepwise by locating a line cursor and continuously clicking the mouse, each activities of the current click can be figured and displayed in a list box, the Select Window. In this Select Window are presented the numerical values of the marked activities. The user can name the selected activities as the P3 in the Fig. 5, can manipulate the proposed parameters of the description by using methods of Fuzzy Logic and can store this description in the DB&KMS. There is a user-friendly way to train the system to recognise specialised events as the ERPs in the complex signal.

However, the true value of the system becomes evident when it is trained to detect and estimate the activities of the complex signal. With the taught high performance computer system, the patterns can be recognised in milliseconds. The templates have to be selected to best represent the pattern which is intended to be recognised.

If a priori defined EEG-activity occurred, described by a stored sequence of activities during the EEG reading, the stimulus was given to the patient. What happens is that the taught high performance computer system recognises the sequence of activities and then starts a predefined action with a defined delay in the range of 50 milliseconds. We name this feature of the system: "stimulus, triggered by the state of the system".

If one wishes to recognise the function of the brain, one must use the most powerful tools: human intelligence, especially the natural language in communicative situations, supported by high performance computer systems, worked in real-time. "Moreover, humans can describe in words what they have just seen. They can also tell us what they are imagining or what they have just dreamt. It is almost impossible to get such information from a monkey." [Crick, p. 110]

In the course of a discussion, an experienced doctor or psychologist is able to obtain a very exact subjective diagnosis of a patient's condition; computer-aided psychological tests serve to strengthen these subjective diagnoses. During the course of these tests, it is often useful for the researcher to continue to examine the behaviour of the experimental subject, to vary the test based on the subject's personality and previously achieved test results, and to sharpen his professional opinion of the subject's condition.

But F. Crick described the problem: Nobody knew the context between EEG-activity and the underlying information process [Crick, p 111].

The communicative situation between patient, physician and computer system in this here described innovative way build the technical support for a new quality of brain research, gives a solution to Cricks problem.

F5: Real-Time Recognition of the ECG by using the high performance HeartScope

Heart-circulation-disease is a civilization disease with a high risk of fatality especially in industrial regions like Europe. At the same time, such industrialized regions exhibit frequently high unemployment patterns coupled with stress related syndromes on the persons considered which might contribute to such diseases. This is the motivation for innovative product development using national and international co-operation to overcome unemployment on the one hand and to recognize persons under increased health risk on the other.

The real-time recognition of the electrical activities of the heart, known as the electrocardiogram (ECG), is a unique feature of the developed system which uses powerful information technologies as its technical basis for continuous monitoring of patients at risk, and recognizes critical situations in real-time with highest accuracy. In addition, state-of-the-art communication technologies are also used for the transmission of relevant data to present the risk situation of the patient to a qualified physician for making decisions. The scientific background for this work is related to the project "Methods of Nonlinear Dynamic for analysis of the ECG, for risk stratification and therapy assessment for heart patients", supported by the German Ministry of Education and Research (Grant 13N7129) and active R&D-activities in close cooperation with pharmacological enterprises.

The ECG system HeartScope based on the technology and methods of the BrainScope.

Fuzzy logic and methods of AI are used to define and recognise activities as QRS-complexes and heart beats in real-time. The innovative methods realise a description of the signal, can be easily manipulated by mathematical methods and can be easily recognized.

The common recognition system recognizes the ECG and builds classes automatically. The experienced physician evaluates the ECG and the classes, marks significant and interesting signal structures in the ECG and in the classes as well as using a powerful Graphical User Interface. This process generates the formal description of the signal in detail, ECG-substructures (P, Q, R, S, T) up to heartbeats and classes. The critical pattern may be failing to identify proper ECG’s, e.g. by cardiac arrest, appearance of iterative ECG’s with an elevated S-T, P-Q segments, e.g. acute myocardial infarction (AMI), or grouped ECG’s with desperate elements, e.g. torsade de points. All of these critical patterns can be adapted individually to properly fit the properties of the individual patients. A permanent monitoring of patients at risks, e.g. after an AMI or at the beginning, during changes or respective dosage adaptation of a specific drug treatment (anti-arrhythmica) gives new insight in these processes. Each of these patterns with its descriptive details is stored in the DB & KMS permitting further analysis. If a pattern does not match with previously described patterns a new class of pattern will be opened. In this way several classes of patterns of usually occurring ECG’s arise and can be offered for matching. Note that patterns indicating harmful situation or being critical for surviving can be introduced in this pattern recognition program by the physician.

F6: Analysis, Feature Extraction, Classification, and Recognition in Real-Time

The application of the method is demonstrated for the single channel ECG using a sampling rate of 512 samples / sec. The classification of single channel ECG is presented in Fig. 7 (only the first red marked channel EKG2 is presented).

The red numbers signify the beat number of the 24-hour ECG, going from 70021 to 70037 in Fig. 7. The blue numbers are the markers for the Inter Beat Interval (IBI) and are going from 360 in minimum to 493 in this example. This high variation is generated by the pathological heartbeat 70022. The green numbers are the automatic generated classes and are going from 1 = garbage class to class 37 in this example.

Class 2 (Klasse 2, Anzahl = number = 35064 beats) and Class 3 (Klasse 3, Anzahl = number = 41999 beats) are the normal heartbeats, differentiated by the slope of R and S. The representation of these normal heartbeats of the ECG is quite different from the representation presented in the literature. The high flexibility of feature extraction, recognition and evolutionary algorithm for classification opened this innovative and flexible way.

Class 15 represents pathological heartbeats, 11 in one day. When Heartbeat 6161 appears first, it doesn’t match with one of the existent classes and is stored in the garbage class. The Heartbeat 9408 matches at first with heartbeat 6161, and they form the class 15. Only 5 Heartbeats are used as Templates for the class description (t_max= 5). Heartbeat 70022 is classified in class 15, and is a member of the class description and is presented in the ongoing ECG, in ECG-channel EKG2 in Fig. 7.

The experienced physician can evaluate the automatic generated classes, can name the classes, can delete non-significant classes, can joint classes and can introduce his knowledge to the automatically generated signal description by naming the classes, using established notions.

For optimal application of the portable the physician uses a two or multi step strategy:

In the first step, the recognition procedure runs on the stationary system HeartScope with high performance and visual evaluation by a qualified expert, a cardiologist.

In the second and further steps are used the portables PhysioCord for mobile ECG-monitoring and recognition of possible risk situations.

These methods for signal recognition worked with the highest accuracy, which employs the latest technologies in the fields of Data Base (DBS) and Knowledge Management Systems (KMS). Each incoming signal is stored in the DBS. These signals are segmented and indexed by the time and by the segment number. The classification is achieved using DB&KMS for the acquisition of personal knowledge in direct communication between the user and the DB&KMS. These stored and indexed signals in context with the introduced knowledge of the expert build up the background knowledge for a new quality of signal recognition.

Important is the demonstrated strategy, that the physician is aided by the computer system. The physician can introduce his own experience and evaluations using the User Interface and the DB & KMS.

The responsibility is in the hand of the physician, aided by powerful Information and Communication Technologies.

F7: PhysioCord – Intelligent Patient Monitoring using Micro Systems and lowest-power Technology

The progress of computer science has brought with it additional demands for the integration of various technologies onto the portable computer and at the same time has given rise to many novel applications. In the application presented here, the minimized portable PhysioCord is used with a standardized interface connecting the amplifier for ECG and other signals. Wireless data transmission is used.

The patterns which characterize the ECG and the risk situations are transmitted from the high-performance HeartScope and are the basis for the next step: The portable recognition system with low power technology is taught by templates for marked details, ECG-substructures, heart beats and class descriptions of the same patient with the same electrode positions. The recognition rate using this strategy is asymptotic near 100%. The recognition procedure is implemented on the high performance portable of the next generation, include high performance computer power and the DB&KMS for signal recognition in real-time named PhysioCord II. In this way, the system can recognize risk situations in real-time with highest accuracy, while connecting the rescue organization by handy-function and transmit relevant data for evaluation and decision making by the physician. The receiver has a powerful Data Base & Knowledge Management System with direct access. The physician can evaluate the received signals in context with the course of the illness and the stored signals in the database. He can decide with high accuracy in minimum time duration. The optimisation of this decision process is the key process in this step. This creates a high chance for saving risk patients in optimal time as illustrated in Fig 8.

F8: Next Generation Portable Microsystems – PhysioCord II

The further development requires highest performance of Information and Communication Technologies, miniaturisation and lowest power consumption. The aim is the integration of the power of the high performance stationary system in the portable by using innovative technologies as

- high precision analogue systems for data acquisition,

- integrated multiprocessor systems in one chip,

- FPGA with integrated processor core and

- integrated broadband wireless systems.

All these high performance systems must be designed with lowest power consumption for the portables.

F9: Next Generation System Architecture

The next Generation System Architecture is characterised by high performance portables. The portable is integrated by wireless data transmission technology for the direct communication in risk situations with high data transmission rates. The high performance stationary system with DB & KMS supports the physician by the evaluation of the incoming signals in risk situations.

F10: Micro System Technology for intelligent Patient Monitoring

F11: Communications in Computer Aided Systems CA*S

Our products, the BrainScope, the HeartScope in combination with the PhysioCord are realised as Computer Aided Systems for patient monitoring, named Patient Monitoring Systems - PatiMonS. This strategy is outlined in more detail:

Communications (C) can be considered as the most critical aspect in such Computer Aided Systems for different applications. Every component of the system relies on fast, accurate and secure communication links. In order to better handle the communications tasks, they have been divided into several meaningful entities:

C1: Communication physician – patient ;

C2: Communication between the patients;

C3: Communication patient and computer technology;

C4: Communication physician and computer technology by the user interface;

C5: Communication between physicians;

C6: Communication between the PatiMonS’s;

C7: signal acquisition and actions of the PatiMonS.

C4 has a special state and is used for the for acquisition of the knowledge of the physicians and is important for the definition of the responsibility, e.g. the physician must have all possibilities for the evaluation of the state of the patient by using the actual incoming data in context to the stored data and knowledge in the DB&KMS. The exchange of data and knowledge between the physicians (C5) by direct or technical supported communication (C5) is an important feature of the designed communication strategy.

F12: CARiManS - Computer Aided Risk Management Systems

The System Architecture of CARiManS - Computer Aided Risk Management System essentially handles all issues for the distributed operation of Risk Management Systems. This operation is necessary as to allow the Communication network to be web based, scalable, platform independent, and easy to use.

The CARiManS involve the following components:

· UI: The User Interface considers the communication with the Risk Manager (C4), aiming the controlling and is able to transfer the expert’s knowledge to CARiManS and realise the communication of the Monitored System with the CARiManS (C3). It also enables the communication between the Risk Manager and the Monitored System (C1) as to enable control of the former to the operation of the latter.

· MS: The unit Microsystems and Sensors handles the acquisition of the Monitored System signals as well as the controlling of the actuators e.g. multimedia technology for stimulation of patients in psychophysiological experiments.

· KM: Data Base & Knowledge Management System; the incoming signals are stored in the DB, indexed by time, recognised events, and structures. The KMS includes the knowledge of the experts, especially in context to the stored signals. XML-technologies are used for the design of the KMS and for the communication between technical systems.

· PD: Portable Devices, signal conditioning and conversion;

· TI: Transmission and Interfaces, security of data transmission; TI include terrestrial trucked radio or satellite technology for wide area communication, wireless LAN ad-hoc networks for disaster site hot spots, and personal or body area networks for frontline personnel, allowing them to act as data sources and synchronise by means of smart connected devices, e.g. robust mobile terminals and sensors.

· SP: Signal Processing; modules for signal analysis, classification, recognition, and evaluation. Special modules are designed for real-time signal recognition.

The CARiManS is designed for an integrated and improved risk management, providing an infrastructure that allows for horizontal and vertical information flow from the patient in risk up to the physician by means of a multi-level wireless signal and data transmission and communication infrastructure, as well as integrated applications that reflect the currently organizational structure adequate to the rescue effort.

F13: CATS – Computer Aided Teaching Systems

Through the concepts of direct communications among students, teacher, and computer, CARiManS special configurations are used for teaching and knowledge dissemination, for training and mentoring purposes, or simply for knowledge transfer, accessible from every region, Europe and elsewhere.

There are two scenarios:

- the case that a physician pushes his newly acquired knowledge towards his colleagues (teaching);

- and the opposite case, where a teacher or a student asks for state-of-the-art knowledge input from other experts (learning).

The modular architecture, consistent, multidisciplinary, and robust design of its components offers a significant impact on areas including skill-profiling of every professional accessing the CATS (Computer-Aided Teaching System), accessing the DB & KMSs (Data Base & Knowledge Management System) via web based user friendly interfaces, automatic notification of important knowledge as soon as new knowledge becomes available.

CATS are important tools for spreading of knowledge and excellence.

F14: References

1. F. Crick: The Astonishing Hypothesis. The Scientific Search for the Soul. Charles Scribner's Sons New York 1993.

2. G. Roscher: Informationsprozesse in technischen Systemen - Modelle und Methoden zur Erforschung des menschlichen Denkens. In Maas, J. F. Das sichtbare Denken, Modelle und Modellhaftigkeit in der Philosophie und den Wissenschaften. Rodopi 1993, S. 153 - 179. http://www.ICSRoscher.de/maas.htm

3. G. Roscher, W. M. Herrmann, K. Henning, D. Wendt, S. Fechner, F. Godenschweger, C. Weiß, E. Abel, A. Rijhwani, J. Martinez, N. Dahan, A. Karavas: Real-time recognition of EEG-activity as basic for a new strategy of psychopysiological experiments gives new insight in information processes in man. Clinical applications of advanced EEG data processing. Rome, May 8-9 1995, p. 47. http://www.ICSRoscher.de/Rom.htm

4. G. Roscher: Mustererkennung in komplexen Signalen in Echtzeit durch die Anwendung der Methoden der Künstlichen Intelligenz und den Einsatz leistungsfähiger Multiprozessorsysteme. Forschungsbericht der DFMRS 1996, ISSN 0944-694X, p. 77-94. http://www.ICSRoscher.de/DFMRS.htm

5. G. Roscher, G. Pogrzeba, D. Emde, F. Neubauer: Application of a Multi-Processor System for Recognition of EEG-Activities in Amplitude, Time and Space in Real-Time. In: E. H. D’Hollander, G. R. Joubert, F. J. Peters, U. Trottenberg (Eds.): Parallel Computing: Fundamentals, Applications and New Directions. 1998 Elsevier Science B. V. p. 89-96. http://www.ICSRoscher.de/Parco.htm

6. G. Roscher; A. Brattström; S. Cho; F. Francesconi; H. Hinrikus; R. Mauser; M. Packianather; G. Pogrzeba; A. Yankovskaya; V. Zvegintsev: Real-time Recognition of ECG by Using Powerful Information and Communication Technology for Intelligent Monitoring of Risk Patients. VDE-Verlag, Proceedings Volume 2, MICRO.tec 2000, Applications, Trends, Visions, VDE World Micro-technologies Congress, September 25.-27. 2000, EXPO 2000, Hannover, Germany, p. 759-762. http://www.ICSRoscher.de/EXPO2.htm

7. G. Roscher et al.: CAEPS - Computer Aided Environment Protection System. Expression of Interest, Sixth Framework Programme of the European Community for research, technological development and demonstration activities. June, 7. 2002. http://eoi.cordis.lu/docs/int_37066.pdf

8. G. Roscher et al.: PatiMon - Patient Monitoring. Expression of Interest, Sixth Framework Programme of the European Community for research, technological development and demonstration activities. June, 7. 2002. http://eoi.cordis.lu/docs/int_37068.pdf

9. G. Roscher et al.: RecoPhone - Recognition of Phonemes. Expression of Interest, Sixth Framework Programme of the European Community for research, technological development and demonstration activities. June, 7. 2002. http://eoi.cordis.lu/docs/int_37066.pdf

S1: Marcia Barinaga: „New Ion Channel May Yield Clues to Hearing. Science, 24. March 2000, Vol 287, p 2132-2133 and Science, 24. March 2000, Vol 287, p 2229-2234: „By studying the electrical currents passing through the membranes of hair cells as they are stimulated, they learned that hair-cell channels are stunningly fast, opening up within microseconds, compared to the milliseconds needed by biochemically activated channels. They are also exquisitely sensitive to the slightest movement and to direction; they open when the tip of the cell’s cilia bundle is deflected by a mere atom’s width – akin to bending the tip of the Eiffel Tower by the width of your thumb. If the cilia bundle moves one way, the channel opens; the other way and it shuts. The channels are also able to register tinny cilia movements on top of a larger constant deflection – a trait that lets us discern meaningful sounds from background noise.”