A review of hybrid brain-computer interface systems <s> Introduction <s> Abstract This paper describes the development and testing of a system whereby one can communicate through a computer by using the P300 component of the event-related brain potential (ERP). Such a system may be used as a communication aid by individuals who cannot use any motor system for communication (e.g., ‘locked-in’ patients). The 26 letters of the alphabet, together with several other symbols and commands, are displayed on a computer screen which serves as the keyboard or prosthetic device. The subject focuses attention successively on the characters he wishes to communicate. The computer detects the chosen character on-line and in real time. This detection is achieved by repeatedly flashing rows and columns of the matrix. When the elements containing the chosen character are flashed, a P300 is elicited, and it is this P300 that is detected by the computer. We report an analysis of the operating characteristics of the system when used with normal volunteers, who took part in 2 experimental sessions. In the first session (the pilot study/training session) subjects attempted to spell a word and convey it to a voice synthesizer for production. In the second session (the analysis of the operating characteristics of the system) subjects were required simply to attend to individual letters of a word for a specific number of trials while data were recorded for off-line analysis. The analyses suggest that this communication channel can be operated accurately at the rate of 0.20 bits/sec. In other words, under the conditions we used, subjects can communicate 12.0 bits, or 2.3 characters, per min. <s> BIB001 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> To understand the endogenous components of the event-related brain potential (ERP), we must use data about the components' antecedent conditions to form hypotheses about the information-processing function of the underlying brain activity. These hypotheses, in turn, generate testable predictions about the consequences of the component. We review the application of this approach to the analysis of the P300 component. The amplitude of the P300 is controlled multiplicatively by the subjective probability and the task relevance of the eliciting events, whereas its latency depends on the duration of stimulus evaluation. These and other factors suggest that the P300 is a manifestation of activity occurring whenever one's model of the environment must be revised. Tests of three predictions based on this “context updating” model are reviewed. Verleger's critique is based on a misconstrual of the model as well as a partial and misleading reading of the relevant literature. <s> BIB002 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Abstract This paper discusses an experimental communication system for severely disabled persons that utilizes electrical responses from the brain. Scientific background and methodology as well as the results of a preliminary evaluation are presented. <s> BIB003 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> When Jean-Dominique Bauby suffered from a cortico-subcortical stroke that led to complete paralysis with totally intact sensory and cognitive functions, he described his experience in The Diving-Bell and the Butterfly1 as “something like a giant invisible diving-bell holds my whole body prisoner”. This horrifying condition also occurs as a consequence of a progressive neurological disease, amyotrophic lateral sclerosis, which involves progressive degeneration of all the motor neurons of the somatic motor system. These ‘locked-in’ patients ultimately become unable to express themselves and to communicate even their most basic wishes or desires, as they can no longer control their muscles to activate communication devices. We have developed a new means of communication for the completely paralysed that uses slow cortical potentials (SCPs) of the electro-encephalogram to drive an electronic spelling device. <s> BIB004 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> A thought translation device (TTD) for brain-computer communication is described. Three patients diagnosed with amyotrophic lateral sclerosis (ALS), with total motor paralysis, were trained for several months. In order to enable such patients to communicate without any motor activity, a technique was developed where subjects learn to control their slow cortical potentials (SCP) in a 2-s rhythm, producing either cortical negativity or positivity according to the task requirement. SCP differences between a baseline interval and an active control interval are transformed into vertical or horizontal cursor movements on a computer screen. Learning SCP self regulation followed an operant-conditioning paradigm with individualized shaping procedures. After prolonged training over more than 100 sessions, all patients achieved self-control, leading to a 70–80% accuracy for two patients. The learned cortical skill enabled the patients to select letters or words in a language-supporting program (LSP) developed for inter-personal communication. The results demonstrate that the fast and stable SCP self-control can be achieved with operant training and without mediation of any muscle activity. The acquired skill allows communication even in total locked-in states. <s> BIB005 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Abstract An internally or externally paced event results not only in the generation of an event-related potential (ERP) but also in a change in the ongoing EEG/MEG in form of an event-related desynchronization (ERD) or event-related synchronization (ERS). The ERP on the one side and the ERD/ERS on the other side are different responses of neuronal structures in the brain. While the former is phase-locked, the latter is not phase-locked to the event. The most important difference between both phenomena is that the ERD/ERS is highly frequency band-specific, whereby either the same or different locations on the scalp can display ERD and ERS simultaneously. Quantification of ERD/ERS in time and space is demonstrated on data from a number of movement experiments. <s> BIB006 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Abstract A brain–computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) is presented which allows human subjects to observe and control changes of their own blood oxygen level-dependent (BOLD) response. This BCI performs data preprocessing (including linear trend removal, 3D motion correction) and statistical analysis on-line. Local BOLD signals are continuously fed back to the subject in the magnetic resonance scanner with a delay of less than 2 s from image acquisition. The mean signal of a region of interest is plotted as a time-series superimposed on color-coded stripes which indicate the task, i.e., to increase or decrease the BOLD signal. We exemplify the presented BCI with one volunteer intending to control the signal of the rostral–ventral and dorsal part of the anterior cingulate cortex (ACC). The subject achieved significant changes of local BOLD responses as revealed by region of interest analysis and statistical parametric maps. The percent signal change increased across fMRI-feedback sessions suggesting a learning effect with training. This methodology of fMRI-feedback can assess voluntary control of circumscribed brain areas. As a further extension, behavioral effects of local self-regulation become accessible as a new field of research. <s> BIB007 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> A brain-computer interface (BCI) gives those suffering from neuromuscular impairments a means to interact and communicate with their surrounding environment. A BCI translates physiological signals, typically electrical, detected from the brain to control an output device. A significant problem with current BCIs is the lengthy training periods involved for proficient usage, which can often lead to frustration and anxiety on the part of the user. Ultimately this can lead to abandonment of the device. The primary reason for this is that relatively indirect measures of cognitive function, as can be gleaned from the electroencephalogram (EEG), are harnessed. A more suitable and usable interface would need to measure cognitive function more directly. In order to do this, new measurement modalities, signal acquisition and processing, and translation algorithms need to be addressed. In this paper, we propose a novel approach, using non-invasive near-infrared imaging technology to develop a user-friendly optical BCI. As an alternative to the traditional EEG-based devices, we have used practical non-invasive optical techniques to detect characteristic haemodynamic responses due to motor imagery and consequently created an accessible BCI that is simple to attach and requires little user training. <s> BIB008 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> The paper describes work on the brain-computer interface (BCI). The BCI is designed to help patients with severe motor impairment (e.g. amyotropic lateral sclerosis) to communicate with their environment through wilful modification of their EEG. To establish such a communication channel, two major prerequisites have to be fulfilled: features that reliably describe several distinctive brain states have to be available, and these features must be classified on-line, i.e. on a single-trial basis. The prototype Graz BCI II, which is based on the distinction of three different types of EEG pattern, is described, and results of online and offline classification performance of four subjects are reported. The online results suggest that, in the best case, a classification accuracy of about 60% is reached after only three training sessions. The offline results show how selection of specific frequency bands influences the classification performance in singletrial data. <s> BIB009 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Brain–computer interface (BCI) systems support communication through direct measures of neural activity without muscle activity. BCIs may provide the best and sometimes the only communication option for users disabled by the most severe neuromuscular disorders and may eventually become useful to less severely disabled and/or healthy individuals across a wide range of applications. This review discusses the structure and functions of BCI systems, clarifies terminology and addresses practical applications. Progress and opportunities in the field are also identified and explicated. <s> BIB010 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Brain activity can be used as a control signal for brain–machine interfaces (BMIs). A powerful and widely acknowledged BMI approach, so far only applied in invasive recording techniques, uses neuronal signals related to limb movements for equivalent, multidimensional control of an external effector. Here, we investigated whether this approach is also applicable for noninvasive recording techniques. To this end, we recorded whole-head MEG during center-out movements with the hand and found significant power modulation of MEG activity between rest and movement in three frequency bands: an increase for ≤7 Hz (low-frequency band) and 62–87 Hz (high-γ band) and a decrease for 10–30 Hz (β band) during movement. Movement directions could be inferred on a single-trial basis from the low-pass filtered MEG activity as well as from power modulations in the low-frequency band, but not from the β and high-γ bands. Using sensors above the motor area, we obtained a surprisingly high decoding accuracy of 67% on average across subjects. Decoding accuracy started to rise significantly above chance level before movement onset. Based on simultaneous MEG and EEG recordings, we show that the inference of movement direction works equally well for both recording techniques. In summary, our results show that neuronal activity associated with different movements of the same effector can be distinguished by means of noninvasive recordings and might, thus, be used to drive a noninvasive BMI. <s> BIB011 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> In a conventional brain–computer interface (BCI) system, users perform mental tasks that yield specific patterns of brain activity. A pattern recognition system determines which brain activity pattern a user is producing and thereby infers the user’s mental task, allowing users to send messages or commands through brain activity alone. Unfortunately, despite extensive research to improve classification accuracy, BCIs almost always exhibit errors, which are sometimes so severe that effective communication is impossible. ::: ::: We recently introduced a new idea to improve accuracy, especially for users with poor performance. In an offline simulation of a “hybrid” BCI, subjects performed two mental tasks independently and then simultaneously. This hybrid BCI could use two different types of brain signals common in BCIs – event-related desynchronization (ERD) and steady-state evoked potentials (SSEPs). This study suggested that such a hybrid BCI is feasible. ::: ::: Here, we re-analyzed the data from our initial study. We explored eight different signal processing methods that aimed to improve classification and further assess both the causes and the extent of the benefits of the hybrid condition. Most analyses showed that the improved methods described here yielded a statistically significant improvement over our initial study. Some of these improvements could be relevant to conventional BCIs as well. Moreover, the number of illiterates could be reduced with the hybrid condition. Results are also discussed in terms of dual task interference and relevance to protocol design in hybrid BCIs. <s> BIB012 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Brain–computer interface (BCI) systems do not work for all users. This article introduces a novel combination of tasks that could inspire BCI systems that are more accurate than conventional BCIs, especially for users who cannot attain accuracy adequate for effective communication. Subjects performed tasks typically used in two BCI approaches, namely event-related desynchronization (ERD) and steady state visual evoked potential (SSVEP), both individually and in a 'hybrid' condition that combines both tasks. Electroencephalographic (EEG) data were recorded across three conditions. Subjects imagined moving the left or right hand (ERD), focused on one of the two oscillating visual stimuli (SSVEP), and then simultaneously performed both tasks. Accuracy and subjective measures were assessed. Offline analyses suggested that half of the subjects did not produce brain patterns that could be accurately discriminated in response to at least one of the two tasks. If these subjects produced comparable EEG patterns when trying to use a BCI, these subjects would not be able to communicate effectively because the BCI would make too many errors. Results also showed that switching to a different task used in BCIs could improve accuracy in some of these users. Switching to a hybrid approach eliminated this problem completely, and subjects generally did not consider the hybrid condition more difficult. Results validate this hybrid approach and suggest that subjects who cannot use a BCI should consider switching to a different BCI approach, especially a hybrid BCI. Subjects proficient with both approaches might combine them to increase information throughput by improving accuracy, reducing selection time, and/or increasing the number of possible commands. <s> BIB013 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> This work introduces a hybrid brain-computer interface (BCI) composed of an imagery-based brain switch and a steady-state visual evoked potential (SSVEP)-based BCI. The brain switch (event related synchronization (ERS)-based BCI) was used to activate the four-step SSVEP-based orthosis (via gazing at a 8 Hz LED to open and gazing at a 13 Hz LED to close) only when needed for control, and to deactivate the LEDs during resting periods. Only two EEG channels were required, one over the motor cortex and one over the visual cortex. As a basis for comparison, the orthosis was also operated without using the brain switch. Six subjects participated in this study. This combination of two BCIs operated with different mental strategies is one example of a “hybrid” BCI and revealed a much lower rate of FPs per minute during resting periods or breaks compared to the SSVEP BCI alone ( FP = 1.46 ± 1.18 versus 5.40 ± 0.90). Four out of the six subjects succeeded in operating the self-paced hybrid BCI with a good performance (positive prediction value PPVb > 0.70). <s> BIB014 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> A new approach to realise computer monitor flickers that can be used to elicit steady-state visual evoked potentials (SSVEP) at a flexible frequency is proposed. An SSVEP-based brain-computer interface (BCI) with 16 targets was then implemented using the proposed method. In an online test on three subjects, a high information transfer rate of 75.4 bits/min was achieved. <s> BIB015 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> Abstract For many years people have speculated that electroencephalographic activity or other electrophysiological measures of brain function might provide a new non-muscular channel for sending messages and commands to the external world – a brain–computer interface (BCI). Over the past 15 years, productive BCI research programs have arisen. Encouraged by new understanding of brain function, by the advent of powerful low-cost computer equipment, and by growing recognition of the needs and potentials of people with disabilities, these programs concentrate on developing new augmentative communication and control technology for those with severe neuromuscular disorders, such as amyotrophic lateral sclerosis, brainstem stroke, and spinal cord injury. The immediate goal is to provide these users, who may be completely paralyzed, or ‘locked in’, with basic communication capabilities so that they can express their wishes to caregivers or even operate word processing programs or neuroprostheses. Present-day BCIs determine the intent of the user from a variety of different electrophysiological signals. These signals include slow cortical potentials, P300 potentials, and mu or beta rhythms recorded from the scalp, and cortical neuronal activity recorded by implanted electrodes. They are translated in real-time into commands that operate a computer display or other device. Successful operation requires that the user encode commands in these signals and that the BCI derive the commands from the signals. Thus, the user and the BCI system need to adapt to each other both initially and continually so as to ensure stable performance. Current BCIs have maximum information transfer rates up to 10–25 bits/min. This limited capacity can be valuable for people whose severe disabilities prevent them from using conventional augmentative communication methods. At the same time, many possible applications of BCI technology, such as neuroprosthesis control, may require higher information transfer rates. Future progress will depend on: recognition that BCI research and development is an interdisciplinary problem, involving neurobiology, psychology, engineering, mathematics, and computer science; identification of those signals, whether evoked potentials, spontaneous rhythms, or neuronal firing rates, that users are best able to control independent of activity in conventional motor output pathways; development of training methods for helping users to gain and maintain that control; delineation of the best algorithms for translating these signals into device commands; attention to the identification and elimination of artifacts such as electromyographic and electro-oculographic activity; adoption of precise and objective procedures for evaluating BCI performance; recognition of the need for long-term as well as short-term assessment of BCI performance; identification of appropriate BCI applications and appropriate matching of applications and users; and attention to factors that affect user acceptance of augmentative technology, including ease of use, cosmesis, and provision of those communication and control capacities that are most important to the user. Development of BCI technology will also benefit from greater emphasis on peer-reviewed research publications and avoidance of the hyperbolic and often misleading media attention that tends to generate unrealistic expectations in the public and skepticism in other researchers. With adequate recognition and effective engagement of all these issues, BCI systems could eventually provide an important new communication and control option for those with motor disabilities and might also give those without disabilities a supplementary control channel or a control channel useful in special circumstances. <s> BIB016 </s> A review of hybrid brain-computer interface systems <s> Introduction <s> A brain-computer interface (BCI) enables communication without movement based on brain signals measured with electroencephalography (EEG). BCIs usually rely on one of three types of signals: the P300 and other components of the event-related potential (ERP), steady state visual evoked potential (SSVEP), or event related desynchronization (ERD). Although P300 BCIs were introduced over twenty years ago, the past few years have seen a strong increase in P300 BCI research. This closed-loop BCI approach relies on the P300 and other components of the ERP, based on an oddball paradigm presented to the subject. In this paper, we overview the current status of P300 BCI technology, and then discuss new directions: paradigms for eliciting P300s; signal processing methods; applications; and hybrid BCIs. We conclude that P300 BCIs are quite promising, as several emerging directions have not yet been fully explored and could lead to improvements in bit rate, reliability, usability, and flexibility. <s> BIB017
A brain-computer interface (BCI) system can provide a communication method to convey brain messages independent from the brain's normal output pathway BIB016 . Brain activity can be monitored using different approaches such as standard scalp-recording electroencephalogram (EEG), magnetoencephalogram (MEG), functional magnetic resonance imaging (fMRI), electrocorticogram (ECoG), and near infrared spectroscopy (NIRS) BIB016 BIB007 BIB011 BIB008 . However, EEG signals are considered as the input in most BCI systems. In this case, BCI systems are categorized based on the brain activity patterns such as event-related desynchronization/synchronization (ERD/ERS), steady-state visual evoked potentials (SSVEPs), P300 component of event related potentials (ERPs), and slow cortical potentials (SCPs) BIB004 BIB003     BIB009 BIB010 BIB001 BIB012 BIB013 BIB014 . Each BCI type has its own shortcoming and disadvantages. To utilize the advantages of different types of BCIs, different approaches are combined, called hybrid BCIs BIB013 BIB014 . In a hybrid BCI, two types of BCI systems can be combined. It is also possible to combine one BCI system with another system which is not BCI-based, for example, combining a BCI system with an electromyogram (EMG)-based system. However, one can debate if this type of system should be defined as hybrid BCI. In the rest of this paper, we assume that if an EEG BCI system is combined with another physiological signal (e.g., EMG) based system, a hybrid BCI system will be constructed. Although different BCI methods can be combined, it should be noted that not all combinations of different brain imaging methods are feasible and possible. One of the limiting factors is the technology. For example, although MEG is a very high resolution brain imaging technique, it is not practical to use it when subjects need to move around. In addition, different techniques and their combinations should be utilized based on the application that the hybrid BCI is going to be used for. The main purpose of combining different systems to form a "hybrid" BCI is to improve accuracy, reduce errors, and overcome disadvantages of each conventional BCI system. Different types of hybrid BCI systems can be defined according to the types of systems which are combined, how systems are joined, and what types of inputs are considered. In non-hybrid BCIs, based on the property of EEG signals used as the input of BCI system, four major EEG-based BCIs are considered: SSVEP, P300, SCP, and ERD/ERS. Evoked response in EEG signals to repetitive visual stimulations is called SSVEP. In a SSVEP BCI paradigm, specific frequencies are allocated to the repetitive stimuli. For SSVEP detection, the frequency spectrum of the EEG is computed. Around the frequency of the repetition of stimulus in which the subject focuses, there will be peak on the frequency spectrum. By detecting this frequency, an intention of the subject can be detected. This can be translated to a control signal for a BCI system. There are some issues about SSVEP BCIs, one is gaze dependence   . Another issue is that in some users, the flickering stimulus is annoying and produces fatigue. Using higher frequencies for the flickering stimuli reduces the annoyance, but on the other hand, it is harder to detect the SSVEP. SSVEP BCIs have some advantages such as no significant training requirement and high information transfer rate BIB003  BIB015 . In P300-based BCIs, intention of the subjects is measured using the P300 component of the brain evoked response BIB017 . After stimulus onset, positive and negative deflections occur in the EEG. These deflections are called event-related potential (ERP) components. Depending on the latency of these deflections, they are grouped as "exogenous components" and "endogenous components"  . The exogenous components occur until about 150 msec after the eliciting stimulus. The endogenous components have longer latency. The largest positive deflection that occurs between 250 to 750 msec after the stimulus onset is called "P300". The P300 component is the most used ERP component in BCI systems. The paradigm that elicits P300 is called the "oddball paradigm" BIB002 . In an oddball paradigm, events that elicit the P300 fall into two classes in which one of the classes is less frequent. Interstimulus interval time and the frequency of the oddball stimulus are among the parameters that determine the amplitude of the P300 component. The first BCI P300-based system was introduced by Farwell and Donchin for spelling characters in 1988 BIB001 . Slow negative voltage shifts that occur in the EEG recorded over sensorimotor cortex, while actual or imagined movement happens  are called SCP. SCP-based BCI consists of series of trials  . Early SCP BCIs were especially slow, since in each trial only one selection was possible and the time needed for each selection was at least 10 sec. The temporal efficiency was improved by Kübler et al. to 4 sec BIB005 . Shortening the time process further was not possible because users were uncomfortable with the shortened trial time. Over the past decade studies about the SCP approach have been limited because of several SCP BCI problems, which reduce the applicability of this type of BCI. Among others, SCP BCIs have three main problems: poor multidimensional control, high probability of error, and long-term training. Rhythmic activity of EEG in terms of event-related desynchronization/synchronization (ERD/ERS) has been used as one of the sources in BCI BIB016 . Motor imagery is one way to induce changes in ERD/ERS and has been used in many BCI systems BIB006 . During motor imagery of movements, ERD occurs predominantly over the contralateral brain motor area and, therefore, can be used as a signal for a BCI system. ERD/ERS BCIs have been used in different applications such as achieving two-dimensional cursor control.