A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> Abstract This paper describes the development and testing of a system whereby one can communicate through a computer by using the P300 component of the event-related brain potential (ERP). Such a system may be used as a communication aid by individuals who cannot use any motor system for communication (e.g., ‘locked-in’ patients). The 26 letters of the alphabet, together with several other symbols and commands, are displayed on a computer screen which serves as the keyboard or prosthetic device. The subject focuses attention successively on the characters he wishes to communicate. The computer detects the chosen character on-line and in real time. This detection is achieved by repeatedly flashing rows and columns of the matrix. When the elements containing the chosen character are flashed, a P300 is elicited, and it is this P300 that is detected by the computer. We report an analysis of the operating characteristics of the system when used with normal volunteers, who took part in 2 experimental sessions. In the first session (the pilot study/training session) subjects attempted to spell a word and convey it to a voice synthesizer for production. In the second session (the analysis of the operating characteristics of the system) subjects were required simply to attend to individual letters of a word for a specific number of trials while data were recorded for off-line analysis. The analyses suggest that this communication channel can be operated accurately at the rate of 0.20 bits/sec. In other words, under the conditions we used, subjects can communicate 12.0 bits, or 2.3 characters, per min. <s> BIB001 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> This study assesses the relative performance characteristics of five established classification techniques on data collected using the P300 Speller paradigm, originally described by Farwell and Donchin (1988 Electroenceph. Clin. Neurophysiol. 70 510). Four linear methods: Pearson's correlation method (PCM), Fisher's linear discriminant (FLD), stepwise linear discriminant analysis (SWLDA) and a linear support vector machine (LSVM); and one nonlinear method: Gaussian kernel support vector machine (GSVM), are compared for classifying offline data from eight users. The relative performance of the classifiers is evaluated, along with the practical concerns regarding the implementation of the respective methods. The results indicate that while all methods attained acceptable performance levels, SWLDA and FLD provide the best overall performance and implementation characteristics for practical classification of P300 Speller data. <s> BIB002 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> A brain-computer interface (BCI) is a communication system that translates brain-activity into commands for a computer or other devices. In other words, a BCI allows users to act on their environment by using only brain-activity, without using peripheral nerves and muscles. In this paper, we present a BCI that achieves high classification accuracy and high bitrates for both disabled and able-bodied subjects. The system is based on the P300 evoked potential and is tested with five severely disabled and four able-bodied subjects. For four of the disabled subjects classification accuracies of 100% are obtained. The bitrates obtained for the disabled subjects range between 10 and 25 bits/min. The effect of different electrode configurations and machine learning algorithms on classification accuracy is tested. Further factors that are possibly important for obtaining good classification accuracy in P300-based BCI systems for disabled subjects are discussed. <s> BIB003 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> Brain–computer interface (BCI) systems do not work for all users. This article introduces a novel combination of tasks that could inspire BCI systems that are more accurate than conventional BCIs, especially for users who cannot attain accuracy adequate for effective communication. Subjects performed tasks typically used in two BCI approaches, namely event-related desynchronization (ERD) and steady state visual evoked potential (SSVEP), both individually and in a 'hybrid' condition that combines both tasks. Electroencephalographic (EEG) data were recorded across three conditions. Subjects imagined moving the left or right hand (ERD), focused on one of the two oscillating visual stimuli (SSVEP), and then simultaneously performed both tasks. Accuracy and subjective measures were assessed. Offline analyses suggested that half of the subjects did not produce brain patterns that could be accurately discriminated in response to at least one of the two tasks. If these subjects produced comparable EEG patterns when trying to use a BCI, these subjects would not be able to communicate effectively because the BCI would make too many errors. Results also showed that switching to a different task used in BCIs could improve accuracy in some of these users. Switching to a hybrid approach eliminated this problem completely, and subjects generally did not consider the hybrid condition more difficult. Results validate this hybrid approach and suggest that subjects who cannot use a BCI should consider switching to a different BCI approach, especially a hybrid BCI. Subjects proficient with both approaches might combine them to increase information throughput by improving accuracy, reducing selection time, and/or increasing the number of possible commands. <s> BIB004 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> This work introduces a hybrid brain-computer interface (BCI) composed of an imagery-based brain switch and a steady-state visual evoked potential (SSVEP)-based BCI. The brain switch (event related synchronization (ERS)-based BCI) was used to activate the four-step SSVEP-based orthosis (via gazing at a 8 Hz LED to open and gazing at a 13 Hz LED to close) only when needed for control, and to deactivate the LEDs during resting periods. Only two EEG channels were required, one over the motor cortex and one over the visual cortex. As a basis for comparison, the orthosis was also operated without using the brain switch. Six subjects participated in this study. This combination of two BCIs operated with different mental strategies is one example of a “hybrid” BCI and revealed a much lower rate of FPs per minute during resting periods or breaks compared to the SSVEP BCI alone ( FP = 1.46 ± 1.18 versus 5.40 ± 0.90). Four out of the six subjects succeeded in operating the self-paced hybrid BCI with a good performance (positive prediction value PPVb > 0.70). <s> BIB005 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> We present a feasibility study of hybrid brain- computer interface (BCI) for control of Functional Electrical stimulation (FES) during grasp rehabilitation. Proposed BCI operates in two stages sequentially. First stage is a steady-state visual evoked potential (SSVEP) based method for selection of one of the three objects for grasping. Second stage is event- related desynchronisation (ERD) based detection of the onset of reach-to-grasp selected object. In the first stage, adequate stimulation pattern for selected object is determined, while the second stage triggers FES to assist desired grasp. Mean detec- tion accuracies of SSVEP based selection and ERD based detection of the onset of movement for three subjects tested were 98.1% and 100% respectively. Obtained results suggest that this approach could be tested for the control of FES dur- ing Functional Electrical Therapy (FET). Hybrid BCI system is by definition composed of multiple BCIs or of a BCI and another system (13). We propose a hybrid system that combines one SSVEP based BCI for automatic selection of the grasping strategy and event- related desynchronisation (ERD) based BCI for triggering of FES. This hybrid BCI operates sequentially in two stag- es, Fig. 1. In the first stage system runs as an object for grasp selector. By object selection, type of grasp i.e. stimu- lation pattern is identified. This selection is based on detec- tion of SSVEP in the user's EEG from the visual cortex. SSVEPs can be evoked by focused visual attention on one of the objects equipped with flashing light sources of differ- ent frequencies. Three SSVEP frequencies are chosen to serve as indicators for three basic grasps: palmar, lateral and precision grasp. In the second stage after the adequate FES pattern has been identified, system waits for information on the onset of reaching movement towards selected object. This information can be extracted from ERD of EEG signal (14). <s> BIB006 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> Brain–computer interface (BCI) systems rely on the direct measurement of brain signals, such as event-related desynchronization (ERD), steady state visual evoked potentials (SSVEPs), P300s, or slow cortical potentials. Unfortunately, none of these BCI approaches work for all users. This study compares two conventional BCI approaches (ERD and SSVEP) within subjects, and also evaluates a novel hybrid BCI based on a combination of these signals. We recorded EEG data from 12 subjects across three conditions. In the first condition, subjects imagined moving both hands or both feet (ERD). In the second condition, subjects focused on one of the two oscillating visual stimuli (SSVEP). In the third condition, subjects simultaneously performed both tasks. We used logarithmic band power features at sites and frequencies consistent with ERD and SSVEP activity, and subjects received real-time feedback based on their performance. Subjects also completed brief questionnaires. All subjects could simultaneously perform the movement and visual task in the hybrid condition even though most subjects had little or no training. All subjects showed both SSVEP and ERD activity during the hybrid task, consistent with the activity in both single tasks. Subjects generally considered the hybrid condition moderately more difficult, but all of them were able to complete the hybrid task. Results support the hypothesis that subjects who do not have strong ERD activity might be more effective with an SSVEP BCI, and suggest that SSVEP BCIs work for more subjects. A simultaneous hybrid BCI is feasible, although the current hybrid approach, which involves combining ERD and SSVEP in a two-choice task to improve accuracy, is not significantly better than a comparable SSVEP BCI. Switching to an SSVEP BCI could increase reliability in subjects who have trouble producing the EEG activity necessary to use an ERD BCI. Subjects who are proficient in both BCI approaches might be able to combine these approaches in different ways and for different goals. <s> BIB007 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> In this paper, an asynchronous brain-computer interface (BCI) system combining the P300 and steady-state visually evoked potentials (SSVEPs) paradigms is proposed. The information transfer is accomplished using P300 event-related potential paradigm and the control state (CS) detection is achieved using SSVEP, overlaid on the P300 base system. Offline and online experiments have been performed with ten subjects to validate the proposed system. It is shown to achieve fast and accurate CS detection without significantly compromising the performance. In online experiments, the system is found to be capable of achieving an average data transfer rate of 19.05 bits/min, with CS detection accuracy of about 88%. <s> BIB008 </s> A review of hybrid brain-computer interface systems <s> SSVEP-Motor Imagery Hybrid BCI. <s> Brain-computer interfaces (BCI) provide a new communication channel between the human brain and a computer without using any muscle activities. Applications of BCI systems comprise communication, restoration of movements or environmental control. Within this study we propose a combined P300 and steady-state visually evoked potential (SSVEP) based BCI system for controlling finally a smart home environment. Firstly a P300 based BCI system was developed and tested in a virtual smart home environment implementation to work with a high accuracy and a high degree of freedom. Secondly, in order to initiate and stop the operation of the P300 BCI a SSVEP based toggle switch was implemented. Results indicate that a P300 based system is very well suitable for applications with several controllable devices and where a discrete control command is desired. A SSVEP based system is more suitable if a continuous control signal is needed and the number of commands is rather limited. The combination of a SSVEP based BCI as a toggle switch to initiate and stop the P300 selection yielded in all subjects very high reliability and accuracy. <s> BIB009
In order to acquire better understanding about a hybrid based on SSVEP and motor imagery BCI, short summaries of approaches are mentioned as follows. In BIB004 , the proposed hybrid was evaluated during the task and was applied under three conditions: ERD BCI, SSVEP BCI, and ERD-SSVEP BCI. During the ERD BCI task, two arrows appeared on the screen. When the left arrow appeared, subjects were instructed to imagine opening and closing their left hand. For the right arrow, subjects imagined opening and closing the corresponding hand. In the SSVEP task, subjects were instructed to gaze at either left (8 Hz) or right (13 Hz) LED depending on which cue appeared. In the hybrid task, when the left arrow was showed, subjects were imagining the left hand opening and closing while gazing at the left LED simultaneously. The task was similar for the right arrow. Results show the average accuracy of 74.8% for ERD, 76.9% for SSVEP, and 81.0% for hybrid. The number of illiterate subjects, who achieved less than 70% accuracy  , reduced to zero from five using the hybrid approach. A hybrid SSVEP/ERD BCI was introduced in BIB005 for orthosis control application. The SSVEP-based BCI was utilized for opening the orthosis at the activating stage, and an ERS-based BCI was used as a switch to deactivate the LEDs that were mounted on the orthosis for SSVEP evocation in the resting stage. The SSVEP-based stage entails four steps for opening and closing the orthosis completely. Frequencies 8 and 13 Hz LEDs were used for the opening and closing tasks, respectively. During training sessions, subjects were instructed to close the brain switch. Then, they were instructed to open and close the orthosis by gazing at the LEDs mounted on the orthosis. In the next stage, the SSVEPbased BCI was turned off by opening the brain switch. This switch was kept open during the resting period. At the end of the resting period, the brain switch was closed, and SSVEP task was repeated. After this experiment, subjects undertook the SSVEP-based BCI task alone and the LEDs were flickering during the resting period. For SSVEP detection, the power density spectrum was used. For the activity period, the true positive rate and false positive rate were measured and for resting period, the false positive rate. It was shown that false positive rate was reduced by more than 50% when hybrid BCI was utilized. SSVEP and ERD were combined in BIB006 to make a twostage hybrid BCI system for triggering a Functional Electrical Stimulation (FES) system. In the first stage, SSVEP was presented for object selection. For evoking SSVEP, three LEDs with 15, 17 and 19 Hz frequencies were considered. The EEG was acquired from O1, O2, and Oz channels while considering Cz as a reference. The object selection task represented three basic grasps: palmar, lateral and precision grasp. For the analysis, Oz channel was chosen as the SSVEP activity in this channel was more noticeable compared to other channels. For SSVEP detection, Butterworth's band pass filters were used to separate frequency bands and a threshold for each subject was fixed manually. After selecting one of the three grasp options based on SSVEP, the next task was reaching movement in which ERD-based BCI was used. EEG signals for this task were recorded from the C3 channel. The Cz channel was used as the reference point. The signal was filtered utilizing Butterworth's band pass filters. The detection algorithm was based on the real-time mu and beta band-power estimation. The signal was compared with the manual adjusted threshold and a drop under the threshold was considered as a movement command. 98.1% accuracy was achieved in the SSVEP stage. Using mu and beta bands, 100% and 98.1% accuracy were achieved, respectively. This study showed that the presented hybrid BCI can be considered as one of the appropriate combinations for FES triggering application. In general, hybrid BCIs are more complex and depending on the types of the BCIs, the difficulty may be increased and user acceptability may be decreased. The user acceptability can be measured based on a questionnaire from the subjects. In BIB007 , subjects found the hybrid BCI slightly more difficult than non-hybrid BCIs. In BIB007 , ERD and SSVEP were combined for a simultaneous hybrid task. Bipolar channels C3, Cz, C4, O1, and O2 were utilized for EEG recording. After training sessions, in the online run for SSVEP task, a cue pointed to the top LED, which was flickering with 8 Hz and then pointed to the bottom 13 Hz LED. Subjects received a real-time feedback from a rectangular appearing on the screen. During the ERD task, a cue pointed to the top of the screen and subjects imagined the opening and closing of both hands. When the cue pointed to the bottom of screen, subjects were instructed to imagine moving both feet. In the hybrid condition, both tasks were done simultaneously. The data from the training sessions was used for setting up the LDA classifier. The cross-validation classification accuracy was calculated for both online and the training sessions. In the training sessions, mean classification accuracy was 79.9%, 98.1%, and 96.5% for ERD, SSVEP, and hybrid condition, respectively. The analysis of the online performance showed that the mean classification accuracy was 76.9%, 94%, and 95.6% for ERD, SSVEP, and hybrid condition. For the same conditions maximum ITR was 3.2, 6.1, and 6.3 bits per min. In another analysis, the ERD and SSVEP features were classified separately in the hybrid BCI which showed that subjects were not doing only one of the tasks. Based on a questionnaire, two subjects indicated that hybrid BCI was much more difficult 4 Advances in Human-Computer Interaction and their performance declined compared to the SSVEP condition. Four subjects indicated that there was not any difference in difficulty of the hybrid condition compared to two other conditions and their performance stayed the same or improved in the hybrid condition. Overall from the questionnaire, the hybrid condition was moderately more difficult. Comparing the results of this experience with the previous one BIB004 , improvement was seen in the ERD results as the performance of the task had been changed (right hand versus left hand movement imagination in the previous study and both hand versus both feet movement imagination in this study). Other results, such as the lower accuracy in ERD condition and the higher performance in SSVEP condition, were consistent. The accuracy in the hybrid BCI is not significantly different from the SSVEP condition. By changing the classification or the combinations of the features, improvement in results may appear. However, the reliability of the system is improved as the SSVEP BCI is added to the conventional ERD BCI system. For subjects with low performance with ERD, or in the case of fatigue, the SSVEP BCI is appropriate option. 3.2. P300-SSVEP Hybrid BCI. P300 and SSVEP BCI were introduced as hybrids in an asynchronous BCI system in BIB008 . It seems the P300 and SSVEP combination worked well as the stimuli for evoking both patterns, which can be shown on one screen simultaneously. The P300 paradigm considered in this study is a 6 × 6 speller matrix based on the original P300 row/column paradigm introduced by Farwell and Donchin BIB001 . Only one frequency is allocated for the SSVEP paradigm. The background color was flashed with a frequency slightly less than 18 Hz. The background color change facilitates the SSVEP detection. During the classification, P300 and SSVEP signals were separated by a band pass filter. The SSVEP was utilized as a control state (CS) detection. When the user was gazing at the screen, the SSVEP was detected and it was assumed that the user intended to send a command. The system detected the P300 target selection and CS simultaneously. For SSVEP detection, the mean power spectral density (PSD) in the narrow band near the desired frequency and the PSD in the wider range near the desired frequency were utilized in an objective function. These values were subtracted from each other and divided over the PSD value from the wide band and the function value was compared to a specified threshold. During the data acquisition, the channels for acquiring EEG signals were not fixed for all subjects. For P300 classification, Bayesian linear discriminant analysis (BLDA) or Fischer's linear discriminant analysis (FLDA) were utilized BIB002 BIB003 . The experiment was presented as an offline and online test. Ten subjects participated in the experiment. In the offline test, forty characters were presented for detection, which were divided into four groups. For better evaluation, SSVEP was presented only to two groups out of four groups. In CS, subjects were instructed to count the number of times they distinguished the highlighted character. In the non-control state (NCS), subjects were instructed to do a mental task like multiplication of two numbers and relax with closed eyes. For four out of five subjects, the accuracy was improved insignificantly during the presence of SSVEP and P300 detection was not determinate. Between the ten character's detection, there was a break of a certain time, which was due to the subject pressing a keyboard button. When the NCS time was almost finished, an auditory cue alerted subjects. For P300 an average classification accuracy of 96.5% and control state detection accuracy of 88% with the information transfer rate (ITR) of 20 bits/min were achieved during the offline test. The online test was presented under a semi synchronous condition. The experiment consisted of blocks with five rounds, for detecting each character. SSVEP detection for at least three out of five runs showed the control state detection by the subject, and P300 was detected during the control state. If the control state was not detected, the "=" character would be shown on the screen. The break time and the auditory alert were the same as the offline test. An average control state detection accuracy of 88.15%, a classification accuracy of 94.44%, and an ITR of 19.05 bits/min were achieved during the online test. P300 and SSVEP combination was also introduced to control smart home environments in BIB009 . P300-based BCI was used for controlling the virtual smart home environment and SSVEP was implemented as a toggle switch for the P300 BCI operation. Results from this experiment show that P300 is suitable for discrete control commands and SSVEP is suitable for continuous control signals. The hybrid BCI achieved high accuracy and reliability in all subjects.