Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing. However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces. To this end, we aim to create a BCI that decodes text directly from neural signals. We implement a framework that initially isolates frequency bands in the input signal encapsulating differential information regarding production of various phonemic classes. These bands form a feature set that feeds into an LSTM which discerns at each time point probability distributions across all phonemes uttered by a subject. Finally, a particle filtering algorithm temporally smooths these probabilities incorporating prior knowledge of the English language to output text corresponding to the decoded word. Further, in producing an output, we abstain from constraining the reconstructed word to be from a given bag-of-words, unlike previous studies. The empirical success of our proposed approach, offers promise for the employment of such an interface by patients in unfettered, naturalistic environments. Neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS) restrict an individual's potential to fully engage with their surroundings by hindering communication abilities. BrainComputer Interfaces (BCI) have long been envisioned to assist such patients as they bypass affected pathways and directly translate neural recordings into text or speech output. However, practical implementation of this technology has been hindered by limitations in speed and accuracy of existing systems [4] . Many patients rely on devices that use motor imagery [10] , or on interfaces that require them to individually identify and spell out text characters such as the "point and click" cursor method 3) A bLSTM creates probability distributions over phonemes at each time point. 4) Probabilities are smoothed and domain knowledge is incorporated using a probabilistic automaton traversed using a particle filtering algorithm. 5) The highest probability word is chosen as the output. [11, 12] . Despite significant work in system optimization, inherent limitations in their designs render them significantly slower than spoken communication. To address these shortcomings, several studies are using electrocorticography (ECoG) and local field potential (LFP) signals [2] . These invasive approaches provide superior signal quality with high temporal and spatial accuracy. Previous work attempted translation to continuous phoneme sequences using invasive neural data [8] ; however, despite their reported higher translation speed, their applications are limited to a reduced dictionary (10-100 words). Other design choices meant to enhance phoneme classification capitalize on prior knowledge of the target words, hindering their generalization to unmodified scenarios. Additionally, a recent study synthesized speech using recordings from speech cortex. Though the authors demonstrate partial transferrability of their decoder amongst patients, their accuracy is again limited to selection of the reconstructed word by a listener from a pool of 25 words and worsens as the pool size increases [3] . Thus, establishing the capability of these approaches to generalize to unconstrained vocabularies is not obvious and has to our knowledge not yet been studied. Here, we present the performance of a two-part decoder network comprising of an LSTM and a particle filtering algorithm on data gathered from six patients. We provide empirical evidence that our interface achieves an average accuracy of 32% calculated against a full corpus, i.e. one encompassing all feasible English words that can be formulated using the entire set of phonemes uttered by a patient, thus marking an important, non-incremental step in the direction of viability of such an interface. Each of the subjects in this study were able to communicate with significantly higher accuracy than chance. Nevertheless, the average word error rate seen in this study (67.8% on average) was higher than the 53% reported in [3] . There were several important differences in these studies, however. The primary difference is that their system produced an audio output that required a human listener to transcribe into a word selection. Despite advances in machine learning and natural language processing, humans have superior ability to use contextual information to find meaning in a signal. Furthermore, that study limited classifications to an output domain set of 50 words, which is generally not sufficient for a realistic communication system. While this study makes a significant addition to existing BCI literature in terms of its avoidance of the traditional bag-of-words approach, our accuracies are lower than those reported in ERP-based BCI studies [12] . Moreover, in order for a BCI system based on translating neural signals to become a practical solution, improvements need to be made either in signal acquisition, machine learning translation, or user strategy. One approach could be to sacrifice some of the speed advantages by having users repeat words multiple times. While this would reduce communication speeds below natural speaking rates, it would still greatly exceed ERP-based methods, while increasing the signals available for classification which could improve system accuracy. However, both this study and previous literature have primarily been concerned with decoding speech/text for patients with intact motor abilities. It is presently unclear how this would translate to intended speech. While the electrodes used in this study are inept to answer this question, given their majority location in the speech cortical areas [9] , we suggest a plausible new experiment: teaching those who can't speak to rethink speech in terms of vocal tract movements. Using electrodes in the sensorimotor cortex [3] and continuous visual feedback of ground truth vocal tract movements for each phoneme's pronounciation, a subject's attention could be entrained to only the (intended or executed) motion of their vocal tract for covert and overt speech respectively. One can then test the transferability of state space models -latent variables comprising of different articulators and observed states corresponding to the time-varying neural signals -between the covert and overt behaviours to better understand and harness the physiological variability between the two to eventually translate current studies into potentially viable devices. The proposed system serves as a step in the direction of a generalized BCI system that can directly translate neural signals into written text in naturalistic scenarios. However, communication accuracies are currently insufficient for a practical BCI device, so future work must focus on improving these and developing an interface to present feedback to users.
We present an open-loop brain-machine interface whose performance is unconstrained to the traditionally used bag-of-words approach.
{"src": "scitldr-AIC_train", "paper_id": "B1lj77F88B"}