Too Fine or Too Coarse? The Goldilocks Composition of Data Complexity for Robust Left-Right Eye-Tracking Classifiers

The differences in distributional patterns between benchmark data and real-world data have been one of the main challenges of using electroencephalogram (EEG) signals for eye-tracking (ET) classification. Therefore, increasing the robustness of machine learning models in predicting eye-tracking positions from EEG data is integral for both research and consumer use. Previously, we compared the performance of classifiers trained solely on finer-grain data to those trained solely on coarse-grain. Results indicated that despite the overall improvement in robustness, the performance of the fine-grain trained models decreased, compared to coarse-grain trained models, when the testing and training set contained the same distributional patterns \cite{vectorbased}. This paper aims to address this case by training models using datasets of mixed data complexity to determine the ideal distribution of fine- and coarse-grain data. We train machine learning models utilizing a mixed dataset composed of both fine- and coarse-grain data and then compare the accuracies to models trained using solely fine- or coarse-grain data. For our purposes, finer-grain data refers to data collected using more complex methods whereas coarser-grain data refers to data collected using more simple methods. We apply covariate distributional shifts to test for the susceptibility of each training set. Our results indicated that the optimal training dataset for EEG-ET classification is not composed of solely fine- or coarse-grain data, but rather a mix of the two, leaning towards finer-grain.


INTRODUCTION
Recently, machine-learning classifiers have found more and more consumer applications as well as "on the field" usage. For example, machine learning models can be used to identify shopping motives relatively early in the search process [19], to detect workload strain in truck drivers [13], and to assess the diagnosis of many neurological diseases such as Autism Spectrum Disorder and Alzheimer's [4,11,30,31]. Machine learning approaches have also shown adequate performance on computer vision bioinformatics applications, medical image analysis, Eye-Tracking (ET) analysis, and Electroencephalography (EEG) analysis [1, 3, 8, 15, 22, 23, 25-27, 29, 36, 37, 40].
As machine learning classification is used in more and more unfamiliar environments, it is increasingly important for these classifiers to be robust. In the EEG-ET research, recent work in this area has focused on determining what machine learning models are best equipped to predict eye position from EEG signals [15] as well as eliminating the noise associated with EEG data collection automatically [17,21,24,28,33,39].
These approaches all examine robustness across a variety of underlying factors of data analysis. In this paper, We focus on the inherent distributional patterns and the underlying differences between finer-grain and coarser-grain data. For this paper, finer-grain data refers to data collected in a more complicated framework in an environment with more uncontrolled conditions. On the contrary, coarser-grain data refers to data collected in a more simplified collection format with more restrictions on the experiment's environment for the same task.
In medical research, fine-grain data collection methods have been explored with great success [9,20]. That is the use of data from more complex, or finer-grained, data collection methods to test for simpler tasks. Oftentimes, systems developed in a coarsegrained "lab conditions" only work in controlled environments, causing difficulties when utilized in uncontrolled conditions [16,34]. Recently, fine-grain data approaches have shown successful results for Covid-19 contact tracing machine learning classifiers [7]. In this paper, We want to emphasize the impact of data granularity as well as the beneficial effects of using fine-grain data in machine learning classification.
Previously, we showed that Machine Learning models trained on fine-grained data are more robust, meaning that they maintain more LG Tasks (Right) [12] consistent accuracies, than models trained on coarse-grained data when tested on data of different complexities [35]. We extend upon our previous findings by examining the idea of training on mixed fine-and coarse-grain datasets. To our knowledge, this approach was not tested with machine learning classifiers before. The goal of mixing datasets of varying data granularity is to improve upon the draw-back we previously found with training on purely finegrain data. That is the model's performance when tested on data of similar complexity drops a significant amount when compared to purely coarse-grain data. We suspect that a mixed complexity training set may perform adequately across a broader range of testing distributional patterns, including different and similar complexities.
In this study, we train machine learning models for left-right eyetracking classification using data from a mix of binary-classified (coarse-grain) and vector-based (fine-grain) collection frameworks. We then compare the results to models trained exclusively on either binary-classified or vector-based data. The goal is to expand upon our previous conclusions regarding the optimal method to increase both accuracy and robustness of machine learning classifiers using fine-and coarse-grain data. Robustness is determined by the accuracy after a covariate distributional shift. The distributional shift in combination with the different mixes of data complexity attempts to mimic realistic data which often contains varying distributional patterns. Since we had previously determined the superiority of fine-grain data in terms of robustness, we hypothesize that classifiers trained using more fine-grain data will attain higher accuracies after covariate distributional shifts are applied.
The purpose is to determine the "Goldilocks" or optimal training set composition of fine-and coarse-grain data for left-right gaze classification in terms of both accuracy and robustness as well as verify that fine-grain data performs better than coarse-grain data using a more encompassing experimental design.

DATA
We used the EEGEyeNet dataset due to its large size [12]. The dataset contains simultaneously-recorded EEG signals and Eye-Tracking data from three different experimental tasks. A summary table for the three tasks' metadata is in Appendix D. In this study, we used data from two experiments: pro-antisaccade (PA) and Large Grid (LG). In the PA trials, participants were asked to focus on a cue that appears on either the screen's center, horizontally left, or horizontally right, as shown in the left screen in Figure 1. Gaze positions in PA were restricted to the horizontal axis and were binary-classified either left or right, relative to the screen's center. In the LG trials, participants were asked to fixate on dots presented one at a time at 26 different screen positions, as shown in the right  Figure 1. Therefore, LG's framework enabled finer-grain data collection since it allowed more freedom for eye movements, and gaze positions were encoded in a two-dimensional format including both angle and amplitude, compared to the one-liner binary-encoded eye movements in PA.

EXPERIMENT DESIGN
The learning objective of the models trained in our experiment was to use EEG brain signals to predict the direction of a subject's gaze along the horizontal axis (whether they are looking to the left or the right). Although predictions for this task using the same dataset were previously made, they were performed exclusively using data from the PA tasks for training and testing [12].
In this paper, we train 13 classifiers using data that is composed of a mixture of both PA and LG data and perform a covariate distributional shift by comparing their performance based on their accuracy when tested on PA and LG data. The different compositions are described in Figure 2

Data Processing
Given the classification nature of our learning problem, data from the LG task, encoded as Angle and Amplitude, should be transformed and relabeled into the expected format as left or right for training left-right classification models. The convention we used for the relabelling process was that for angle ; when | | < 2 the data is classified as right, otherwise the data is classified as left. This logic was confirmed by the dataset's authors in Appendix A and is shown in Figure 3.

Models Training
As per EEGEyeNet authors' recommendation, we used the minimally preprocessed EEG data. Data processing was done using the NumPy library, and the model implementations were installed from the SKlearn library [18]. From a broader perspective, models were   LG PA Mixed  LG PA Mixed (60- 40) LG PA Mixed (80-20) LG PA trained and tested on 4 different combinations: models trained on mixed data and tested on PA data, models trained on mixed data and tested on LG data, models trained on PA data and tested on mixed data, and models trained on LG data and tested on mixed data. For each of the 4 combinations, the mixed data had 6 different compositions of PA and LG data, shown in Figure 2, which lead to 24 different combinations. However, 4 of these combinations are counted twice, leading to only 20 different combinations. A summary of the 20 unique combinations is shown in Table 1.

MODELS 4.1 Machine Learning Models
In this study, machine learning models operate on features extracted from the data rather than the data itself. Feature extraction has been applied in two steps. First, [12] applied a band-pass filter in frequencies in the range [8 -13 HZ] on the acquired signals through all trials. This choice of frequencies is based on suggestions from [6]. Following the filtering step, the Hilbert transform was applied, resulting in a complex time series from which targeted features were extracted for learning models. Since we are considering a classification problem, we experimented with classification-only models and models that can be applied to both classification and regression problems.

Linear Classifiers:
Linear classifiers gather discriminant classifiers that use linear decision boundaries between the feature vectors of each class [15]. In this paper, we will use Linear and Radial-Basis-function Support Vector Classifiers (SVC) and Normal and Shrinkage Linear Discriminant Analysis (LDA) classifiers. All four algorithms still perform well after several decades since they were first implemented in this field, with SVC outperforming other classifiers, especially for two-classes problems [2,14,15].

Ensemble Classifiers.
Ensemble (or voting) classifier is a machine learning classification algorithm that trains with different classification models and makes predictions through ensembling their predictions to make a stronger classification. These algorithms have been the gold standard for several EEG-based classification experiments [25]. In our study, we used Random Forest, XGBoost, GradientBoost, RUSBoost, and AdaBoost.

Naive Bayes and Decision Tree Classifiers. Naive Bayes (NB)
Classifier is the statistical Bayesian classifier [5]. It assumes that all variables are mutually correlated and contribute to some degree towards classification. It is based on the Bayes' Theorem and is commonly used with high dimensional inputs. On the other hand, a decision tree is not a statistically based one; rather, it is a data mining induction technique that recursively partitions the dataset using a depth-first greedy algorithm until all data is classified with a particular class. Both NB and the decision tree are relatively fast and well suited for large data. Furthermore, they can deal with noisy data, which makes them well suited for EEG classification applications [10].

Deep Learning Models
Deep learning is a subfield of machine learning algorithms in which computational models learn features from hierarchical representations of input data through successive non-linear transformations [29]. Deep learning methods, especially Convolutional Neural Network (CNN), performed well in several previous EEG band power (feature) based research [25]. Still, these methods have not demonstrated convincing and consistent improvements [15]. Given so and the expected high run time for such algorithms due to the dataset's large size, we only included a simple and relatively fast multi-layer perception neural network (MLP) in our experiment.

RESULTS
We trained and tested the machine learning models using datasets composed of both fine-and coarse-grain data (pro-antisaccade and "transformed" large grid) and compared the results. Thus, we had    20 combinations of training and testing datasets described in Table  1.
Determining the Goldilocks composition of fine-and coarsegrain data for training is the main objective of this paper as it will indicate the optimal method of maintaining high accuracy and consistency. We also verified the results of fine-versus coarse-grain data by finding the accuracies of models when trained on PA/LG and tested on mixed data.
Results regarding the accuracies of select models as well as the average accuracy for each combination are provided in Tables 2, 3, 4, and 5. Table 6 is a summary of Tables 2 and 3.
The average accuracy of models trained on PA and tested on mixed is 90.2% as shown by the average of the averages in Table 4. The average accuracy of models trained on LG and tested on mixed is 91.3% as shown by the average of the averages in Table 5. The accuracy of models train on LG is also more compact and closer to the average accuracy. Clearly, finer-grain data is in general more applicable to real life data as it is more accurate across a broader range of distributional patterns. This confirms our previous findings across a more complete range of data complexity.
Note that although we did not include standard deviations in our tables, we did calculate them, but since they were insignificant (generally less than 0.1%) we decided to not include them in our results.  Table 6 describes the average accuracy from 13 different machine learning model for left-right ET classification trained on data of different data complexity compositions. This paper utilizes the benchmark data from the EEGEyeNet dataset and 13 machine learning models to create left-right classifiers trained on data of varying compositions of fine-and coarsegrain data. The accuracies of the models are then tested using two testing sets with distinctly different distributional patterns (PA and LG). In this way, the effects of covariate distributional shifts are more apparent and provide insight on the optimal data complexity composition of fine-and coarse-grain data in order to reduce the impact of such phenomenon. The results also provide useful information on the types of machine learning classifiers that should be used for EEG-ET classification tasks.

Finding the Goldilocks Composition
Previously, we had concluded that machine learning classifiers trained purely on fine-grain data outperform machine learning classifiers trained purely on coarse-grain data in terms of robustness. This was confirmed by Tables 4 and 5. Therefore, we theorized that the ideal composition of data complexity would lean towards finergrain data.
The hypothesis is confirmed by Table 6. The composition of 20% coarse-grain data and 80% fine-grain data produced the most consistently accurate results as shown by the average percentages. This suggests that the optimal training set for EEG-ET classification should not be created using data of solely one distributional complexity, but rather a mix of both fine-and coarse-grain data, leaning towards more fine-grain data.

Determining the Best Classifiers for EEG-ET Classification
The consistently most accurate classifiers across Tables 4, 5, 2, and  3 are XGBoost, GradientBoost, and RandomForest. Alternatively, the consistently worst performing classifiers were RBF SVC and GaussianNB. Therefore regardless of distributional complexity, XG-Boost, GradientBoost, and RandomForest should be used to classify EEG-ET data.

Future Recommendations and Improvements
To further advance the work provided in this study, four steps are highlighted for future exploration. Although deep learning models were excluded due to inconsistent results [15], the main reason was due to restraints in time and resources. As deep learning models, especially CNN and Attention [32,38], have shown promising results in regards to EEG-ET classification, it is important to thoroughly explore the effects of fineversus coarse-grain data on the robustness of such models.
Additionally, the angle value is currently the only indicator used to determine whether the saccade is towards the left or the right. The amplitude was completely ignored in data processing. The amplitude could be incorporated by using it as a weighting/scaling factor to indicate the left-right extension and solve this as a regression problem. Based on the predictions made by the regression value, we can then classify it as either left or right utilizing finergrain data.
Furthermore, although we determined an approximately optimal training set, we are almost certain this is not the Goldilocks composition for mixed data complexity. Perhaps an application of gradient descent would be able to more precisely compute such a composition for each machine learning model and specific machine learning task. Additionally due to time restraints, we were unable to test our machine learning models trained on mixed distributional complexity against a broader range of testing sets like we did for exclusively fine-or exclusively coarse-grain trained classifiers in Tables 4 and 5.
Although our work discusses EEG-ET classification, the results can be potentially relevant to other machine learning models in other applications. We highly encourage further investigation on training machine learning models on data composed of different granularities across an array of other research topics.

CONCLUSION
The motivation behind this work was to extend upon our previous work on determining whether training machine learning models on finer-grain data leads to more robust models as well as to analyze the effects of training machine learning classifiers on datasets composed of mixed data complexity. We verified our previous findings across a broader range of distributional patterns as well as determined that ensemble methods are the most suitable machine learning classifiers for EEG-ET classification tasks. Furthermore we identified that EEG-ET machine learning classifiers seem to produce the most consistent results when the training set contains a mix of distributional patterns, leaning towards finer-grain data. We hope that future applications of practical EEG-ET interfaces utilize data of varying distributional patterns to increase classification accuracy and robustness.

APPENDICES 9.1 Appendix A
After emailing Ard Kastrati, Ard verified that for angle in radians: = 0 is right = 2 is down = is left = − 2 is up

Appendix B
The models were trained and tested on this environment settings: The total space occupied by the dataset on the device is 69.0574 GB, and the total time for training and testing was 30 mins on average.

Appendix C
The code for data processing and evaluation is provided here: https: //github.com/ayahia1/KDD-ML

Appendix D
The main reason for using EEGEyeNet in our study was its relatively large size. The table below presents the number of subjects, the number of sample data points, and the length of the recording time for each of the three experimental tasks in the EEGEyeNet dataset. The numbers below should prove the large size of this dataset.
Notes: First, in the table, experimental tasks are referred to as "paradigms. " Secondly, although the total number of subjects in the table is 486, some performed more than one experiment and, thus, referenced twice. The total number of unique subjects, when the dataset paper was first published, was 356. Finally, the recording time column contains the numbers rounded to the nearest hour.