
Section  describes the EEG data used in this work and the classification performance obtained by state of the art deep models. In particular:

Section REF  provides the data collection procedure and pre-processing details of the three datasets used in this work: (a) Block-Design Visually-Evoked (BDVE) dataset — the dataset based on a block design presented in [1]}; (b) Block-Design Blanks (BDB) dataset — an additional dataset containing EEG recordings of the same subjects who underwent the experiment in [1]}, while they were shown a blank screen for 10 seconds between each of the 40 image classes, as per the original acquisition protocol; (c) Rapid-Design Visually-Evoked (RDVE) dataset — EEG data collected during our replication of the rapid-design experiment presented in [3]}.

Section REF  re-analyzes the results of the models proposed in [1]}, , [5]} as well as of state-of-the-art deep models on all three above datasets. The summary of the findings is as follows:

Classification accuracy in the block-design BDVE dataset, when the EEG data is correctly filtered, reaches approximately 50%, which is significant over 40 classes. This is lower than the \(\sim \) 83% reported in [1]}, but that analysis was biased by slow EEG activity.
Thus, we agree that classifying raw EEG data, without any pre-processing, may inflate classifier performance, as stated in [3]} and already discussed in [5]}.

Classification accuracy on the BDB dataset is only slightly above chance, suggesting a negligible temporal correlation in [1]}'s data.

Classification accuracy on the RDVE dataset when using block-level labels, as done in [3]}, is also slightly above chance. This stands in stark contrast to the robust classification performance reported in [3]}.
Analogously, performance is at chance when using the correct labels.
We argue that the high accuracy obtained in [3]} on the rapid-design experiment with block-level labels is the direct result of a) an incorrect experimental and recording protocol, which increases the experiment duration, removes breaks, and undermines the subjects' attention level, and b) the use per-subject classification.


In section REF , we attempt to understand why the pattern of classification accuracy that we found is so different from those in [3]}. In particular, we are only able to reproduce
[3]}'s results with our data when we intentionally contaminate it with temporal correlation and magnify its impact on classification results through per-subject analysis as in [3]}. This suggests that data in [1]} and our new data show a significantly reduced temporal correlation compared to data in [3]}. Again, the reason for the large bias in data in [3]} is due to an incorrect experimental design that does not follow EEG/ERP design guidelines and to problematic procedures such as per-subject analysis.


In Section , we comment in detail on the experimental design utilized in [3]} and the differences, with related consequences on results, from the designs in [1]}. We also report what cognitive neuroscience suggests would be a more appropriate design.

In section , we carefully analyze [3]} and expose the plethora of false claims, misinterpretations of our previous work and of basic machine learning concepts, and unverified opinions which appear to have the sole purpose of hurting the credibility of the authors of [1]}, , , , .

