
We agree that inordinately optimistic claims may led to misallocation of valuable resources; analogously, we believe that refuting claims should be grounded to solid scientific practices and it cannot be based on small-scale experiments (as theirs and our work was). Indeed, in the experimental sciences, it is well known that to demonstrate a null result, one needs a considerable amount of data, i.e., the burden of proof to show that something is absent is higher than the burden of proof to show that something is present, because it is always possible that a larger sample size would reveal differences , .

The above point highlights the importance of moving the discussion of the cognitive experiment design to a more appropriate community, able to evaluate properly and more critically design choices. Attempting to substantiate the validity of cognitive designs (which we have demonstrated being wrong) with hundreds of tables to give the impression of soundness, as [1]} did, is definitively incorrect. Indeed, cognitive neuroscientists would immediately spot the differences between the experimental designs in [1]} and [3]}, and the consequences on study outcomes of such differences.

The authors of [1]} mention “more modest but legitimate and important advances in the field”, but we did not see any of these advances discussed in [1]}.

The authors of [1]} also claim “when the community as a whole appears to suffer from widespread but problematic practices”, but again in [1]} we did not see any analysis of “community practices" (see Related Work section in [1]}) beyond  [3]}, , , , . Generalizing to the whole community based on the analysis of a few works and on small scale experiments, designed in opposition to standard cognitive neuroscience practices, is definitively wrong.

We believe that these works [3]}, , , ,  should be taken as they are, i.e., multimodal learning methods aimed at integrating 2D data with 1D noisy EEG time series. We are aware of the limitations of these experiments from a cognitive perspective. Decoding the human brain surely requires large interdisciplinary efforts. What we can do as machine learning researchers is to propose models and release code and data. And this is exactly what we have done in [3]}.

We recognize the importance of “appropriately caution the community” (it unclear what community they are referring to: PAMI or cognitive neuroscience?) to prevent any possible errors or biases, but at the same time "the community" must be aware of the counter-effects of papers like [1]} that appear to simply attack the credibility of researchers using mostly speculative and barely scientific claims. This will only serve to discourage researchers from releasing code and data or even from working in this new AI/cognitive neuroscience field.
A while ago,  showed that the Caltech-101 dataset  is extremely biased. If a paper like [1]} had been published, criticizing the Caltech-101 dataset, the machine learning community would have lost out on all the valuable classification methods that came from the Caltech-101 dataset. Instead, these methods have contributed significantly to the advancement of image classification research, despite the quality of the dataset.

