In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance. Convolutional Neural Networks (CNNs) are the current state-of-the-art approach for image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2015; Huang et al., 2017) . The goal of image classification is to assign an image-level label to an image. Typically, it is assumed that an object (or concept) that correlates with the label is clearly visible and occupies a significant portion of the image Krizhevsky, 2009; Deng et al., 2009 ). Yet, in a variety of real-life applications, such as medical image or hyperspectral image analysis, only a small portion of the input correlates with the label, resulting in low signal-to-noise ratio. We define this input image signal-to-noise ratio as Object to Image (O2I) ratio. The O2I ratio range for three real-life datasets is depicted in Figure 1 . As can be seen, there exists a distribution shift between standard classification benchmarks and domain specific datasets. For instance, in the ImageNet dataset (Deng et al., 2009 ) objects fill at least 1% of the entire image, while in histopathology slices (Ehteshami Bejnordi et al., 2017) cancer cells can occupy as little as 10 −6 % of the whole image. Recent works have studied CNNs under different noise scenarios, either by performing random input-to-label experiments (Zhang et al., 2017; or by directly working with noisy annotations (Mahajan et al., 2018; Jiang et al., 2017; Han et al., 2018) . While, it has been shown that large amounts of label-corruption noise hinders the CNNs generalization (Zhang et al., 2017; , it has been further demonstrated that CNNs can mitigate this label-corruption noise by increasing the size of training data (Mahajan et al., 2018) , tuning the optimizer hyperparameters (Jastrzębski et al., 2017) or weighting input training samples (Jiang et al., 2017; Han et al., 2018) . However, all these works focus on input-to-label corruption and do not consider the case of noiseless input-to-label assignments with low and very low O2I ratios. In this paper, we build a novel testbed allowing us to specifically study the performance of CNNs when applied to tiny object classification and to investigate the interplay between input signal-to-noise ratio and model generalization. We create two synthetic datasets inspired by the children's puzzle book Where's Wally? (Handford, 1987) . The first dataset is derived from MNIST digits and allows us for two medical imaging datasets (CAME-LYON17 (Ehteshami Bejnordi et al., 2017) and MiniMIAS (Suckling, 1994) ) as well as one standard computer vision classification dataset (ImageNet (Deng et al., 2009) ). The ratio is defined as O2I = Although low input image signal-to-noise scenarios have been extensively studied in signal processing field (e.g. in tasks such as image reconstruction), less attention has been devoted to low signal-tonoise classification scenarios. Thus, in this paper we identified an unexplored machine learning problem, namely image classification in low and very low signal-to-noise ratios. In order to study such scenarios, we built two datasets that allowed us to perform controlled experiments by manipulating the input image signal-to-noise ratio and highlighted that CNNs struggle to show good generalization for low and very low signal-to-noise ratios even for a relatively elementary MNIST-based dataset. Finally, we ran a series of controlled experiments 9 that explore both a variety of CNNs' architectural choices and the importance of training data scale for the low and very low signal-to-noise classification. One of our main observation was that properly designed CNNs can be trained in low O2I regime without using any pixel-level annotations and generalize if we leverage enough training data; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the O2I ratio. Thus, with our paper (and the code release) we invite the community to work on data-efficient solutions to low and very low signal-to-noise classification. Our experimental study exhibits limitations: First, due to the lack of large scale datasets that allow for explicit control of the input signal-to-noise ratios, we were forced to use the synthetically built nMNIST dataset for most of our analysis. As a real life dataset, we used crops from the histopathology CAMELYON dataset; however, due to relatively a small number of unique lesions we were unable to scale the histopathology experiments to the extent as the nMNIST experiments, and, as result, some conclusions might be affected by the limited dataset size. Other large scale computer vision datasets like MS COCO (Lin et al., 2014 ) exhibit correlations of the object of interest with the image background. For MS COCO, the smallest O2I ratios are for the object category "sports ball" which on average occupies between 0.3% and 0.4% of an image and its presence tends to be correlated with the image background (e. g. presence of sports fields and players). However, future research could examine a setup in which negative images contain objects of the categories "person" and "baseball bat" and positive images also contain "sports ball". Second, all the tested models improve the generalization with larger dataset sizes; however, scaling datasets such as CAMELYON to tens of thousands of samples might be prohibitively expensive. Instead, further research should be devoted to developing computationally-scalable, data-efficient inductive biases that can handle very low signal-to-noise ratios with limited dataset sizes. Future work, could explore the knowledge of the low O2I ratio and therefore sparse signal as an inductive bias. Finally, we studied low signal-to-noise scenarios only for binary classification scenarios 10 ; further investigation should be devoted to multiclass problems. We hope that this study will stimulate the research in image classification for low signal-to-noise input scenarios.
We study low- and very-low-signal-to-noise classification scenarios, where objects that correlate with class label occupy tiny proportion of the entire image (e.g. medical or hyperspectral imaging).
{"src": "scitldr-AIC_train", "paper_id": "H1xTup4KPr"}