I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough. So is there a rule of thumb on how many data of dimension DIM should one collect for training a NN-Model ? For example, it depends on the number of features or the kind of NN-Model or something else ?Any help will be appreciated.
Data set size versus data dimension, is there a rule of thumb?
machine learning;neural network;data mining;dataset;deep learning
In this video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work.As a general rule of thumb, size of dataset should be at-least about 10x it's dimension and should be independent of the model used.Also, this link has summaries from some of the relevant papers, viz.For a finite sized data with little or no a priori information, the ratio of the sample size to dimensionality must be as large as possible to suppress optimistically biased evaluations of the performance of the classifier.This says, the ratio of size of dataset (sample) to dimension should be as large as possible to reduce classifier bias towards a particular class.The ratio of the sample size to dimensionality should vary inversely  proportional to the amount of available knowledge about the class  conditional densities.In a classifier setting, the more knowledge we have for each class' probability density, lesser can be the sample-size to dimension ratio. In simpler terms, we can say we should be including as much data as possible and if we are not able to do that, include as much information as possible in the small dataset itself, because for any model to work we need to feed it with high variance dataset.