I would like to build a dataset to classify some elements. Of course a big thing for me is to make sure that my test dataset will reflect the future inference dataset. The most obvious way to do this is to randomly sample the elements to label. The second way that I would like to discuss here could be to create a classifier, throw some data at it, and iteratively correct the mistakes of the classifier by relabeling the missclassified instances. I think both are valid, but I was wondering if one method could converge faster than the other?Do you know any other paradigm to create a valid training dataset ? 
Gathering data: random sampling vs iterative correction of classifier
classification;dataset;multilabel classification