Q:

K fold cross validation sample mean question

When doing k fold cross validation, the data is randomly separated. Does it make sense to throw out a data set if it is very uncharacteristic of the general data? 
If I don't have a very large data set, any random sample from that data set might have a very different mean and standard deviation from the complete data set. 
Should I verify that the standard deviation and mean from random samples taken for K fold cross validation match that of the intended real world data? 
Edit:
To clairfy, if you have 100 samples with standard deviation $\sigma_0$ and mean $\mu_0$, and you do $K10$ fold cross validation, then you will have $10$ randomly generated test sets, each with their own $\sigma_i$ and $\mu_i$. 
If $\sigma_i$ and $\mu_i$ are radically different from $\sigma_0$ and mean $\mu_0$, then does it make sense to throw out that particular test set?

A:

Let me make this even more general. The core of your question seems to be if there are any problems (or benefits) with removing "atypical" cases from the validation set. From what I understood, you judge the cases to be "atypical" if they differ from the whole dataset that you have.
If you removed the cases, then this would obviously improve your validation scores, but only because you were cheating and not testing on the hard cases! There is no reason whatsoever why you shouldn't test your model on valid data. The whole idea about cross-validation is to use the validation set to check performance of your model on dataset that differs from the one that you used for training. Even more, we often deliberately choose the validation set to differ from training set (e.g. with time-series data, we split it to use the "past" for training and "future" for testing). The reason for cross-validation is to estimate the potential performance on the unseen data, if you are sure that the data you haven't seen is the same as the one you used for training, then there is no reason for cross-validation.
The different story would be if you had incorrect data (impossible or corrupted values, wrong measurements etc.) in your validation set, then you would remove them, but only then. 

