I've been trying to find an answer to this question for a long time. What are the relationships/differences between Bias, Variance and  Residuals?I think I do understand Bias, Variance and Residuals as separate concepts. Correct me if I'm wrong - Bias is the difference between the average expected results from different runs of the model and the true values from data. Variance is the variability in the expected results (predictions) of a given data point between different runs of the model. Residual is the difference between the expected results from a model and the true values from data. y - y^ Residual seems somewhat similar to Bias. But is that my misunderstanding?Please explain to me in plain words and also simple equations the relationships or differences between these three.Thanks in advance.
What are the relationships/differences between Bias, Variance and Residuals
machine learning;predictive modeling;statistics
I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.Bias, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.Variance, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the stability of a given algorithm/parameter set when exposed to different training and testing sets.(N.B. This can be confusing because there is a specific definition of variance used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)Residuals are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.