We know that bias exists, and that it affects our judgment and perception. This effect has to do with user's experience in life. That experience is taken care of by the brain, and if you counter a situation again, you have a predefined pattern of how to react. This can be changed over time, if other experiences is added. Wikipedia defines cognitive bias as: a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality. Implicit in the concept of a pattern of deviation is a standard of comparison with what is normatively expected. In other disciplines, measurement is a valuable tool. One would wish for a scale of deviation from what is normally expected. Is it possible to quantify cognitive bias?
Is it possible to quantify cognitive bias?
methodology;measurement;bias
If you come to this question from the bayesian tradition, then there is only one place where you can sneak in bias: your prior. This dovetails nicely with the wikipedia definition:a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.Since bayesian updating is considered to be 'rational', the only place to sneak in 'irrationality' (in quotes because we are using these terms very loosely) is in the prior before you were given any evidence/observations. So for a bayesian, measuring bias is measuring the prior.Measuring a priorConveniently,  Kalish et al. (2007) have a nice mechanism for measuring people's priors: have $n$ participants: $1, ... , n$ and give the first one some real input-output pairs on the relevant task to learn from. To train the $i + 1$th participant:take the $i$th participant, give them some inputs and ask them what they think the output should be,use the input-output pairs they generate to train the $i+1$th participant.Then, towards the end of the chain, the participants will start to converge towards their prior or inherent bias.ExampleA real example is of people's bias in functional relationships. The first person is given 25 $(x,y)$ pairs from some function $y = f(x)$. A person at stage $i$ is given 25 $(x,y)$ pairs from the person at stage $i - 1$. The person is then tested by being given an $x$ value and asked for a $y$, 25 times. The results of this are passed on to the person at trial $i + 1$ as the training data. The results:Show that people have a strong bias towards positive linear relationships. One could apply a similar procedure to other domains, but of course finding a good way to do so is a particular domain is an important research question for an experimental psychologist.ReferencesKalish, M., Griffiths, T. & Lewandowsky, S. (2007). Iterated learning: Intergenerational knowledge transmission reveals inductive biases. Psychonomic Bulletin & Review, 14, 288-294. doi: 10.3758/BF03194066