Our analysis suffers from several important caveats. First, it is well known that there are many more fairness metrics than the two we consider here, and that it’s statistically impossible for a single algorithm to achieve parity on all these metrics e.g., [1]}. Second, benchmark risk assessment instrument datasets often contain numerous measurement biases and errors which can make them ill-suited for making claims about real-world impact without carefully considering the the complicated socio-technical systems (in this case, the US criminal justice system) in which they are used [2]}. Finally, comparisons to proprietary algorithms will always be difficult to make precise without either significant reverse engineering or pressure from companies to lead to more transparent algorithms [3]}.
