To further illustrate our point about the inherent challenges of open-ended inputs and domains, and tie it to the possibility of harm from language models, we consider a problem domain that language models are typically not (or not yet) deployed on, but which is associated with societal concerns: recidivism prediction. Some have pointed out that even beyond specific concerns about fairness, recidivism prediction simply should not be a task for machine learning [1]}. We agree and we do not believe that language models should be used for recidivism prediction. However, because the application is so inherently questionable, it provides a compelling example of how harmful abilities can emerge quietly in unexpected ways as generative models scale. It is likely that such abrupt emergence also occurs in many other contexts where the harms are more subtle. We study a case where the problems are flagrant in order to clearly demonstrate our thesis.
