We evaluate bias with respect to the <cite>WinoBias</cite> dataset (<cite>Zhao et al., 2018a</cite>) , a benchmark of paired male and female coreference resolution examples following the Winograd format (Hirst, 1981; Rahman and Ng, 2012; Peng et al., 2015) .