<story><title>Amazon scraps secret AI recruiting tool that showed bias against women</title><url>https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G</url></story><parent_chain><item><author>kareemsabri</author><text>This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that female resumes don’t match, as closely, the resumes of successful past candidates. Perhaps I’m ignorant about AI, but I don’t see why the number of candidates of each gender shouldn’t increase the strength of the signal. “Aggressiveness” in the resume may be correlated but not causal. If the AI was fed the heights of the candidates, it might reject women for being too short, but that would not indicate height is a criteria of Amazon recruiters hiring.</text></item><item><author>fuscy</author><text>The eye opening thing here is not that the AI failed, but why it failed.&lt;p&gt;At start the AI is like a baby, it doesn&amp;#x27;t know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion.&lt;p&gt;The AI becoming biased tells that the &amp;quot;teacher&amp;quot; was biased also. So actually Amazon&amp;#x27;s recruiting process seems to be a mess with the technical skills on the resume amounting to zilch, gender and the aggressiveness of the resume&amp;#x27;s language being the most important (because that&amp;#x27;s how the human recruiters actually hired people when someone put a resume).&lt;p&gt;The number of women and men in the data set shouldn&amp;#x27;t matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). What matters is the rejection rate which it learned from the data.. The hiring process is inherently biased against women.&lt;p&gt;Technically one could say that the AI was successful because it emulated the current Amazon hiring status.</text></item></parent_chain><comment><author>danShumway</author><text>This is a subtle point but worth stating -- AI does not mirror or copy human reasoning.&lt;p&gt;AI is designed to get the same &lt;i&gt;results&lt;/i&gt; as a human. How it gets to those results is often very, very different. I&amp;#x27;m having trouble finding it, but there was an article a while back trying to do focus tracking between humans and computers for image recognition. What they found was that even when computers were relatively consistent with humans in results, they often focused on different parts of the image and relied on different correlations.&lt;p&gt;That doesn&amp;#x27;t mean that Amazon isn&amp;#x27;t biased. I mean, let&amp;#x27;s be honest, it probably is; there&amp;#x27;s no way a company this large is going to be able to perfectly filter or train every employee and on average tech bias trends against women. BUT, the point is that even if Amazon were to completely eliminate bias from every single hiring decision it used in its training data, an AI still might introduce a racial or gendered bias on its own if the data were skewed or had an unseen correlation that researchers didn&amp;#x27;t intend.</text></comment>
<story><title>Amazon scraps secret AI recruiting tool that showed bias against women</title><url>https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G</url></story><parent_chain><item><author>kareemsabri</author><text>This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that female resumes don’t match, as closely, the resumes of successful past candidates. Perhaps I’m ignorant about AI, but I don’t see why the number of candidates of each gender shouldn’t increase the strength of the signal. “Aggressiveness” in the resume may be correlated but not causal. If the AI was fed the heights of the candidates, it might reject women for being too short, but that would not indicate height is a criteria of Amazon recruiters hiring.</text></item><item><author>fuscy</author><text>The eye opening thing here is not that the AI failed, but why it failed.&lt;p&gt;At start the AI is like a baby, it doesn&amp;#x27;t know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion.&lt;p&gt;The AI becoming biased tells that the &amp;quot;teacher&amp;quot; was biased also. So actually Amazon&amp;#x27;s recruiting process seems to be a mess with the technical skills on the resume amounting to zilch, gender and the aggressiveness of the resume&amp;#x27;s language being the most important (because that&amp;#x27;s how the human recruiters actually hired people when someone put a resume).&lt;p&gt;The number of women and men in the data set shouldn&amp;#x27;t matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). What matters is the rejection rate which it learned from the data.. The hiring process is inherently biased against women.&lt;p&gt;Technically one could say that the AI was successful because it emulated the current Amazon hiring status.</text></item></parent_chain><comment><author>BurningFrog</author><text>Control question for if you&amp;#x27;re making a certain intellectual mistake.&lt;p&gt;The data set will also have skewed heavily against people named &amp;quot;David&amp;quot;. Probably only ~1% of the successful applicants.&lt;p&gt;Would you also expect the machine to be biased against candidates named David?</text></comment>