<story><title>Are popular toxicity models simply profanity detectors?</title><url>https://www.surgehq.ai/blog/are-popular-toxicity-models-simply-profanity-detectors</url></story><parent_chain><item><author>echen</author><text>One of the problems with real world machine learning is that engineers often treat models as pure black boxes to be optimized, ignoring the datasets behind them. I&amp;#x27;ve often worked with ML engineers who can&amp;#x27;t give you any examples of false positives they want their models to fix!&lt;p&gt;Perhaps this is okay when your datasets are high-quality and representative of the real world, but they&amp;#x27;re usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like &amp;quot;this is fucking awesome!&amp;quot; as toxic, even though they&amp;#x27;re actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity.&lt;p&gt;(So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!)&lt;p&gt;Many of the new, massive scale language models use the Perspective API to measure their safety. But we&amp;#x27;ve noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.</text></item></parent_chain><comment><author>DoItToMe81</author><text>Even if totally &amp;#x27;accurate&amp;#x27; to the dataset, there&amp;#x27;s the issue that a lot of &amp;#x27;toxicity&amp;#x27; and offence is completely culturally based. Take &amp;#x27;cunt&amp;#x27; for example. Used in casual, informal conversation here, but deemed incredibly offensive by some Americans. Or, more broadly, &amp;#x27;thumbs up&amp;#x27; could mean &amp;quot;Good job&amp;quot; or &amp;quot;Stick it up your arse&amp;quot;.</text></comment>
<story><title>Are popular toxicity models simply profanity detectors?</title><url>https://www.surgehq.ai/blog/are-popular-toxicity-models-simply-profanity-detectors</url></story><parent_chain><item><author>echen</author><text>One of the problems with real world machine learning is that engineers often treat models as pure black boxes to be optimized, ignoring the datasets behind them. I&amp;#x27;ve often worked with ML engineers who can&amp;#x27;t give you any examples of false positives they want their models to fix!&lt;p&gt;Perhaps this is okay when your datasets are high-quality and representative of the real world, but they&amp;#x27;re usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like &amp;quot;this is fucking awesome!&amp;quot; as toxic, even though they&amp;#x27;re actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity.&lt;p&gt;(So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!)&lt;p&gt;Many of the new, massive scale language models use the Perspective API to measure their safety. But we&amp;#x27;ve noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.</text></item></parent_chain><comment><author>tracker1</author><text>The false negative example you have us exactly the argument I&amp;#x27;ve used to fight against language filters and sensors in smaller sites.&lt;p&gt;In the end you can be profane and very positive. You can also have strike language while being incredibly vile and negative.</text></comment>