Q:

Feedback on The Unfriendly Robot

Kevin and I just posted to the blog about automatically flagging comments on Stack Overflow with The Unfriendly Robot. We also talked about the robot recently here. 
After you give it a read we're interested in your thoughts/feedback and would like to answer any questions you might have. Thanks! 
EDIT
Some questions are difficult to answer in the comments so I'm going to reproduce them here and answer them.

From jcsahnwaldt says GoFundMonica

From 2009 until mid-2018, comments could be flagged as "offensive". Happened for ~0.1% of all comments. Now we can flag comments as "unfriendly or unkind". When was the new flag introduced? (I guess in mid-2018?) How many comments are flagged as "unfriendly or unkind"? In your study with moderators and other users, the median person classified ~3.5% as "unfriendly". What was the average "unfriendly" flag percentage in that study? Did you try to measure how much of the difference between ~0.1% and ~3.5% is due to "unfriendly" being a broader criterion than "offensive"?

I agree with you that the definitions of offensive and unfriendly are different and that (by truthiness of gut) the frequency of unfriendliness should be higher than the frequency of offensiveness. We introduced the "unfriendly or unkind" flag in August 2018 in conjunction with the Code of Conduct change, and we kept the "offensive" flag around. What happened to the percentage of posts flagged before and after the change?

This a monthly box plot of the percentage of posts flagged each day, by flag type. In red is the offensive flag which corresponds to the box plot in the blog post. In blue is the unfriendly-or-unkind flag. When the unfriendly flag was released we see offensive flag usage drop way way down. There was copy change here too so "offensive" isn't what users see anymore, they see this...

Regardless, I think the graph shows the right thing, that what's truly offensive is way lower than what we used to think (if just going by the name of the flag). Unfriendly-or-unkind flag usage jumps up for a few months, and then settles back down to where the old offensive flag was. This is support for the hypothesis that the Stack Overflow system has an underflagging problem. Even when we expanded the definition of what you should be flagged, overall throughput wasn't meaningfully different. I don't think this result gives us any ability to answer "How much of the difference is from the broader definition?", though. 

A:

Has culture been considered in any way as part of this research? In sociolinguistics, offensiveness is not an inherent property of a comment that can be calculated from analysis of vocabulary and grammar, but something that must be understood in the cultural context in which it was uttered.
As an example, in some Australian Aboriginal cultures, referring to dead relatives by name is per se considered highly offensive. If the majority of Stack Overflow participants were from such cultures, it is highly likely that "refers to dead relative by name" would be a valid reason to flag or even a specific named flag or suspension reason ("This account is temporarily suspended for referring to the dead by name"). It isn't, so what gives? Do we want to stick a finger in visitors' faces, telling them,

Declined - We know your deceased father invented that famous algorithm, but we aren't going to stop people from calling it the Roger Smith Machine! We don't follow that culture here and don't recognize its rules on offensiveness! Stop flagging!

Historically, I would have understood an adjudication related to culture-bound rules to revolve around intent or knowledge - e.g. when Mary posted a comment suggesting "Did you try the Roger Smith Machine?", did she know that his son found seeing his father's name online highly offensive and is there any indication that she specifically wanted him to suffer additional distress? The article itself mentions that "circumstance or intent" no longer matters.

Did you omit any consideration of culture?
Did you assume a general "Stack Overflow Community Culture" based on your understanding of what constitutes current consensus on community norms here?
Did you use your own culture to frame your understanding of what offensive content is and is not?

As another example, cultures have a great deal of variance in how appropriate it is to point out an error made by someone else. In some cultures (E.g. Japanese), one must make only vague hints lest one be accused of insensitivity (e.g. "when one programs, one might look out for memory leaks...."), while in others (e.g. Israeli or Scottish), one may generally point out the error explicitly and then launch into a tirade of "friendly" insults ("you forgot to unallocate that pointer, numbskull"). Which culture prevails in a dispute as to whether a comment is offensive?
Your mention of how passersby might interpret an old comment makes me think about culture in a different way. Previously, I understood the intersection of culture and offensive comments on Stack Overflow to relate to the participants themselves - that is, a comment was considered offensive if and only if it was offensive according to the culture of the commentator and/or the intended recipient of the comment. If we allow anyone from any culture to interpret for themselves whether to interpret any comment as offensive, I fear that could lead to a lack of standards. Maybe you and I communicate just fine, but then someone comes along a year later who comes from a culture in which failing to invoke a deity before every comment is considered super-highly offensive, they flag both of us, and boom, we are banned for failing to consider this third party.

Can we even be sensitive to all cultures at the same time?
Can we be the Culture Police for everyone in the world?
Are we to standardize on a single "site culture" with corresponding site-specific social mores and to require everyone to conform to such rules as a condition of participation?

The only real solution I see is to have a single site standard, but then that feeds right back in to the idea that Stack Overflow might just have to be culturally insensitive to some people. Can we live with that?

A:

Interesting read. Detecting unfriendliness is a form of sentiment analysis, and my employer has been doing something similar for the last 20 years so I recognize quite a few of the challenges.
We fully agree with the notion that you can't just take humans out of the loop. But AI is very good at prioritizing work for humans. You can train neural networks to also produce confidence scores. These shouldn't be interpreted as "percentages correct", but only to sort the flags for the human moderators to judge. Start with the ones that the network classifies as unfriendly with high confidence, and work down the list. 
As you gain experience with the AI, you may find that the highest confidence scores are certain hits, so you can in fact deprioritize those. Instead, pending human verification you label them as unfriendly. 
One area we're actively looking into is the ranking of sentiment. Your current classifier is a binary classifier, because you train it on two classes. Something is either friendly or unfriendly. But your human baseline can be more nuanced than that, and your AI can learn that.
As you might notice, unlike Jason Punyon and Dave Robinson, I have no hesitation using the term AI. To me that is a broad category. I agree that this is also Machine Learning, and also Pattern Recognition. Those are overlapping subfields of AI.

A:

feature-request
It would be useful to have a FAQ or help page on how to write comments in a more respectful and constructive way.
I know this is briefly gone over in the CoC, but it would be nice to have a list of what's seen as "unfriendly" and what's not, and some examples. Perhaps there might be a way to word constructive comments in a way that's not seen as rude.

