Community norm violations can impair constructive communication and collaboration online. As a defense mechanism, community moderators often address such transgressions by temporarily blocking the perpetrator. Such actions, however, come with the cost of potentially alienating community members. Given this tradeoff, it is essential to understand to what extent, and in which situations, this common moderation practice is effective in reinforcing community rules. In this work, we introduce a computational framework for studying the future behavior of blocked users on Wikipedia. After their block expires, they can take several distinct paths: they can reform and adhere to the rules, but they can also recidivate, or straight-out abandon the community. We reveal that these trajectories are tied to factors rooted both in the characteristics of the blocked individual and in whether they perceived the block to be fair and justified. Based on these insights, we formulate a series of prediction tasks aiming to determine which of these paths a user is likely to take after being blocked for their first offense, and demonstrate the feasibility of these new tasks. Overall, this work builds towards a more nuanced approach to moderation by highlighting the tradeoffs that are in play.
The popularity, availability, and ubiquity of information and communication technologies create new opportunities for online harassment. The present study evaluates factors associated with young adult women's online harassment experiences through a multi-factor measure accounting for the frequency and severity of negative events. Findings from a survey of 659 undergraduate and graduate students highlight the relationship between harassment, well-being, and engagement in strategies to manage one's online identity. We further identify differences in harassment experiences across three popular social media platforms: Facebook, Twitter, and Instagram. We conclude by discussing this study's contribution to feminist theory and describing five potential design interventions derived from our data that may minimize these negative experiences, mitigate the psychological harm they cause, and provide women with more proactive ways to regain agency when using communication technologies. A wide range of behavior may be seen as destructive to online communities. Yet behavior that is ‘bad’ in one community may be celebrated in another. The work of community maintenance is therefore strongly contextual, involving complex choices due to differing norms, community cross-membership, and the need to invoke fairness. The experienced, “lived in” work of moderators; how they enact norms and make choices about social maintenance, remains poorly understood. Our study addresses this gap, using a negotiated order lens. We employed netnographic techniques, analyzing online interviews with moderators of sub-communities in Reddit, and records of critical incidents. We find that moderators are intuitive prosecutors who draw on a variety of logics to accomplish their work. Controlling bad behavior, articulating and enforcing norms is, therefore, a collective accomplishment through which moderators make choices, create a jurisprudence record, and reconcile nested community norms in the maintenance of social order.  The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users. User-generated content online is shaped by many factors, including endogenous elements such as platform affordances and norms, as well as exogenous elements, in particular significant events. These impact what users say, how they say it, and when they say it. In this paper, we focus on quantifying the impact of violent events on various types of hate speech, from offensive and derogatory to intimidation and explicit calls for violence. We anchor this study in a series of attacks involving Arabs and Muslims as perpetrators or victims, occurring in Western countries, that have been covered extensively by news media. These attacks have fueled intense policy debates around immigration in various fora, including online media, which have been marred by racist prejudice and hateful speech. The focus of our research is to model the effect of the attacks on the volume and type of hateful speech on two social media platforms, Twitter and Reddit. Among other findings, we observe that extremist violence tends to lead to an increase in online hate speech, particularly on messages directly advocating violence. Our research has implications for the way in which hate speech online is monitored and suggests ways in which it could be fought.  Users organize themselves into communities on web platforms. These communities can interact with one another, often leading to conflicts and toxic interactions. However, little is known about the mechanisms of interactions between communities and how they impact users. Here we study intercommunity interactions across 36,000 communities on Reddit, examining cases where users of one community are mobilized by negative sentiment to comment in another community. We show that such conflicts tend to be initiated by a handful of communities—less than 1 of communities start 74 of conflicts. While conflicts tend to be initiated by highly active community members, they are carried out by significantly less active members. We find that conflicts are marked by formation of echo chambers, where users primarily talk to other users from their own community. In the long-term, conflicts have adverse effects and reduce the overall activity of users in the targeted communities. Our analysis of user interactions also suggests strategies for mitigating the negative impact of conflicts—such as increasing the direct engagement between attackers and defenders. Further, we design classifiers to predict whether conflict will occur by creating an LSTM model which combines graph embeddings, user, community, and text features, and we also use these techniques to predict if a user will participate in a conflict. Altogether, this work presents a data-driven view of community interactions and conflict, and paves the way towards healthier online communities.  I conduct an experiment which examines the impact of group norm promotion and social sanctioning on racist online harassment. Racist online harassment de-mobilizes the minorities it targets, and the open, unopposed expression of racism in a public forum can legitimize racist viewpoints and prime ethnocentrism. I employ an intervention designed to reduce the use of anti-black racist slurs by white men on Twitter. I collect a sample of Twitter users who have harassed other users and use accounts I control (“bots”) to sanction the harassers. By varying the identity of the bots between in-group (white man) and out-group (black man) and by varying the number of Twitter followers each bot has, I find that subjects who were sanctioned by a high-follower white male significantly reduced their use of a racist slur. This paper extends findings from lab experiments to a naturalistic setting using an objective, behavioral outcome measure and a continuous 2-month data collection period. This represents an advance in the study of prejudiced behavior.    This exploratory work studies the effects of emerging app features on the cyberbullying practices in high school settings. These include the increasing prevalence of image video content, perceived ephemerality, anonymity, and hyperlocal communication. Based on qualitative analysis of focus groups and follow-up individual interviews with high school students, these features were found to influence the practice of cyberbullying, as well as creating negative socio-psychological effects. For example, visual data was found to be used in cyberbullying settings as evidence of contentious events, a repeated reminder, and caused a graphic impact on recipients. Similarly, perceived ephemerality of content was found to be associated with "broken expectations" with respect to the apps and severe bullying outcomes for those affected. Results shed light on an important technology-mediated social phenomenon of cyberbullying, improve understanding of app use (and abuse) by the teenage user population, and pave the way for future research on countering appcentric cyberbullying. Recent research on uninhibited behavior in computer‐mediated communication (CMC) systems have suggested that flaming is social‐context dependent and not a media characteristic of CMC. This study takes a closer look at the social context in which flaming occurs, which need not necessarily be developed online but, as well, can be the social, religious, and political background and affiliations of the participants. The study analyzed messages posted during 1 week to 4 Usenet social newsgroups that represent different national and cultural groups. The levels of flaming in these groups were found to be higher than any reported in other studies. The findings show that the frequency of flaming differed between the newsgroups, and differed within newsgroups according to the general topic under discussion, confirming that social context and not the medium is the primary determinant of online uninhibited behavior. © 1998 John Wiley & Sons, Inc. This article introduces and discusses bot-based collective blocklists (or blockbots) in Twitter, which have been developed by volunteers to combat harassment in the social networking site. Blockbots support the curation of a shared blocklist of accounts, where subscribers to a blockbot will not receive any notifications or messages from those on the blocklist. Blockbots support counterpublic communities, helping people moderate their own experiences of a site. This article provides an introduction and overview of blockbots and the issues that they raise about networked publics and platform governance, extending an intersecting literature on online harassment, platform governance, and the politics of algorithms. Such projects involve a far more reflective, intentional, transparent, collaborative, and decentralized way of using algorithmic systems to respond to issues like harassment. I argue that blockbots are not just technical solutions but social ones as well, a notable exception to common technologically determinist solutions that often push responsibility for issues like harassment to the individual user. Beyond the case of Twitter, blockbots call our attention to collective, bottom-up modes of computationally assisted moderation that can be deployed by counterpublic groups who want to participate in networked publics where hegemonic and exclusionary practices are increasingly prevalent.