Less is more? How demographic sample weights can improve public opinion estimates based on Twitter data
An important limitation in previous studies of political behavior using Twitter data is the lack of information about the sociodemographic characteristics of individual users. This paper addresses this challenge by developing new machine learning methods that will allow researchers to estimate the age, gender, race, party affiliation, propensity to vote, and income of any Twitter user in the U.S. with high accuracy. The training dataset for these classifiers was obtained by matching a massive dataset of 1 billion geolocated Twitter messages with voting registration records and estimates of home values across 15 different states, resulting in a sample of nearly 250,000 Twitter users whose sociodemographic traits are known. I illustrate the value of these new methods with two applications. First, I explore how attention to different candidates in the 2016 presidential primary election varies across demographic groups within a panel of randomly selected Twitter users. I argue that these covariates can be used to adjust estimates of sentiment towards political actors based on Twitter data, and provide a proof of concept using presidential approval. Second, I examine whether social media can reduce inequalities in potential exposure to political messages. In particular, I show that retweets (a proxy for inadvertent exposure) have a large equalizing effect in access to information. Twitter data is widely acknowledged to hold great promise for the study of social and political behavior (Mejova, Weber, and Macy 2015; Jungherr 2015) In a context of plummeting survey response rates, tweets represent unfiltered expressions of political opinions, which have been found to be correlated with offline opinions and behavior (O'Connor et al. 2010; DiGrazia et al. 2013; Vaccari et al. 2013) More generally, Twitter data also allows researchers to easily and unobtrusively observe social interactions in real time, and to measure consumption of political information with a level of granularity that could only be achieved in the past at great cost. Despite the great promise of this source of data, an important challenge that remains to be overcome is the lack of sociodemographic information about Twitter users. Unlike Working paper. This version: February 24, 2015. The author gratefully acknowledges financial support from the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation. other social media platforms, Twitter does not require its users to provide basic information about themselves, such as gender or age. As a result, researchers interested in working with Twitter data cannot construct survey weights to recover the representativeness of their samples in the same way that survey researchers combine probability sampling with post stratification weights to reduce sampling selection bias (Schober et al. 2016) Beyond this methodological concern, the availability of individual level covariates would expand the range of questions that can be studied with Twitter data. For example, if we were interested in measuring support for political candidates in a primary election, it would allow us to subset only those that are affiliated with that party. Being able to identify income, gender, and race would enable studies of social segregation in online settings. We could also study social inequalities in political behavior at a much more granular level if we were able to observe the individual traits of Twitter users. The contribution of this paper is to develop new methods to estimate the age, gender, race, party affiliation, propensity to vote, and income of any Twitter user in the U.S. This work improves upon previous studies on latent attribute inference based on Twitter data (Al Zamal, Liu, and Ruths 2012; Chen et al. 2015; Mislove et al. 2011; Pennacchiotti and Popescu 2011; Rao et al. 2010) in two different ways. First, by relying on a ground truth dataset at least two orders of magnitude larger than those used in previous studies, this method achieves significantly better performance in this task. Second, and most importantly, the features used to predict Twitter users' latent traits can be measured using no more than 5 API calls per user, which makes it easy to scale to large datasets. This paper also provides two applications of these methods to questions of substantive interest. First, I examine how attention to different candidates in the 2016 presidential primary election varies across sociodemographic groups using a panel of 200,000 users, randomly selected. This panel design overcomes some of the difficulties inherent to selfselection bias and, in combination with the sampling weights that can now be computed using the latent traits estimated with the method introduced here, could potentially allow researchers to recover the representativeness of estimates based on Twitter data. Second, I examine how exposure to political information on Twitter varies across sociodemographic groups. Merging data about who retweeted particular political messages with the lists of who each individual in this panel of 200,000 users follows, I am able to quantify direct (via following) and indirect exposure (via retweets) This analysis shows that even if direct exposure is highly unequal across social groups, the differences are significantly reduced once inadvertent exposure is considered. This result highlights the potential of social media to reduce inequalities in access to political information. Background and Related Work Previous studies have approached the problem of estimating the sociodemographic characteristics of Twitter users using one of two approaches. One option is to apply supervised machine learning methods to a training dataset of users whose traits are known, usually by human coding. For example, Cheng (2015) used Amazon Mechanical Turk to label the ethnicity, gender, and age of 2,000 users, and then ran different classifiers using features from users' tweets, their neighbors, and their profile pictures. Pennacchiotti (2011) employed a similar method with a sample of 6,000 users who stated their ethnicity in their descriptions, and 10,000 users who added themselves to a public directory of Democrats and Republicans on Twitter. Al Zamal (2012) used the same source for political orientation, and 400 tweets from users announcing their own birthday to identify age. They considered a similar set of features both information about users' tweets and about their friends and followers. A second approach is to rely on indirect methods, such as extracting Twitter users' names, and comparing those with existing datasets with distributions of gender by first name and of ethnicity by last name to compute a probability of being male or female, and Caucasian, AfricanAmerican, etc. (Mislove et al. 2011) A different type of indirect approach was used by Culotta et al (2015) using website audience data, they show that followers of the Twitter accounts of these websites have a similar demographic composition. Within this category we would also find unsupervised methods that detect latent communities based on interactions on Twitter, building upon the assumption that behavior is homophilic (Conover et al. 2012; Barbera 2015a) Both approaches have limitations that make it difficult to scale these methods to large samples of users. Indirect methods do not perform well when sociodemographic traits that are not heavily correlated with behavior, and name based methods cannot be applied when new names are not included in the lists of names tagged by gender, which limits their applicability. For example, nearly 110,00 of 250,000 (44% randomly selected U.S. Twitter users (see Applications section) did not report a first name that appears in the Social Security Administration baby names dataset (Blevins and Mullen 2015) Supervised methods do not suffer from this problem but, because of their use of small, self selected samples, they require collecting "costly" features in order to achieve high accuracy. Measuring features such as the text of a user's neighbors (her followers and those that she follows) is very time consuming because it requires hundreds of API calls, making this method impractical for any sample larger than a few thousand users. The aim of this paper is to develop a new approach that overcomes these limitations and allows any researcher to (1) estimate the age, gender, race/ethnicity, income, propensity to vote, and party affiliation of (2) any Twitter user, and (3) with fewer than 5 API calls per user. Method Even if the sociodemographic characteristics of Twitter users cannot be directly observed, there are at least two different types of information that researchers could use to infer them. Text of users' tweets. A range of previous studies have shown significant differences in language use between men and women (Newman et al. 2008) liberals and conservatives (Sylwester and Purver 2015) individuals of different age (Schwartz et al. 2013) and race groups (Florini 2013) Language use indicates not only differences in personality or opinions, but also in interests and activities, which may also be correlated with users' sociodemographic characteristics. Text in microblogging platform such as Twitter often includes emoji characters ideograms that include facial expressions, objects, flags, among others, and which often can convey more complex ideas than single words. To test whether language use predicts users' latent traits, I will estimate two models, of increasing complexity: first, a logistic classifier with Elastic Net regularization (Zou and Hastie 2005) using only emoji characters as features (bagof emoji) second, a logistic classifier with Elastic Net regularization and Stochastic Gradient Descent (SGD) learning using word counts as features (bag of words, BOW) and applying a TF IDF transformation. To reduce the size of the feature matrix, I will only consider emoji and words used by more than 1% and