The Pushshift Reddit Dataset
Jason Baumgartner1,*, Savvas Zannettou2,,, Brian Keegan3, Megan Squire4, Jeremy Blackburn5,,
1Pushs
The Pushshift Reddit Dataset
Social media data has become crucial to the advancement of scientific
understanding. However, even though it has become ubiquitous, just collecting
large-scale social media data involves a high degree of engineering skill set
and computational resources. In fact, research is often times gated by data
engineering problems that must be overcome before analysis can proceed. This
has resulted recognition of datasets as meaningful research contributions in
and of themselves. Reddit, the so called "front page of the Internet," in
particular has been the subject of numerous scientific studies. Although Reddit
is relatively open to data acquisition compared to social media platforms like
Facebook and Twitter, the technical barriers to acquisition still remain. Thus,
Reddit's millions of subreddits, hundreds of millions of users, and hundreds of
billions of comments are at the same time relatively accessible, but time
consuming to collect and analyze systematically. In this paper, we present the
Pushshift Reddit dataset. Pushshift is a social media data collection,
analysis, and archiving platform that since 2015 has collected Reddit data and
made it available to researchers. Pushshift's Reddit dataset is updated in
real-time, and includes historical data back to Reddit's inception. In addition
to monthly dumps, Pushshift provides computational tools to aid in searching,
aggregating, and performing exploratory analysis on the entirety of the
dataset. The Pushshift Reddit dataset makes it possible for social media
researchers to reduce time spent in the data collection, cleaning, and storage
phases of their projects.
Jason Baumgartner Savvas Zannettou Brian Keegan Megan Squire Jeremy Blackburn