arxiv:2304.02134 region:us
---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/datasets-cards
{}
---

# Dataset Card for Participation and Division of Labor in User-driven Audits
The project website for the research associated with this dataset is: https://userdrivenaudits.github.io/ 

## Dataset Description
This dataset is the complete dataset we used for our research study, "Participation and Division of Labor in User-driven Audits."

- **Homepage:** https://userdrivenaudits.github.io/ 
- **Paper (preprint):** https://arxiv.org/pdf/2304.02134.pdf 
- **Point of Contact:** 
Please contact Sara Kingsley for additional information or if you have questions about this data-set. Sara can be reached at skingsle[at]cs.cmu.edu

### Dataset Summary
This dataset is the complete dataset we used to analyze in our CHI'23 paper, `Participation and Division of Labor in User-driven Audits.` 

## Dataset Structure
The dataset is provided in an excel file. 

### Data Fields

The dataset contains the following features/column headers:
- created_at: date/timestamp for when a tweet was published
- text: this is the text of the tweet
- hashed conversation id: this is an anonymized identifier that represents the conversation to which the tweet belongs. Note, we hashed the original conversation ID in hopes to help protect user privacy
- hashed_author_id: this is an anonymized identifier that represents an unique identifier for the author of the tweet. Note, we hashed the original author ID in hopes to protect user privacy
- top_producer: this column identifies if the user was a top producer of content for an audit. 1 = top producer; 0 = not top producer
- top_broadcaster: this column identifies if the user was a top broadcaster of content for an audit. 1 = top broadcaster; 0 = not top producer  
- case: this column identifies which user-driven audit the tweet was associated with. AC = Apple Card; TC = Twitter Cropping; INR = ImageNet Roulette; PAI = PortraitAI
- DOL_Label_prediction: this is the label for the user role the tweet played in the audit, this label was automatically inferred using our SVM Division of Labor classifier
- DOL_Label_prediction_amplification: this column identifies whether the tweet was classified as one where the user's tweet played the role of 'amplification'
- DOL_Label_prediction_booster: this column identifies whether the tweet was classified as one where the user's tweet played the role of 'escalation'
- DOL_Label_prediction_contextualization: this column identifies whether the tweet was classified as one where the user's tweet played the role of 'contextualization'
- DOL_Label_prediction_data_collection: this column identifies whether the tweet was classified as one where the user's tweet played the role of 'evidence collection'
- DOL_Label_prediction_data_hypothesizing: this column identifies whether the tweet was classified as one where the user's tweet played the role of 'hypothesizing'
- DOL_Label_prediction_irrelevant: this column identifies whether the tweet was marked irrelevant by our classifier.

All the remaining column data are identical to the column data that can be retrieved (during the time of our study at least) from the Twitter API. Please refer to the Twitter API documentation for reference/information about these column data. Please contact Sara Kingsley if this information becomes unavailable at some point.


### Data Splits

We split this data 70-30 into train/test sets to train our classifiers. 

## Dataset Creation
We created this dataset starting in September/October 2020. 

### Curation Rationale

Please see our CHI'23 paper's methodology section for more information on our rationale for creating the dataset. The preprint is available here: https://arxiv.org/pdf/2304.02134.pdf

### Source Data
Twitter API

#### Initial Data Collection and Normalization

Please see our CHI'23 paper's methodology section for more information on our rationale for creating the dataset. The preprint is available here: https://arxiv.org/pdf/2304.02134.pdf


#### Who are the source language producers?
Please see our CHI'23 paper's methodology section for more information on our rationale for creating the dataset. The preprint is available here: https://arxiv.org/pdf/2304.02134.pdf


### Annotations
Please see our CHI'23 paper's methodology section for more information on our rationale for creating the dataset. The preprint is available here: https://arxiv.org/pdf/2304.02134.pdf


#### Annotation process
Please see our CHI'23 paper's methodology section for more information on our rationale for creating the dataset. The preprint is available here: https://arxiv.org/pdf/2304.02134.pdf


#### Who are the annotators?
The authors of this paper: https://arxiv.org/pdf/2304.02134.pdf


### Personal and Sensitive Information
We hashed the original identifiers in the dataset that could allow people to retrieve information about the users who published the tweets in the dataset.

Warning: it is possible, if a tweet is still published publicly on twitter, to search for the original tweet using the text of the tweet. We ask and encourage users of our dataset not to do this.

## Considerations for Using the Data
People may use this dataset to replicate the work we did in this paper: https://arxiv.org/pdf/2304.02134.pdf

### Social Impact of Dataset
Please see: https://arxiv.org/pdf/2304.02134.pdf

### Discussion of Biases
Please see: https://arxiv.org/pdf/2304.02134.pdf

### Other Known Limitations
Please see this information in our paper: https://arxiv.org/pdf/2304.02134.pdf

## Additional Information

### Dataset Curators
Please see: https://arxiv.org/pdf/2304.02134.pdf 

### Citation Information
Rena Li, Sara Kingsley, Chelsea Fan, Proteeti Sinha, Nora Wai, Jaimie Lee, Hong Shen, Motahhare Eslami, and Jason Hong. 2023. Participation and Division of Labor in User-Driven Algorithm Audits: How Do Everyday
Users Work Together to Surface Algorithmic Harms? In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/3544548.3582074

Preprint is available, here: https://arxiv.org/pdf/2304.02134.pdf

### Contributions
Rena Li,
Sara Kingsley,
Chelsea Fan,
Proteeti Sinha,
Nora Wai,
Jaimie Lee,
Hong Shen,
Motahhare Eslami,
Jason Hong