region:us
---
dataset_info:
  features:
  - name: subreddit
    dtype: string
  - name: created_utc
    dtype: int64
  - name: score
    dtype: int64
  - name: body
    dtype: string
  - name: predicted_language
    dtype: string
  - name: probability
    dtype: float64
  - name: year
    dtype: float64
  - name: day
    dtype: float64
  - name: month
    dtype: float64
  - name: time
    dtype: string
  - name: label_identity_attack
    dtype: float64
  - name: label_insult
    dtype: float64
  - name: label_obscene
    dtype: float64
  - name: label_severe_toxicity
    dtype: float64
  - name: label_threat
    dtype: float64
  - name: label_toxicity
    dtype: float64
  splits:
  - name: train
    num_bytes: 1852133546
    num_examples: 4476667
  download_size: 1024144293
  dataset_size: 1852133546
---
# Dataset Card for "Reddit_fi_2006_2022"


## Dataset Description

- **Point of Contact:** [RASMUS](https://www.linkedin.com/in/rasmustoivanen/)
- **Size of csv filee on disk files:** 1542.75 MB
- **Size of the generated parquet files:** 970 MB

### Dataset Summary

Reddit_fi is a filtered and post-processed corpus consisting of comments from [Reddit](https://reddit.com/).

Some words of caution at this stage however. Subreddits were not filtered as in ScandiReddit to filter out any specific subreddits that could have hate speech, toxicity, biased. Be careful when training language models with this data and curate you dataset properly.

All Reddit comments from January 2006 up until December 2022 were downloaded through [PushShift](https://files.pushshift.io/reddit/comments/), after which these were filtered based on the FastText language detection model by using confidence score of 70% was as a limit.

We also filter out shorter than 30 character messages based on body field.

This project was inspired by https://huggingface.co/datasets/alexandrainst/scandi-reddit creator https://www.saattrupdan.com/. Kudos to you!

### Filtering disclaimer. Toxicity and bias

The dataset is provided as is and high likely includes toxic, biased etch. material. You should carefully curate this dataset for your needs. To label toxic messages, we used Finnish toxicity classifier [TurkuNLP/bert-large-finnish-cased-toxicity](https://huggingface.co/TurkuNLP/bert-large-finnish-cased-toxicity) released by TurkuNLP. This dataset includes 6 different toxicity labels with their predicted scores for each message. You can use those labels and scores to filter out toxic messages. 

We evaluated subreddits with over 500 messages and decided to provide a list that based on our fast analysis should be filtered out:

[FinlandOnlyfans, 
Warframe, 
Finnishbitches, 
vitunluurangot, 
WTF, 
SaatananTeletapit, 
FinnishWhores, 
pics, 
iidapiiroinen123, 
okkamuretardi, 
FinnishGenderCritical, 
onlyfanssuomi, 
SuomiBannatut, 
jumalattaret, 
jumalattaret2, 
jumalattaretPro, 
HommaInAction, 
snappisensuroimaton]

### Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. 
You can also use this dataset for various data analysis things


### Languages

The dataset is available in Finnish

### Data Instances

An example from the dataset looks as follows.
```
{
   "subreddit": "arkisuomi",
   "created_utc": 1671152007,
   "score": 1,
   "body": "oatlyn iKaffe on maitoa parempaa kahvissa, en jois pelkästään kuitenkaan",
   "predicted_language": "__label__fi",
   "probability": 0.9783772230148317,
   "year": 2022.0,
   "day": 16.0,
   "month": 12.0,
   "time": "00:53:27",
   "label_identity_attack": 0.00018978118896484375,
   "label_insult": 0.00058746337890625,
   "label_obscene": 0.00142669677734375,
   "label_severe_toxicity": 6.723403930664062e-05,
   "label_threat": 0.0004100799560546875,
   "label_toxicity": 0.01025390625
}
```

### Data Fields

The data fields are the same among all splits.

- `subreddit`: `string`
- `created_utc: `int64`
- `score`: `int64`
- `body`: `string`
- `predicted_language`: `string` 
- `probability`: `float64` 
- `year`: `float64`
- `day`: `float64`
- `month`: `float64`
- `time`: `string`
- `label_identity_attack`: `float64`
- `label_insult`: `float64`
- `label_obscene`: `float64`
- `label_severe_toxicity`: `float64`
- `label_threat`: `float64`
- `label_toxicity`: `float64`


### Language Distribution
- fi:  4,476,667

### Top-5 Subreddit Distribution
- Suomi: 3 546 657
- snappijuorut: 469 592
- LakkoPostaukset: 64 198
- snappisensuroimaton: 6 248
- mina_irl: 2 828



## Dataset Creation

### Curation Rationale

The Finnish language does not have that many open source social media datasets. One notable dataset is Suomi24 but it has restricted access.

### Source Data

The raw Reddit data was collected through [PushShift](https://files.pushshift.io/reddit/comments/).


## Additional Information

### Dataset Curators

[Rasmus Toivanen](https://www.linkedin.com/in/rasmustoivanen/) 
curated this dataset.

### Licensing Information

The dataset is licensed under the [CC BY 4.0
license](https://creativecommons.org/licenses/by/4.0/).