task_categories:text-classification task_categories:conversational task_categories:text-generation task_categories:token-classification annotations_creators:crowdsourced language_creators:found size_categories:1M<n<10M language:en license:cc-by-nc-sa-4.0 livestream stream chat messages vtuber vtubers region:us
---
license: cc-by-nc-sa-4.0
task_categories:
- text-classification
- conversational
- text-generation
- token-classification
annotations_creators:
- crowdsourced
language_creators:
- found
language:
- en
tags:
- livestream
- stream
- chat
- messages
- vtuber
- vtubers
pretty_name: OSHIChats v1
size_categories:
- 1M<n<10M
---

## OSHIChats v1 (August 2023)
OSHIChats v1 is a dataset of 8.06 million high-quality filtered English chat messages collected from various [VTuber](https://en.wikipedia.org/wiki/VTuber) live streams.

Compared to our previous dataset, [pykeio/vtuber-chats-2023-filtered-en-8.7M](https://huggingface.co/datasets/pykeio/vtuber-chats-2023-filtered-en-8.7M), we make the following improvements:
- Include stream topic information
- Far more accurate nickname detection using NLP
  - Previously we did not match names like "dad" (nickname for Mori Calliope) or "mom" (nickname for Nina Kosaka) because they were too general. Now, we analyze the context and other information about the stream to determine whether to match such nicknames.
- Detect and normalize fan names like takodachi or pentomo

## Usage
Once you gain access to the dataset, you'll also need to log in to Hugging Face CLI with `huggingface-cli login`.

```py
from datasets import load_dataset

chats_dataset = load_dataset('pykeio/oshichats-v1-2308', split='train', revision='refs/convert/parquet')
chats_dataset[0]
# {'liver': 'FgXWZOUZA2oYHNr6qDmsTQ', 'stream': {'id': 'JHBv4BA_Y84', 'topic': 'Twisted_Wonderland'}, 'is_super': False, 'message': "i think i've grown to dislike them ", 'author': 'chxrry_head', 'time': [1660106235135797, 2126652]}
```

## Samples
```json
{
  "liver": "kieJGn3pgJikVW8gmMXE2w",
  "stream": {
    "id": "dMUhbAcI5gk",
    "topic": "minecraft"
  },
  "is_super": false,
  "message": "yay <|liver:bW9t|> is streaming while I'm awake!",
  "author": "Redribbon Vicky",
  "time": [1651976493761550, 44936]
}

{
  "liver": "yl1z3jo3XHR1riLFKG5UAg",
  "stream": {
    "id": "TgEX7HFqTYc",
    "topic": "Donkey_Kong"
  },
  "is_super": false,
  "message": "Stop running <|liver:QW1l|><|:ameHeh:|><|:ameHeh:|><|:ameHeh:|>",
  "author": "Anon",
  "time": [1616291612238864, 889273]
}
```

## Data fields
- `liver`: ID of the YouTube channel hosting the stream which the chat message came from.
- `stream`: Information about the stream.
  - `id`: Video ID of the YouTube stream.
  - `topic`: Topic of the stream (or `null` if a topic could not be determined). This can be things like `talk`, `Minecraft`, `Singing`, `GTA`, `Asmr`, etc.
- `is_super`: Whether or not the message is a Superchat (donation).
- `message`: Contents of the message. For consistency and ease of use on downstream tasks, we replace certain words with easily matchable special tokens:
  * `<|liver:{b64}|>`: The substring refers to the host of the stream.
  * `<|liver-fans:{b64}|>`: The substring refers to a nickname given to the fanbase of the host of the stream, e.g. aloupeeps or takodachis.
  * `<|known-collaborator:{channelID}:{b64}|>`: The substring refers to a fellow VTuber that is present in the stream.
  * `<|maybe-collaborator:{channelID}:{b64}|>`: The substring refers to a fellow VTuber that may or may not be part of the stream.
  * `<|collaborator-fans:{channelID}:{b64}|>`: The substring refers to the fanbase of a collaborator present in the stream.
  * `<|:{emote}:|>`: Represents a channel emote.
  * Note that `channelID` is a YouTube channel ID, and `b64` is the original substring encoded as base64.
- `author`: The username of the author.
- `time`: A tuple containing the Unix timestamp of when the message was sent, and the relative time since the start of the stream.

## License
Licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/); you must give attribution, you may not use the dataset for commercial purposes, and you must distribute any transformations or copies of the dataset under the same license. [Contact us](mailto:contact@pyke.io) for alternative/commercial licensing.