<jupyter_start><jupyter_text>The channels I watched LOL are: Tubbo yamikazexz riotgames nightblue3 iwilldominate sneakylol thebausffs tfblade loltyler1 shiphtur tarzaned jankos ratirl drututt katevolved wingsofdeath trick2g karasmai ipav999 anniebot boxbox corejj lol_selfmade  nisqyy ikeepittaco gamergirl tobiasfateThe channels I watched PUBG are: Tubbo danucd tgltn ibiza hambinooo break halifax pubg_andymh5 grizz alisa summit1g chocotaco feyd fuzzfaze49 ashek gagod chad taryn jowybear shrimzyhttps://www.twitch.tv/tubboThe tutorial: https://www.learndatasci.com/tutorials/how-stream-text-data-twitch-sockets-python/  Setup<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
my_project_folder = 'Data-X: GGWP Toxic Behavior Public Data'
%cd drive/My Drive/{my_project_folder}/scraping
server = 'irc.chat.twitch.tv'
port = 6667
nickname = 'aki_niki'  # your username
token = '' # your token 
channel = '#tobiasfate'
# instantiate a socket
import socket
sock = socket.socket()
sock.connect((server, port))
sock.send(f"PASS {token}\n".encode('utf-8'))
sock.send(f"NICK {nickname}\n".encode('utf-8'))
sock.send(f"JOIN {channel}\n".encode('utf-8'))
resp = sock.recv(2048).decode('utf-8')
resp<jupyter_output><empty_output><jupyter_text>Write into files<jupyter_code>import logging

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s — %(message)s',
                    datefmt='%Y-%m-%d_%H:%M:%S',
                    handlers=[logging.FileHandler('chat.log', encoding='utf-8')])
logging.info(resp)
# This process is keep collecting data. Interrupt it to go to the next step

from emoji import demojize

while True:
    resp = sock.recv(2048).decode('utf-8')

    if resp.startswith('PING'):
        sock.send("PONG\n".encode('utf-8'))
    
    elif len(resp) > 0:
        logging.info(demojize(resp))
        
# run in terminal: tail -f chat.log
# sample message
msg = "2020-11-24_16:54:53 — :flam1ngoat!<EMAIL> PRIVMSG #tobiasfate :I think"
#'2018-12-10_11:26:40 — :spappygram!<EMAIL> PRIVMSG #ninja :Chat, let Ninja play solos'
# date: split it off and parse it
from datetime import datetime

time_logged = msg.split()[0].strip()

time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')

time_logged
username_message = msg.split('—')[1:]
username_message = '—'.join(username_message).strip()

username_message
import re

username, channel, message = re.search(':(.*)\!.*@.*\.tmi\.twitch\.tv PRIVMSG #(.*) :(.*)', username_message).groups()
#  (.*) — will capture part of the string

print(f"Channel: {channel} \nUsername: {username} \nMessage: {message}")
import pandas as pd

def get_chat_dataframe(file):
    data = []

    with open(file, 'r', encoding='utf-8') as f:
        lines = f.read().split('\n\n\n')
        
        for line in lines:
            try:
                time_logged = line.split('—')[0].strip()
                time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')

                username_message = line.split('—')[1:]
                username_message = '—'.join(username_message).strip()

                username, channel, message = re.search(
                    ':(.*)\!.*@.*\.tmi\.twitch\.tv PRIVMSG #(.*) :(.*)', username_message
                ).groups()

                d = {
                    'dt': time_logged,
                    'channel': channel,
                    'username': username,
                    'message': message
                }

                data.append(d)
            
            except Exception:
                pass
            
    return pd.DataFrame().from_records(data)
        
    
df = get_chat_dataframe('chat.log')
df.set_index('dt', inplace=True)

print(df.shape)

df
sock.close()<jupyter_output><empty_output><jupyter_text>Conver to CSV<jupyter_code># Get only the message column as only that is required for the model prediction
df2 = df['message'].to_frame() 
df2 = df2.drop_duplicates() # drop duplicates
game_platform = "pubg_twitch"
df2.to_csv('../data/scraped/' + game_platform + '.csv', encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14257860102364123, 'fraction_numerical': 0.01681696319766025, 'mean_word_length': 3.2006141248720574, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '164803', 'license': 'Apache-2.0', 'n_tokens_mistral': 1473, 'n_tokens_neox': 1382, 'n_words': 302, 'path': 'student-projects/fall-2020/GGWP-Identify-Toxic-Behavior-in-Gaming-with-Public-Data/scraping/Twitch_Scraper.ipynb', 'repo_name': 'UCBerkeley-SCET/DataX-Berkeley', 'size': '4159'}