<jupyter_start><jupyter_text>[Module 1.0] Tweet 입력 데이터 클린징 및 S3에 업로드이 노트북에서는 아래와 같은 작업을 진행 합니다.- Tweet 데이타를 클린징- Tweet 데이타 통계 확인- 총 990개의 이모티콘 레이블을 10개로 한정 함- Emoticon to Index의 사전 생성- Tweet Data를 Train 및 Test로 분리    - Train_01_text, Train_01_label, Train_02_text, Train_02_label 분리    - Test text, Test label로 분리- Train_01_text, Train_01_label --> tweet_file_01.csv.gz 후 S3에 업로드- Train_02_text, Train_02_label --> tweet_file_02.csv.gz 후 S3에 업로드총 소요시간은 약 1분 걸립니다.---Original Data Source:    - D<NAME>, Deep Learning Cookbook. Ch7, Suggesting Emojishttps://www.amazon.com/Deep-Learning-Cookbook-Practical-Recipes/dp/149199584X<jupyter_code>import pandas as pd
import re
import boto3
import numpy as np
import os<jupyter_output><empty_output><jupyter_text>Look at Raw Tweet Data- 참고로 Tweet 데이터는 아래 책에 제공하는 데이타를 data 폴더에 미리 다운로드 했습니다.- Original Data Source    - D<NAME>, Deep Learning Cookbook. Ch7, Suggesting Emojis<jupyter_code>file_name = os.path.join('data', "emojis.csv")
def read_data(file_name):
    all_tweets = pd.read_csv(file_name, names=['content', 'sentiment'])
    return all_tweets

raw_tweets = read_data(file_name)
print("all_tweets shape: ", raw_tweets.shape)
raw_tweets.head(11)<jupyter_output>all_tweets shape:  (806204, 2)<jupyter_text>Tweet Data Cleaning 중복 데이터 제거를 합니다.<jupyter_code>def drop_duplicate_records(all_tweets):
    all_tweets_cleaned_df = all_tweets.drop_duplicates() 
    return all_tweets_cleaned_df

tweet_temp_df = drop_duplicate_records(raw_tweets)
print("Unique Tweets : \n {} ".format(tweet_temp_df.shape))
num_tweets = tweet_temp_df.shape[0]
all_tweets_cleaned_df = tweet_temp_df<jupyter_output><empty_output><jupyter_text>Tweet Data Shape- 데이터는 Content 와 Sentiment의 두개의 컬럼으로 구성<jupyter_code>all_tweets_cleaned_df.head()
# show emoticon stat
num_emoticons = all_tweets_cleaned_df['sentiment'].nunique()
print("The number of unique emoticon: {}".format(num_emoticons)) # The number of unique emoticon: 990
print("Top 10 emoticons: ")
all_tweets_cleaned_df['sentiment'].value_counts()[0:10]<jupyter_output>The number of unique emoticon: 990
Top 10 emoticons:<jupyter_text>Tweet 의 단어 갯수각 Tweet에 대한 단의의 개수에 대한 통계를 내었습니다. 중위수 기준으로 단어의 갯수는 8개 입니다.<jupyter_code>import pandas as pd

def num_words_sentence(sentence):
    tokens = sentence.split()
    len_tokens = len(tokens)
    return len_tokens
    
def store_num_words(texts):    
    num_tokens= list()
    for text in texts:
        len_tokens = num_words_sentence(text)
        num_tokens.append(len_tokens)
                     
    return num_tokens
        
num_tokens_data = store_num_words(all_tweets_cleaned_df['content'])    
                   
df_num_tokens = pd.DataFrame(num_tokens_data, columns=['num_tokens'])    

print("Statistics of the number of words per tweet: ")
df_num_tokens.describe()<jupyter_output>Statistics of the number of words per tweet:<jupyter_text>이모티콘 레이블을 10개로 한정 이모티콘의 레이블을 10개로 한정하기 위해, Top10 을 제외한 모든 데이타는 삭제 함..<jupyter_code>num_threshold_lables = 9300 # process more than 9000 for lables
num_use_records = num_tweets # total of tweets

def filter_less_label(all_tweets, num_use_records,   num_filter_less_label):
    # Desc: filter the total data for being less label with specific number
    use_tweets = all_tweets[0:num_use_records]
    tweets = use_tweets.groupby('sentiment').filter(lambda c:len(c) > num_filter_less_label)

    return tweets

tweets = filter_less_label(all_tweets_cleaned_df, num_use_records,   num_threshold_lables)
tweets.sentiment.value_counts()<jupyter_output><empty_output><jupyter_text>레이블의 밸런싱10개의 레이블 중에서 가장 작은 갯수가 있는 레이블 기준으로 데이타를 가공 합니다.10개의 레이블은 각각 9419개씩의 Tweet의 갯수를 가집니다.<jupyter_code>from sklearn.utils import resample

print(tweets.sentiment.unique())
emoji_0_df = tweets.query("sentiment == '🤔'")
emoji_1_df = tweets.query("sentiment == '❤'")
emoji_2_df = tweets.query("sentiment == '😭'")
emoji_3_df = tweets.query("sentiment == '😂'")
emoji_4_df = tweets.query("sentiment == '🙄'")

emoji_5_df = tweets.query("sentiment == '😍'")
emoji_6_df = tweets.query("sentiment == '💕'")
emoji_7_df = tweets.query("sentiment == '😊'")
emoji_8_df = tweets.query("sentiment == '😩'")
emoji_9_df = tweets.query("sentiment == '🔥'")

minority_count = min(emoji_0_df.shape[0],
                     emoji_1_df.shape[0],
                     emoji_2_df.shape[0],
                     emoji_3_df.shape[0],
                     emoji_4_df.shape[0],
                     emoji_5_df.shape[0],
                     emoji_6_df.shape[0],
                     emoji_7_df.shape[0],
                     emoji_8_df.shape[0],
                     emoji_9_df.shape[0],                     
                    )
minority_count

emoji_0_df = resample(emoji_0_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_1_df = resample(emoji_1_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)


emoji_2_df = resample(emoji_2_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_3_df = resample(emoji_3_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_4_df = resample(emoji_4_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_5_df = resample(emoji_5_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_6_df = resample(emoji_6_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_7_df = resample(emoji_7_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_8_df = resample(emoji_8_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

emoji_9_df = resample(emoji_9_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

df_balanced = pd.concat([
    emoji_0_df, emoji_1_df,
    emoji_2_df, emoji_3_df,
    emoji_4_df, emoji_5_df,
    emoji_6_df, emoji_7_df,
    emoji_8_df, emoji_9_df,    
])
df_balanced = df_balanced.sample(frac=1).reset_index(drop=True)

print("Shape: ", df_balanced.shape)
print(df_balanced.sentiment.value_counts())<jupyter_output>['🤔' '❤' '😭' '😂' '🙄' '😍' '💕' '😊' '😩' '🔥']
Shape:  (94190, 2)
🔥    9419
😊    9419
😂    9419
😭    9419
😍    9419
❤    9419
💕    9419
😩    9419
🤔    9419
🙄    9419
Name: sentiment, dtype: int64<jupyter_text>Emoticon to index 의 사전 생성추후에 추론시에 나온 레이블에 대해서 이모티콘으로 변환하기 위해 Dictionary 개체를 작성 합니다.<jupyter_code>from TweetData import TweetData # Custom class for handling input data
tweet_data = TweetData(df_balanced)
# make emoji_to_idx and save it into data
emoji_to_idx = tweet_data.make_sentimet_label()<jupyter_output>data/emoji_to_idx.pickle is saved
{'❤': 0, '💕': 1, '🔥': 2, '😂': 3, '😊': 4, '😍': 5, '😩': 6, '😭': 7, '🙄': 8, '🤔': 9}<jupyter_text>Tweet 데이터를 Train text, Train label, Test text, Test label으로 분리<jupyter_code>texts, labels = tweet_data.make_texts_lables()
print("text: {}, label: {}: \n".format(texts[0:5], labels[0:5]))
df = pd.DataFrame(data={'content':texts, 'label':labels})
df.label<jupyter_output><empty_output><jupyter_text>Train 용으로 2개의 파일을 두개를 만들고 (전체 대비 0.45, 0.45), 1개의 테스트 파일(0.1)을 생성 함<jupyter_code>train_text, train_label, test_text, test_label = tweet_data.split_train_test_data(texts, labels, 0.9)
train_text_01, train_label_01, train_text_02, train_label_02 = tweet_data.split_train_test_data(train_text, train_label, 0.5)

DATA_COLUMN = 'TWEET'
LABEL_COLUMN = 'LABEL'

tweet_file_01_df = pd.DataFrame({DATA_COLUMN: train_text_01,LABEL_COLUMN:train_label_01})
tweet_file_02_df = pd.DataFrame({DATA_COLUMN: train_text_02,LABEL_COLUMN:train_label_02})
tweet_file_test_df = pd.DataFrame({DATA_COLUMN: test_text,LABEL_COLUMN:test_label})
print(tweet_file_01_df.info())
print(tweet_file_02_df.info())
print(tweet_file_test_df.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 42385 entries, 0 to 42384
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   TWEET   42385 non-null  object
 1   LABEL   42385 non-null  int64 
dtypes: int64(1), object(1)
memory usage: 662.4+ KB
None
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 42386 entries, 0 to 42385
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   TWEET   42386 non-null  object
 1   LABEL   42386 non-null  int64 
dtypes: int64(1), object(1)
memory usage: 662.4+ KB
None
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 9419 entries, 0 to 9418
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   TWEET   9419 non-null   object
 1   LABEL   9419 non-null   int64 
dtypes: int64(1), object(1)
memory usage: 147.3+ KB
None<jupyter_text>**2개의 Train 파일을 data/split/ 에 저장**- 이 파일은 핛습의 용도로 사용함**1개의 Test파일을 data/test/에 저장**- 이 파일은 추후에 추론시에 사용함<jupyter_code>save_split_data_dir = 'data/split'
save_test_data_dir = 'data/test'
tweet_file_01_file = "tweet_file_01.csv"
tweet_file_02_file = "tweet_file_02.csv"
tweet_file_test_file = "tweet_file_test.csv"
os.makedirs(save_split_data_dir, exist_ok=True)
os.makedirs(save_test_data_dir, exist_ok=True)
tweet_data.save_input_data(save_split_data_dir, tweet_file_01_file , tweet_file_01_df)
tweet_data.save_input_data(save_split_data_dir, tweet_file_02_file , tweet_file_02_df)
tweet_data.save_input_data(save_test_data_dir, tweet_file_test_file , tweet_file_test_df)

tweet_file_01_file_path = os.path.join(save_split_data_dir, tweet_file_01_file)
tweet_file_02_file_path = os.path.join(save_split_data_dir, tweet_file_02_file)
print("tweet_file_01_file_path: ", tweet_file_01_file_path)
print("tweet_file_02_file_path: ", tweet_file_02_file_path)<jupyter_output>tweet_file_01_file_path:  data/split/tweet_file_01.csv
tweet_file_02_file_path:  data/split/tweet_file_02.csv<jupyter_text>2개의 Train 파일을 gzip으로 압축함<jupyter_code>! gzip -f {tweet_file_01_file_path}
! gzip -f {tweet_file_02_file_path}<jupyter_output><empty_output><jupyter_text>로컬에 저장된 2개의 gzip Train 파일을 S3에 저장 함.<jupyter_code>import boto3
import sagemaker

# Get region 
session = boto3.session.Session()
region_name = session.region_name

# Get SageMaker session & default S3 bucket
sagemaker_session = sagemaker.Session()
bucket = sagemaker_session.default_bucket()
s3_destination_path_csv = 's3://{}/tweet_emoticon/csv'.format(bucket)
print(s3_destination_path_csv)
!aws s3 cp  $save_split_data_dir $s3_destination_path_csv/ --recursive
! aws s3 ls {s3_destination_path_csv}/
%store s3_destination_path_csv
%store save_split_data_dir<jupyter_output>Stored 's3_destination_path_csv' (str)
Stored 'save_split_data_dir' (str)
{'fraction_non_alphanumeric': 0.10647032574743417, 'fraction_numerical': 0.038375725122713075, 'mean_word_length': 3.424003158310304, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '161595', 'license': 'MIT', 'n_tokens_mistral': 4837, 'n_tokens_neox': 4843, 'n_words': 843, 'path': 'Tweet-BERT/1.1.Prepare-Tweet-Data.ipynb', 'repo_name': 'gonsoomoon-ml/RecommendEmoticon', 'size': '11215'}