<jupyter_start><jupyter_text>Data Understanding ExtractingThe main goal of this notebook is to provide insights about existing data and eventually prototype anything that is connected with ETL Pipeline.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Let's read data<jupyter_code>messages = pd.read_csv('data/csv/disaster_messages.csv', index_col=False)
messages.head()
messages.shape<jupyter_output><empty_output><jupyter_text>So it looks like we have around 26248 messages about disaster with english translation and original language. Now let's see what kind of genre's are available.<jupyter_code>messages['genre'].unique()<jupyter_output><empty_output><jupyter_text>So there are three genres direct, social and news that are available.<jupyter_code>categories = pd.read_csv('data/csv/disaster_categories.csv', index_col=False)
categories.head()
categories.shape<jupyter_output><empty_output><jupyter_text>Now let's perform the concatenation of both dataframes<jupyter_code>df = pd.concat([messages, categories], axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>CleaningNow as the data is extracted, now let's go with cleaning the data.As first step let's remove these 'id' columns.<jupyter_code>df = df.drop(columns=['id'])
df.head()<jupyter_output><empty_output><jupyter_text>Let's see which columns contain missing values.<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>It looks like the 'original' column contains missing values. However the 'message' column does not have any missing item. So let's see how messages looks like when 'original' is missing.<jupyter_code>df[(df['original']).isna()]['message']<jupyter_output><empty_output><jupyter_text>So the data is messages is in english - so then is the message is already in english there is no need to fill 'original' column. So let's focus only on original column then (we don't need to drop any row due to NaN value). We could drop 'original' column, but let's leave it, as we don't need to safe space dramatically. Remove duplicates.<jupyter_code>df['message'].duplicated().sum()
df = df.drop_duplicates(['message'])
df['message'].duplicated().sum()
df.head()<jupyter_output><empty_output><jupyter_text>Now let's clean the 'categories' column<jupyter_code>categories = df['categories'].str.split(pat=';', expand=True)
row = categories.iloc[0]
category_colnames = [r.split('-')[0] for r in row]
categories.columns = category_colnames

for column in categories:
    # set each value to be the last character of the string
    categories[column] = categories[column].apply(lambda val: val.split('-')[1])

    # convert column from string to numeric
    categories[column] = pd.to_numeric(categories[column], downcast='integer')

df = df.drop(columns=['categories'])
df = pd.concat([df, categories], axis=1)
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11416637660981552, 'fraction_numerical': 0.003480682213713888, 'mean_word_length': 7.258620689655173, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '81555', 'license': 'MIT', 'n_tokens_mistral': 898, 'n_tokens_neox': 867, 'n_words': 259, 'path': 'data/Data Understanding.ipynb', 'repo_name': 'danpeczek/disaster-response-pipeline', 'size': '2873'}