<jupyter_start><jupyter_text>Exploratory Data Analysis <NAME> & <NAME>The purpose of this notebook will be to look at the distribution of scores and decide appropriate score thresholds for our dataset  Import Key Dependencies<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

sns.set(style="white", palette="colorblind", color_codes=True)
print(sns.__version__)
assert sns.__version__ == '0.9.0' #Ensures we can use sns.scatterplot below<jupyter_output>0.9.0<jupyter_text>Load our cleaned datasetThis has been generated from make_jokes_dataset_CLEAN.py and creates the clean jokes dataset (removing special characters, ensuring the punchline and setup have a maximum wordcount etc). Note that the jokes are not clean in the classical sense of the word!<jupyter_code>df1 = pd.read_csv('../data/processed_data/jokes_dataset_CLEAN.csv')
df1 = df1.iloc[:,1:]
df1.insert(loc=0,column='datetime',value='Not given')

# df2 = pd.read_csv('../data/processed_data/scraped_data_v1_CLEAN.csv')
# df2 = df2.iloc[:,2:]
# df = pd.concat([df1,df2])

# df = pd.read_csv('data/scraped_data_v2.csv')
# df = df.iloc[:,2:]
print(len(df))
# Drop duplicates with the same ID
df = df.drop_duplicates('id') 
print(len(df))
# Catch any jokes posted under different id's but with the same body/title
df = df.drop_duplicates('body')
df = df.drop_duplicates('title')
'[removed]' in ['[removed]','[deleted]']
df.body = df.body.astype('str')
df.title = df.title.astype('str')
df.head()
df.body[2].split(' ')
df['body_length'] = [len(df.body[i].split(' ')) for i in range(df.shape[0])]
df['title_length'] = [len(df.title[i].split(' ')) for i in range(df.shape[0])]
df.describe()
print("The number of jokes with score 0 is:",len(df[df["score"] == 0]))
# Look at the distribution of jokes between a maximum and minimum score
min_score = 5
max_score = 2000
df = df[(min_score <= df["score"]) & (df["score"] <= max_score)]
df.describe()
fig, (ax1, ax2, ax3) = plt.subplots(3,1, figsize=(5,15))

sns.scatterplot(df.body_length, df.score, ax=ax1)
sns.scatterplot(df.title_length, df.score, ax=ax2)
sns.scatterplot(df.title_length, df.body_length, ax=ax3)

ax1.set_title('Score v Body Length')
ax2.set_title('Score v Title Length')
ax3.set_title('Body Length v Title Length')

ax1.set(xlabel='Body Length', ylabel='Score')
ax2.set(xlabel='Title Length', ylabel='Score')
ax3.set(xlabel='Title Length', ylabel='Body Length')

#fig.savefig('plots/scatter_plots.png', dpi=300, bbox_inches='tight')
fig, (ax1, ax2) = plt.subplots(2,1, figsize=(5,10))

sns.distplot(df.score[df.score < 200], kde=False, ax=ax1)
sns.distplot(df.body_length[df.body_length<300], kde=False, label='body', ax=ax2)
sns.distplot(df.title_length, kde=False, label='title', ax=ax2)

ax1.set_title('Distribution of Scores')
ax2.set_title('Distribution of Joke Lengths')
ax1.set(xlabel='Score')
ax2.set(xlabel='Joke Length')
plt.legend()

#fig.savefig('plots/histograms.png', dpi=300, bbox_inches='tight')<jupyter_output>C:\Users\<NAME>\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Map scoresIn this section we will be "compressing" the dataset between a certain range. To do this, we pick a certain value, and then linearly interpolate from this value up until the max_score and compress this large interval onto a smaller one. E.g we take the scores from 200 -> 2000 and linearly interpolate them so that they are in the range 200 -> 250<jupyter_code>map_score_range = [200,250] # This will define where we start thresholding our scores and using our mapping
temp_df = df[(map_score_range[0] <= df["score"]) & (df["score"] <= max_score)]

# Map range using linear interpolation
temp_df["score"] = (map_score_range[0] + (map_score_range[1] - map_score_range[0])*
                    ((temp_df["score"] - 200) / (max_score - 200))).astype('int') 
temp_df.head()
# Combine the dataframe with jokes <= map_score_range[0] with the temp_df, sample them to "shuffle them" and reset index
df = pd.concat([df.drop(temp_df.index),temp_df]).sample(frac=1).reset_index(drop=True)
df.head()
fig, (ax1, ax2) = plt.subplots(2,1, figsize=(5,10))

sns.distplot(df.score, kde=False, ax=ax1)
sns.distplot(df.body_length[df.body_length<300], kde=False, label='body', ax=ax2)
sns.distplot(df.title_length, kde=False, label='title', ax=ax2)

ax1.set_title('Distribution of Scores')
ax2.set_title('Distribution of Joke Lengths')
ax1.set(xlabel='Score')
ax2.set(xlabel='Joke Length')
plt.legend()

#fig.savefig('plots/histograms.png', dpi=300, bbox_inches='tight')<jupyter_output>C:\Users\<NAME>\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval
{'fraction_non_alphanumeric': 0.1467684857515366, 'fraction_numerical': 0.025144347178245482, 'mean_word_length': 6.065789473684211, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '172015', 'license': 'MIT', 'n_tokens_mistral': 1992, 'n_tokens_neox': 1887, 'n_words': 495, 'path': 'scripts/.ipynb_checkpoints/exploratory_data_analysis-checkpoint.ipynb', 'repo_name': 'RyanJenkinson/nlp-jcag', 'size': '5399'}