<jupyter_start><jupyter_text>Exploratory Data AnalysisWorking on: https://www.kaggle.com/c/google-quest-challenge/<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

from pathlib import Path
from matplotlib_venn import venn2
data = Path('data')
train = data/'train.csv'
test = data/'test.csv'
sample = data/'sample_submission.csv'

train_df = pd.read_csv(train)
test_df = pd.read_csv(test)
sample_df = pd.read_csv(sample)
print("Train Rows", len(train_df))
print("Test Rows", len(test_df))
train_df.head()
test_df.head()<jupyter_output><empty_output><jupyter_text>Unique Users If all users are unique we probably don't care about the user columns. If they are not, then there may be some signal here.<jupyter_code># Surprisingly to me, there are duplicate users
print("Unique question user pages:", len(train_df['question_user_page'].unique()))
# Let's look at some of the duplicate users
train_df['question_user_page'].value_counts()[:5]
# Surprisingly to me, there are duplicate users
print("Unique answer user pages:", len(train_df['answer_user_page'].unique()))
# Let's look at some of the duplicate users
train_df['answer_user_page'].value_counts()[:5]<jupyter_output><empty_output><jupyter_text>Let's see if there is any overlap between the training and test sets:<jupyter_code># Train set question askers overlapped with test set question askers
np.intersect1d(train_df['question_user_page'].unique(), test_df['question_user_page'].unique())
# Train set question answerers overlapped with test set question answerers
np.intersect1d(train_df['answer_user_page'].unique(), test_df['answer_user_page'].unique())
# Train set question askers overlapped with test set question answerers
np.intersect1d(train_df['question_user_page'].unique(), test_df['answer_user_page'].unique())
# Train set question answerers overlapped with test set question askers
np.intersect1d(train_df['answer_user_page'].unique(), test_df['question_user_page'].unique())<jupyter_output><empty_output><jupyter_text>So there is a little bit of overlap between test and training set, but not a ton.Let's plot these overlap relations between the training and test set as venn diagrams.<jupyter_code>plt.figure(figsize=(23,13))

plt.subplot(321)
venn2([set(train_df.question_user_page.unique()), set(test_df.question_user_page.unique())], set_labels = ('Train set', 'Test set') )
plt.title("Common question_user_page in training and test data", fontsize=15)
#plt.show()

#plt.figure(figsize=(15,8))
plt.subplot(322)
venn2([set(train_df.answer_user_page.unique()), set(test_df.answer_user_page.unique())], set_labels = ('Train set', 'Test set') )
plt.title("Common answer_user_page in training and test data", fontsize=15)
#plt.show()

plt.show()

plt.figure(figsize=(23,13))


#plt.figure(figsize=(15,8))
plt.subplot(321)
venn2([set(train_df.question_user_page.unique()), set(train_df.answer_user_page.unique())], set_labels = ('Train set', 'Test set') )
plt.title("Common users in both question & answeer in train data", fontsize=15)

#plt.figure(figsize=(15,8))
plt.subplot(322)
venn2([set(test_df.question_user_page.unique()), set(test_df.answer_user_page.unique())], set_labels = ('Train set', 'Test set') )
plt.title("Common users in both question & answeer in test data", fontsize=15)

plt.subplots_adjust(wspace = 0.5, hspace = 0.5,
                    top = 0.9)<jupyter_output><empty_output><jupyter_text>Distribution of Target Variables We can look at the distribution of target variables to get a sense for what ranges these values typically take. For starters, we can just summarize them with pandas.<jupyter_code>targets = list(sample_df.columns[1:])
train_df[targets].describe()<jupyter_output><empty_output><jupyter_text>So we can see from the above that `question_asker_intent_understanding` has a pretty high mean value of `0.893` while others like `question_conversational` have a much lower mean value of `0.057`.Why is this useful? First, it gives us some background intuition about the typical values for these targets. But more importantly, if we're disciplined data scientists, we will run the above code against the output predictions our model makes. When we do this we should pay close attentiont to areas in which the two distributions differ and think carefully about steps we might take to help them align. The above can give us a general sense for each target variable, but we can go one step further and plot them to see the shapes of their distributions.<jupyter_code>fig, axes = plt.subplots(6, 5, figsize=(18, 15))
axes = axes.ravel()
bins = np.linspace(0, 1, 20)

for i, col in enumerate(targets):
    ax = axes[i]
    sns.distplot(train_df[col], label=col, kde=False, bins=bins, ax=ax)
    # ax.set_title(col)
    ax.set_xlim([0, 1])
    ax.set_ylim([0, 6079])
plt.tight_layout()
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>Once again if we're disciplined, we should run this code against our output predictions and look for problems. (They don't have to match, but occasionally mismatches might indicate a problem we should take a moment to think about)  Investigate Host Websites Is there any information that we can leverage based on the host website? Do some sites attract users with better spelling? Is there much overlap between the hosts in the training set and the test set?<jupyter_code># Train set question askers overlapped with test set question answerers
print("Unique hosts in train:", len(train_df['host'].unique()))
print("Unique hosts in test:", len(test_df['host'].unique()))
print("Intersecting hosts:", len(np.intersect1d(train_df['host'].unique(), test_df['host'].unique())))<jupyter_output>Unique hosts in train: 63
Unique hosts in test: 55
Intersecting hosts: 54<jupyter_text>So it looks like there's actually a lot of overlap between the training set hosts and those in the test set. Next, let's look at the best and worst scores for each target when grouped by host. Do certain sites promote poor spelling? Opinion based questions? etc.<jupyter_code>def group_by_host_and_get_score(df, column):
    grouped_df = df.groupby('host') \
       .agg(count=(column, 'size'), mean_score=(column, 'mean')) \
       .reset_index() \
       .sort_values(by='mean_score')
    
    return grouped_df
for target in targets:
    
    grouped_df = group_by_host_and_get_score(train_df, target)
    print('----------------------')
    print(target)
    print(grouped_df[:5])
    print(grouped_df[-5:])
    print()<jupyter_output>----------------------
question_asker_intent_understanding
                                  host  count  mean_score
22  expressionengine.stackexchange.com     34    0.790850
31                    mathoverflow.net     82    0.837398
13            crypto.stackexchange.com     35    0.838095
18               dsp.stackexchange.com     29    0.842912
29              math.stackexchange.com    187    0.851456
                           host  count  mean_score
16        diy.stackexchange.com     97    0.965636
6     biology.stackexchange.com     53    0.968553
12    cooking.stackexchange.com     67    0.978441
36  meta.math.stackexchange.com      2    1.000000
33           meta.askubuntu.com      2    1.000000

----------------------
question_body_critical
                                  host  count  mean_score
45          robotics.stackexchange.com      2    0.333333
11        codereview.stackexchange.com     39    0.367521
22  expressionengine.stackexchange.com     34    0.460784
18      [...]<jupyter_text>Question and Answer Lengths Approximately how many words are present in each question and answer? This may end up being important when we decide how long of sentences we would like to feed to our model.<jupyter_code>num_words_title = train_df['question_title'].apply(lambda x: len(x.split()))
num_words_question = train_df['question_body'].apply(lambda x: len(x.split()))
num_words_answer = train_df['answer'].apply(lambda x: len(x.split()))
# Title
print(num_words_title.describe())
num_words_title.hist()
# Question Body
print(num_words_question.describe())
num_words_question.hist()
# Answer Body
print(num_words_answer.describe())
num_words_answer.hist()<jupyter_output>count    6079.000000
mean      133.022208
std       155.570789
min         3.000000
25%        47.000000
50%        88.000000
75%       163.000000
max      2376.000000
Name: answer, dtype: float64
{'fraction_non_alphanumeric': 0.10923570664447878, 'fraction_numerical': 0.036728871983834545, 'mean_word_length': 4.431891542930924, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '616754', 'license': 'MIT', 'n_tokens_mistral': 2827, 'n_tokens_neox': 2570, 'n_words': 799, 'path': '00_ExploratoryDataAnalysis.ipynb', 'repo_name': 'JoshVarty/GoogleQALabeling', 'size': '8413'}