<jupyter_start><jupyter_text>May 2021 kaggle TPSThis notebook carries out initial exploration of the training data to get an understanding of the distribution and completeness of each column and relationship to the target.Data [description](https://www.kaggle.com/c/tabular-playground-series-may-2021/data): "The dataset is... synthetic, but based on a real dataset and generated using a CTGAN. The original dataset deals with predicting the category on an eCommerce product given various attributes about the listing. Although the features are anonymized, they have properties relating to real-world features".<jupyter_code># imports for EDA
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# warning and plotting preferences
import warnings
warnings.simplefilter("ignore")
sns.set_style("whitegrid")
sns.set_context("paper")
pd.options.display.max_columns = None
pd.options.display.float_format = "{:.2f}".format<jupyter_output><empty_output><jupyter_text>EDANo idea what to expect and unfamiliar with the competition this data was based on - here goes:<jupyter_code>df = pd.read_csv("../data/raw/train.csv")
df.head()
df.info()
df.describe().T<jupyter_output><empty_output><jupyter_text>ID column aside, we're looking at a sparse matrix of 50 columns. No null values, although curiously a couple of columns contain negatives. There is a maximum of 66.<jupyter_code>zed = df.drop(columns=['id', 'target']).to_numpy()
plt.spy(zed.T, precision=0, aspect='auto')
plt.autoscale()
plt.show()
df.target.value_counts()
sns.countplot(data=df, x="target")
plt.show()<jupyter_output><empty_output><jupyter_text>Focusing on the target, we have four unbalanced classes. Class 1, 4, and 3 respectively making up less than 9%, 13% and 22% of the total. Revisit the sparsity plot by class to see if there are any obvious patterns that stand out:<jupyter_code>zed1 = df[df['target'] == 'Class_1'].drop(columns=['id', 'target']).to_numpy()
zed2 = df[df['target'] == 'Class_2'].drop(columns=['id', 'target']).to_numpy()
zed3 = df[df['target'] == 'Class_3'].drop(columns=['id', 'target']).to_numpy()
zed4 = df[df['target'] == 'Class_4'].drop(columns=['id', 'target']).to_numpy()

fig, axs = plt.subplots(2, 2, sharex=False, sharey=True)

axs[0, 0].spy(zed1.T, aspect='auto')
axs[0, 0].set_title("Class 1")

axs[0, 1].spy(zed2.T, aspect='auto')
axs[0, 1].set_title("Class 2")

axs[1, 0].spy(zed3.T, aspect='auto')
axs[1, 0].set_title("Class 3")

axs[1, 1].spy(zed4.T, aspect='auto')
axs[1, 1].set_title("Class 4")

plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14384615384615385, 'fraction_numerical': 0.021538461538461538, 'mean_word_length': 6.26536312849162, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '578919', 'license': 'MIT', 'n_tokens_mistral': 932, 'n_tokens_neox': 881, 'n_words': 256, 'path': 'notebooks/kaggle_tps_202105.ipynb', 'repo_name': 'redhqs/tps-202105', 'size': '2600'}