<jupyter_start><jupyter_text>In this notebook file, I perform data exploration and cleaning for the kickstarter classification. Through this, I mainly rely on `pandas` and `sklearn` for data exploration and pre-processing. I utilize `matplotlib`, `seaborn` packages for data visualization.The data exploration here is broken to the following steps: 1. Set-up dependencies 2. Load data 3. Explore data structure 4. Check on missing values 5. Check on duplicate rows 6. Parse datetime variables  7. Check on outliers 8. Encode categorical variables 10. Save data  1. Set-up DependenciesFirst things first, I import all dependencies and set up the current and the project directories:<jupyter_code>%matplotlib inline
import os, inspect
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder


# get path to current and root directories
CURR_DIR = os.path.dirname(inspect.getabsfile(inspect.currentframe()))
ROOT_DIR = os.path.dirname(CURR_DIR)<jupyter_output>/home/aidin/miniconda3/envs/kickstarter/lib/python3.7/site-packages/sklearn/utils/__init__.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sequence<jupyter_text>2. Load dataI load data to a pandas data frame with `read_csv` method and name it `df` :<jupyter_code>fname = os.path.join(ROOT_DIR, "kickstarter", "data", "raw", "data.csv")
df = pd.read_csv(fname)<jupyter_output><empty_output><jupyter_text>I make a deep copy of the data frame first. This allows me to modify data freely while I always have the original data for further reference.<jupyter_code>df_raw = df.copy(deep=True)<jupyter_output><empty_output><jupyter_text>3. Explore data structureOnce the data has been loaded, I look at the basic structure of the data using the `head()` method to print out the first five rows. I also print out the column names, the shape and the data types. By this, I can get a general idea of the structure of data and data columns.<jupyter_code>print("-"*30 + " columns " + "-"*30 + os.linesep)
print(repr(df.columns))
print("-"*30 + " shap e" + "-"*30 + os.linesep)
print(df.shape)
print("-"*30 + " data types " + "-"*30 + os.linesep)
print(df.dtypes)
print("-"*30 + " top 5 rows " + "-"*30 + os.linesep)
df.head()<jupyter_output>------------------------------ columns ------------------------------

Index(['ID', 'name', 'category', 'main_category', 'currency', 'deadline',
       'goal', 'launched', 'pledged', 'state', 'backers', 'country',
       'usd pledged', 'usd_pledged_real', 'usd_goal_real'],
      dtype='object')
------------------------------ shap e------------------------------

(378661, 15)
------------------------------ data types ------------------------------

ID                    int64
name                 object
category             object
main_category        object
currency             object
deadline             object
goal                float64
launched             object
pledged             float64
state                object
backers               int64
country              object
usd pledged         float64
usd_pledged_real    float64
usd_goal_real       float64
dtype: object
------------------------------ top 5 rows ------------------------------<jupyter_text>Here, I can see  that the kickstarter data frame has 15 columns. All the column labels are in lowercase which is the way I usually prefer to work with. There are no redundant columns and everything looks fine.Moreover, I see the data frame includes several categorical variables that can have connections to the target prediction variable (state). I might consider encoding of the categorical variables into numeric variables. Let's look at the unique values of each categorical variables:<jupyter_code>for column in ["main_category", "category", "currency", "country", "state"]:
    print(f"{column} labels:  ", repr(df[column].unique()))<jupyter_output>main_category labels:   array(['Publishing', 'Film & Video', 'Music', 'Food', 'Design', 'Crafts',
       'Games', 'Comics', 'Fashion', 'Theater', 'Art', 'Photography',
       'Technology', 'Dance', 'Journalism'], dtype=object)
category labels:   array(['Poetry', 'Narrative Film', 'Music', 'Film & Video', 'Restaurants',
       'Food', 'Drinks', 'Product Design', 'Documentary', 'Nonfiction',
       'Indie Rock', 'Crafts', 'Games', 'Tabletop Games', 'Design',
       'Comic Books', 'Art Books', 'Fashion', 'Childrenswear', 'Theater',
       'Comics', 'DIY', 'Webseries', 'Animation', 'Food Trucks',
       'Public Art', 'Illustration', 'Photography', 'Pop', 'People',
       'Art', 'Family', 'Fiction', 'Accessories', 'Rock', 'Hardware',
       'Software', 'Weaving', 'Gadgets', 'Web', 'Jazz', 'Ready-to-wear',
       'Festivals', 'Video Games', 'Anthologies', 'Publishing', 'Shorts',
       'Electronic Music', 'Radio & Podcasts', 'Apps', 'Cookbooks',
       'Apparel', 'Metal', 'Comedy', 'Hip-Hop'[...]<jupyter_text>Here, the results show that the target `state` column include 6 unique values while the current objective of the project only concerns with the success-failure classification of kickstarters. To adapt the data to this specific problem, I take out data rows with states labeled as `canceled`, `live`, `undefined` and `suspended`. This results in a subset of kickstarter data that only includes `failed` and `successful` labels.<jupyter_code>df = df.loc[df.state.isin(["failed", "successful"])]<jupyter_output><empty_output><jupyter_text>To avoid blank space in the column label I rename `usd pledged` to `usd_pledged`:<jupyter_code>df.rename(index=str, columns={"usd pledged": "usd_pledged"}, inplace=True)<jupyter_output><empty_output><jupyter_text>4. Check on missing values Now that I have generated some basic information about the data, I take a deeper look into the issues that data might pose and perform data cleaning if needed.I start off by investigating for missing values. The missing (null) values is one of the most common issue with raw data wherein the pandas terminology are basically referred by NaN values. The missing values can be related to when an entry in data is not filled properly, the data entry is not available or there is a computational error. I look at NaN values in pandas data frame by `isnull()` method:<jupyter_code>n_rows_with_nan = df.isnull().sum()
n_rows_with_nan<jupyter_output><empty_output><jupyter_text>Let's compute what percentage of columns contain NaN values:<jupyter_code>n_rows = df.shape[0]
nan_percentage = n_rows_with_nan.values / n_rows * 100

nan_percentage<jupyter_output><empty_output><jupyter_text>This shows that the `name` and `usd_pledged` are the only columns that contain null values. Knowing that there exist two separate columns for real used pledged amounts, `usd_pledged` and `used_pledged_real`, the column with missing values can easily be discarded. In the case of the `name` column, hopefully, there are not many missing values included. So, I assume that the dropping rows with null values will not heavily affect the end performance of the modeling. Moving forward, I drop the whole `used_pledged_real` column and any rows with `null` values:<jupyter_code>df.dropna(axis=0, how="any", subset=["name"], inplace=True)
df.drop(axis=1, columns=["usd_pledged"], inplace=True)
df.isnull().values.any()<jupyter_output><empty_output><jupyter_text>5. Check on duplicate rows In the next step, I check on the `id`s or `names` columns for duplicate rows. By this, I will try to discard redundant data records.<jupyter_code>n_rows_with_duplicates = df.duplicated(subset=["ID", "name"])
n_rows_with_duplicates.any()<jupyter_output><empty_output><jupyter_text>There are no duplicate rows in the data frame.  6. Parse datetime variablesThe `deadline` and `launched` columns that describe temporal stamps in the data have string `object` type values. Let's cast these two columns to pandas `datatime64` object. Casting temporal values as `datatime64` object is a quite efficient when decomposing temporal data-time data to sub-temporal stamps e.g. getting day, month or year from a 'month/day/year' object. This avoids any string-like processing and is useful when one wants to generate relative time data from absolute time stamps. Dropping the time component of date-time inputs, the following converts the `deadline` and `launched` columns as `datetime64`:<jupyter_code>df.deadline = pd.to_datetime(df.deadline, format="%Y-%m-%d")
df.launched = pd.DataFrame(pd.to_datetime(df.launched, format="%Y-%m-%d"))
df.deadline = pd.to_datetime(df.deadline.dt.date)
df.launched = pd.to_datetime(df.launched.dt.date)
df.dtypes<jupyter_output><empty_output><jupyter_text>As another observation, both `launced` and `deadline` columns in the raw data only relate to absolute temporal values, recorded daily. I suppose that daily temporal resolution here could not correlate well to the Kickstarter state prediction problem. To deal with this issue, I add up a delta time variable that encodes the duration of kickstarters. Given that the type of time stamps has been casted to `datetime64`, the kickstarter duration can be easily obtained as follows:<jupyter_code>df["duration"] = (df.deadline - df.launched)
df.duration = df.duration.dt.days.astype("float32")<jupyter_output><empty_output><jupyter_text>Moreover, I suspect that kickstarter events are more often monthly or seasonal. That is, the season or the month when a kickstarter project starts that can have connections to its success or failure. Per this hypothesis, It is worthy to also consider seasonal and monthly components of `laucned` timestamps into the classification problem. So, let's generate two separate `quarter` and `month` columns that encode kickstarters launch events:<jupyter_code>df["month"] = df.launched.dt.month.astype("float32")
df["quarter"] = df.launched.dt.quarter.astype("float32")
df.head()<jupyter_output><empty_output><jupyter_text>7. Check on outliersHaving the data formated, let's look at the possible outliers. I analyze the outliers through extreme value analsis via scatter plots and box plots of the continuous variables:<jupyter_code>fig, axes = plt.subplots(ncols=2, figsize=(10,6), sharey=True)
_ = sns.stripplot(y="state", x="goal", data=df, ax=axes[0], orient="h")
_ = sns.stripplot(y="state", x="usd_goal_real", data=df, ax=axes[1], orient="h")
fig, axes = plt.subplots(ncols=2, figsize=(10,6), sharey=True)
_ = sns.stripplot(y="state", x="pledged", data=df, ax=axes[0], orient="h")
_ = sns.stripplot(y="state", x="usd_pledged_real", data=df, ax=axes[1], orient="h")
fig, axes = plt.subplots(ncols=1, figsize=(5,6), sharey=True)
_ = sns.stripplot(y="state", x="backers", data=df, ax=axes, orient="h")
fig, ax = plt.subplots(nrows=(5), figsize=(10,8))
sns.boxplot(data = df[["goal"]]
            ,orient="h",
            palette="Set2",
            width=200,
            ax=ax[0])
sns.boxplot(data = df[["goal"]]
            ,orient="h",
            palette="Set2",
            width=200,
            ax=ax[1])
sns.boxplot(data = df[["usd_goal_real"]]
            ,orient="h",
            palette="Set2",
            width=200,
            ax=ax[2])
sns.boxplot(data = df[["usd_pledged_real"]]
            ,orient="h",
            palette="Set2",
            width=200,
            ax=ax[3])
sns.boxplot(data = df[["backers"]]
            ,orient="h",
            palette="Set2",
            width=200,
            ax=ax[4])<jupyter_output><empty_output><jupyter_text>One can see here that several points exist beyond the upper extreme ranges. This suggests, there are significant amont of outliers and could have have negative effects on the performance of classfication. Not explicitly removing outliers, I will try to address this issue by `sckikt-learn` robust scaler whilde doing data normalization.  8. Encode categorical variablesThere are 5 variables in the data with categorical values: `category`, `main_category`, `currency`, `country` and `state`. To be able to integrate these variables into the classification problem, they need to be encoded into numerical values. This can be easily achieved by label encoding. I use the Scikit-Learn `LabelEncoder` for label encoding.<jupyter_code>label_encoder = LabelEncoder()
df["category_code"] = label_encoder.fit_transform(df.category).astype("float32")
df["main_category_code"] = label_encoder.fit_transform(df.main_category).astype("float32")
df["country_code"] = label_encoder.fit_transform(df.country).astype("float32")
df["state_code"] = label_encoder.fit_transform(df.state)<jupyter_output><empty_output><jupyter_text>9. Drop redundant variablesLstly, I drop the rendundant columns and cast all data variables as `float`.<jupyter_code>df.drop(axis=1, columns=[
    "ID",
    "name",
    "launched",
    "deadline",
    "currency",
    "category", 
    "main_category",
    "country", 
    "state"
], inplace=True)
df.backers = df.backers.astype("float32")<jupyter_output><empty_output><jupyter_text>10. Save clean dataReset the index and save the processed data frame as `csv`:<jupyter_code>df.reset_index(inplace=True)
fname = os.path.join(ROOT_DIR, "kickstarter", "data", "ext", "data_clean.csv")
df.to_csv(fname)
df.dtypes
df.head()
df.state_code.unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12711037742927891, 'fraction_numerical': 0.010805132437908007, 'mean_word_length': 4.33546837469976, 'pattern_counts': {'":': 1, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '540441', 'license': 'MIT', 'n_tokens_mistral': 4114, 'n_tokens_neox': 3902, 'n_words': 1377, 'path': 'notebooks/0.1.explore.data.ipynb', 'repo_name': 'aidinhass/kickstarter', 'size': '13327'}