<jupyter_start><jupyter_text>Project IntroductionThe goal of this project is to see whether we can predict a project's success on Kickstarter given information such as the project name, category, launch and deadline dates, and target funding goal. The raw data was obtained from <NAME> on Kaggle (https://www.kaggle.com/kemical/kickstarter-projects). I use the `ks-projects-201801.csv` version of the data.This notebook is organized into sections based on the main tasks including:* Data Cleaning* Exploratory Data Analysis (EDA)* Data Transforms* Feature Selection* Model Development* Hyperparameter Tuning* Ensemble Modeling* Neural Network ModelingKey Findings: in progress Import Modules and DataWe will use a fairly standard set of machine learning libraries in our analysis including:* numpy* pandas* matplotlib* seaborn* sklearn<jupyter_code>import os
from warnings import filterwarnings
from pickle import dump, load
filterwarnings('ignore', message='numpy.dtype size changed')
filterwarnings('ignore', message='Using a non-tuple sequence for multidimensional indexing is deprecated')
filterwarnings('ignore', message='numpy.core.umath_tests is an internal NumPy module and should not be imported')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler
from sklearn.feature_selection import RFE, SelectKBest, chi2
from sklearn.model_selection import train_test_split, KFold, cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.pipeline import Pipeline<jupyter_output>/home/addisonklinke/.local/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d<jupyter_text>Next we load the raw data downloaded from Kaggle<jupyter_code>os.chdir('../data/')
raw = pd.read_csv('kickstarter.csv', parse_dates=[5, 7])
raw.dtypes<jupyter_output><empty_output><jupyter_text>Data CleaningTo begin, let's familiarize ourselves with the dataset<jupyter_code>raw.describe()<jupyter_output><empty_output><jupyter_text>First, we notice that there are two columns for the amount of money pledged: `usd pledged` and `usd_pledged_real`. According to documentation for the dataset, the former was converted by Kickstarter and the later by the fixer.io API. Looking at rows 1, 41, 43, and others we can see that the amount in `pledged` does not match `usd pledged` even though the currency for the project is USD. Therefore, `usd_pledged_real` is the more reliable column to use.We will also drop the project ID since that is a random number which should have no correlation to the project's success. Lastly, the `goal` column is reported in original currency. Since we are converting all amounts to USD for consistency, this is not necessary.<jupyter_code>clean = raw.drop(columns=['usd pledged', 'ID', 'goal'])
clean.rename(columns={'usd_pledged_real':'pledged_usd', 'usd_goal_real':'goal_usd'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Next, let's look at our target variable `state` and see how it is distributed. The majority of projects are either failed of successful. Canceled is the next biggest category.<jupyter_code>clean.groupby('state').size().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>We can simplify this to a binary classification problem with failed or canceled as 0 and successful as 1. Additionally, since we are trying to predict a project's `state` when it launches, any information from the future should be removed.<jupyter_code>clean = clean[clean.state.isin(['successful', 'failed', 'canceled'])]
clean.state = clean.state.replace({'successful':1, 'failed':0, 'canceled':0})
clean.drop(columns=['pledged', 'backers', 'pledged_usd'], inplace=True)<jupyter_output><empty_output><jupyter_text>It would be interesting to know the duration of each project. First, `launched` needs to be split into its day and time components, and then we can subtract the day component from `deadline`. While we're manipulating `launched`, let's also turn `launch_time` into a numeric format (decimal hours on a 24 hour clock). Checking our work with a quick histogram of `duration`, it is suspicious that there are projects that lasted longer than 14,000 days (over 38 years)!<jupyter_code>clean['launch_day'] = [d.date() for d in clean.launched]
clean['launch_time'] = [d.time() for d in clean.launched]
clean[['h', 'm', 's']] = clean.launch_time.astype(str).str.split(':', expand=True).astype(float)
clean['launch_time'] = clean.h.values + clean.m.values / 60 + clean.s.values / 3600
clean.drop(columns=['h', 'm', 's'], inplace=True)
clean['duration'] = [(d - l).days for d, l in zip(clean.deadline.dt.date, clean.launch_day)]
clean.drop(columns='launched', inplace=True)

fig = plt.figure()
plt.hist(clean.duration)
plt.xlabel('Project Duration (Days)')
plt.ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Looking at the year from `launch_time`, there are some projects which supposedly started in 1970. However, Kickstarter didn't even exist back then so these are most likely data entry errors and we will remove them. Making a `launch_year` category and then querying the dataframe is an easy way to accomplish this. When the histogram for duration is re-plotted, it looks much more reasonable.<jupyter_code>print(np.unique([d.year for d in clean.launch_day]))
clean['launch_year'] = [d.year for d in clean.launch_day]
clean.query('launch_year != 1970', inplace=True)

fig = plt.figure()
plt.hist(clean.duration)
plt.xlabel('Project Duration (Days)')
plt.ylabel('Count')
plt.show()<jupyter_output>[1970 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018]<jupyter_text>To make separating our target variable (`state`) from the features, we will move it to the last column. Then the cleaned data is written to a CSV so we don't need to repeat these steps in the future.<jupyter_code>cols = list(clean.columns)
cols.append(cols.pop(cols.index('state')))
clean = clean.reindex(columns=cols)
clean.to_csv('kickstarter-clean.csv', index=False)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis (EDA)Let's start by seeing how projects are distributed by category. If you're just starting the notebook from this section, you'll want to read in `kickstarter-clean.csv` (the CSV we saved after data cleaning).<jupyter_code>clean = pd.read_csv('kickstarter-clean.csv', parse_dates=[4, 7])
main_categories = clean.groupby('main_category').size().sort_values(ascending=False)
n_main = [i / 1000 for i in list(main_categories)]
labels_main = list(main_categories.index)

ind = np.arange(len(main_categories))
plt.bar(ind, n_main)
plt.ylabel('Count (Thousands)')
plt.xlabel('Main Category')
plt.title('Kickstarter Project Categories')
plt.xticks(ind, labels_main, rotation=90)
plt.show()<jupyter_output><empty_output><jupyter_text>How does sucess vary with factors like duration, goal, and time of day the project was launched?<jupyter_code>plt.figure(figsize=(10, 6))
sns.distplot(clean.query('state == 1').duration, hist=False, kde=True, kde_kws={'shade': True, 'linewidth': 3}, 
             label='Success', color='green')
sns.distplot(clean.query('state == 0').duration, hist=False, kde=True, kde_kws={'shade': True, 'linewidth': 3}, 
             label='Failure', color='red') 
plt.legend(title='Project Status')
plt.xlabel('Project Duration (Days)')
plt.ylabel('Density')
plt.xlim((0, 65))
plt.show()<jupyter_output><empty_output><jupyter_text>The vast majority of projects are 30 days long regardless of their status. The next largest peaks are at 60 and 45 days, indicating that projects tend to be scheduled in terms of months (1, 1.5, 2, etc). There is a slight trendtowards shorter durations leading to an increased chance of success (more 15-25 day project succeeded).Now we can look at the project funding goal (converted to USD for a fair comparison across all the currencies). Plotting on a log10 scale helps compress the data into a viewable range since goals range from \$0.01 up to over \$166,000,000<jupyter_code>sns.distplot(np.log10(clean.query('state == 1').goal_usd.values), hist=False, kde=True, 
             kde_kws={'shade': True, 'linewidth': 3}, label='Success', color='green')
sns.distplot(np.log10(clean.query('state == 0').goal_usd.values), hist=False, kde=True, 
             kde_kws={'shade': True, 'linewidth': 3}, label='Failure', color='red') 
plt.legend(title='Project Status')
plt.xlabel('Project Funding Goal (log10 USD)')
plt.ylabel('Density')
plt.show()<jupyter_output><empty_output><jupyter_text>The distinction between failure and sucess is a more pronounced here than for duration - projects with a larger funding goal failed more often. The most common goals were around \$10k USD.What about time of day the project was launched? If people are most likely to hear about a project at its inception, it would be critical to launch at a time when people are available (for instance, weekday evenings).<jupyter_code>sns.distplot(clean.query('state == 1').launch_time, hist=False, kde=True, kde_kws={'shade': True, 'linewidth': 3}, 
             label='Success', color='green')
sns.distplot(clean.query('state == 0').launch_time, hist=False, kde=True, kde_kws={'shade': True, 'linewidth': 3}, 
             label='Failure', color='red') 
plt.legend(title='Project Status')
plt.xlabel('Project Launch Hour (Military Time)')
plt.ylabel('Density')
plt.show()<jupyter_output><empty_output><jupyter_text>Failed projects look like they may launch more often at the extreme start/end of days, but the trend is not super strong. Another angle to explore the data from in the future is using Natural Language Processing (NLP). We could see whether there are words or phrases in the project names that correspond well with the project's success.  Data TransformsFor modeling purposes, we will need a numeric representation of all the columns containing datetimes (`deadline`, `launch_day`, and `launch_time`) and categorical strings (`name`, `category`, `main_category`, `currency`, and `country`). Let's start with representing the datetime columns numerically. As we learned above, there are few projects lasting longer than 60 days, so the year information in `deadline` and `launch_day` should be similar. The remaining month and day information can be encoded as day of year (i.e. 1 to 365). Adding a cosine transform is useful here since it indicates to the model that days 1 and 365 are actually more similar than days 1 and 300. Similarly, we can convert `launch_time` into a decimal hour representation on a 24 hour clock and apply another cosine transform.<jupyter_code>clean = pd.read_csv('kickstarter-clean.csv', parse_dates=[4, 7])
transformed = clean
transformed.launch_day = [np.cos(d.timetuple().tm_yday / 365 * 2 * np.pi) for d in clean.launch_day] 
transformed.deadline = [np.cos(d.timetuple().tm_yday / 365 * 2 * np.pi) for d in clean.deadline]
transformed.launch_time = np.cos(clean.launch_time.astype('float') / 24 * 2 * np.pi)<jupyter_output><empty_output><jupyter_text>Next we'll deal with the categorical string columns. Since all monetary amounts have already been converted to USD, there is no need for the `currency` column in modeling. Until we do further NLP research, the best way to encode the project name will be its length. The remaining categorical variables can be one-hot encoded using the `get_dummies()` function from pandas. After performing the transforms, we print the column `dtypes` to double check that all variables are numerical now.<jupyter_code>transformed.drop(columns='currency', inplace=True)
transformed.name = clean.name.str.len()
transformed.dropna(inplace=True) # Some strings returned NA
transformed = pd.get_dummies(transformed)
transformed.dtypes<jupyter_output><empty_output><jupyter_text>Now that all variables are numerical, we can check for any pairs that are highly correlated (i.e. $R > 0.8$) since those will adversely affect modeling. Unsurprisingly, `launch_day` and `deadline` are highly correlated. Let's keep `launch_day` over `deadline` since it's more closely related to the start of a project. There are also a few `category` / `main_category` combinations which appear to contain duplicate information - for instance theater and design. We'll keep the `main_category` version of each of those since that should apply to more rows of the dataframe. Note this does not mean we are removing theater or design projects from the data, but rather not giving our model access to that label information.<jupyter_code>print(transformed.corr().unstack().sort_values().drop_duplicates())
transformed.drop(columns=['deadline', 'category_Theater', 'category_Product Design'], inplace=True)<jupyter_output><empty_output><jupyter_text>Once again, we'll make sure the `state` column is last in the dataframe to make it easier to extract the features.<jupyter_code>cols = list(transformed.columns)
cols.append(cols.pop(cols.index('state')))
transformed = transformed.reindex(columns=cols)<jupyter_output><empty_output><jupyter_text>Feature SelectionSince all our cleaning and data pre-processing was saved out to `kickstarter-transformed.csv`, we'll read that in first. Then, the pandas dataframe needs to be converted to numpy arrarys `x` and `y` to work with the modeling functions in sklearn. These are then split into 80% training and 20% testing. We'll run our intial models on the training data *only* and save the test set as a final validation.<jupyter_code>seed = 1234
y = transformed.state.values
x = transformed.values[:, :-1]
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=seed)
print(x.shape)<jupyter_output>(370445, 201)<jupyter_text>There are just over 200 features in our dataset! Chances are not all of them will be useful for predicting a project's chance of success, so we'll the `sklearn.feature_selection` module to tease out which features show the most promise. To start, let's look at feature importance values from a Random Forest model.<jupyter_code>selector = RandomForestClassifier(random_state=seed)
selector.fit(x_train, y_train)
importance = selector.feature_importances_
importance_relative = importance / max(importance) * 100
indices = np.argsort(importance_relative)

plt.barh(range(len(indices)), importance_relative[indices], align='center')
plt.xlabel('Relative Importance (%)')
plt.ylabel('Feature Index')
plt.show()<jupyter_output><empty_output><jupyter_text>As expected, there are about 10 to 25 features which have substantially more importance than the rest. Let's re-plot with only the top 10 and add y-axis labels to see what those are.<jupyter_code>features = transformed.columns
top = 10

plt.barh(range(len(indices[-top:])), importance_relative[indices[-top:]], align='center')
plt.xlabel('Relative Importance (%)')
plt.yticks(range(len(indices[-top:])), [features[i] for i in indices[-top:]])
plt.show()<jupyter_output><empty_output><jupyter_text>The results from this look quite definitive and agrees with our observations during EDA about `goal_usd`. Going forward we will retain the top 6 variables. Now we are ready to start testing different models.<jupyter_code>x = x[:, indices[-6:]]
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=seed)
print(x.shape)<jupyter_output>(370445, 6)<jupyter_text>Model DevelopmentAs a baseline to compare our other models to, let's consider what accuracy a classifier could achieve if it always predicted the majority class.<jupyter_code>classes = transformed.groupby('state').size().sort_values(ascending=True) 
classes[0] / sum(classes)<jupyter_output><empty_output><jupyter_text>Due to the slight class imbalance, we can expect 63.8% accuracy just by predicting `state = 0` (failed) every time. Let's compare this benchmark to several different machine learning algorithms. These will include 2 linear algorithms (Logistic Regression and Linear Discriminant Analysis) and 3 non-linear algorithms (K Nearest Neighbors, Decision Trees, and Naive Bayes). 10-fold cross validation provides a good estimate of model performance on new data. To ensure the results are reproducible, we'll use the seed we set manually above.<jupyter_code>models = [('LR', LogisticRegression()), ('LDA', LinearDiscriminantAnalysis()),
          ('KNN', KNeighborsClassifier()), ('DT', DecisionTreeClassifier()), ('NB', GaussianNB())]
names = [n for n, m in models]
ctrl = KFold(n_splits=10, random_state=seed)
results = [cross_val_score(m, x_train, y_train, cv=ctrl) for n, m in models]

fig = plt.figure()
ax = fig.add_subplot(111)
plt.boxplot(results)
plt.title('Performance Comparison')
plt.xlabel('Algorithm Abbreviation')
plt.ylabel('Cross Validation Accuracy')
ax.set_xticklabels(names)
plt.show()<jupyter_output><empty_output><jupyter_text>Before making any final decisions, let's try adding a scaling pre-processing step to the data and see how it affects performance. To implement this in sklearn, we'll need to define a pipeline for each model.<jupyter_code>pipelines = [('ScaledLR',  Pipeline([('Scaler', StandardScaler()), ('LR', LogisticRegression())])), 
             ('ScaledLDA', Pipeline([('Scaler', StandardScaler()), ('LDA', LinearDiscriminantAnalysis())])), 
             ('ScaledKNN', Pipeline([('Scaler', StandardScaler()), ('KNN', KNeighborsClassifier())])), 
             ('ScaledDT',  Pipeline([('Scaler', StandardScaler()), ('DT', DecisionTreeClassifier())])),
             ('ScaledNB',  Pipeline([('Scaler', StandardScaler()), ('NB', GaussianNB())]))]

names = [n for n, p in pipelines]
ctrl = KFold(n_splits=10, random_state=seed)
results = [cross_val_score(p, x_train, y_train, cv=ctrl) for n, p in pipelines]

fig = plt.figure()
ax = fig.add_subplot(111)
plt.boxplot(results)
plt.title('Scaled Performance Comparison')
plt.xlabel('Algorithm Abbreviation')
plt.ylabel('Cross Validation Accuracy')
ax.set_xticklabels(names)
plt.show()<jupyter_output><empty_output><jupyter_text>Unfortunately even after scaling the data, none of the candidate algorithms are performing significantly better than our benchmark. This may indicate the need for a more complex model, so we'll try a couple different ensemble methods next to see if there is any improvement. If those don't show any promise, we can always come back to the above algorithms and see if we can boost their performance either by optimizing hyperparameters or using some form of regularization.  Ensemble ModelingWe'll start with the default parameters of 3 different ensemble methods: Random Forest, AdaBoost, and Gradient Boosting Machines. Scaling looks like it hurt the performance of our other algorithms, so we'll start training the ensembles on the unscaled data.<jupyter_code>ensembles = [('RF', RandomForestClassifier()), 
             ('Ada', AdaBoostClassifier()), 
             ('Grad', GradientBoostingClassifier())]
names = [n for n, e in ensembles]
results = [cross_val_score(e, x_train, y_train, cv=ctrl) for n, e in ensembles]
fig = plt.figure()
ax = fig.add_subplot(111)
plt.boxplot(results)
plt.title('Ensemble Comparison')
plt.xlabel('Ensemble Abbreviation')
plt.ylabel('Cross Validation Accuracy')
ax.set_xticklabels(names)
plt.show()
print('Gradient Boosting accuracy: {:.4f}%'.format(results[2].mean() * 100))<jupyter_output>Avg. Gradient Boosting accuracy: 67.5077%<jupyter_text>This is nothing fantastic, but it's also the first time we've seen an algorithm consistently perform better than our benchmark! Let's check whether scaling would help Gradient Boosting learn better.<jupyter_code>grad_scaled = Pipeline([('Scaler', StandardScaler()), ('Grad', GradientBoostingClassifier())])
results = cross_val_score(grad_scaled, x_train, y_train, cv=ctrl)
print('Scaled Gradient Boosting accuracy: {:.4f}%'.format(results.mean() * 100))<jupyter_output>Scaled Gradient Boosting accuracy: 67.4935%<jupyter_text>Hyperparameter TuningScaling the input features did not help Gradient Boosting, so let's see if we can optimize some of the algorithm's hyperparameters instead. Some options we may want to consider are:* `n_estimators`: number of boosting iterations to perform* `max_depth`: number of splits each individual estimator tree is allowed to have.* `learning_rate`: controls the contribution of each individual estimator tree* `max_features`: fraction of features to consider when deciding a split* `subsample`: fraction of samples used to fit each individual estimator treeTo start, we'll look at `n_estimators` since it might be the case that our ensemble didn't have enough iterations to learn. From there we can further optimize the other parameters.<jupyter_code>param_grid = dict(n_estimators = np.arange(50, 500, 50))
ctrl = KFold(n_splits=3, random_state=seed)
grid = GridSearchCV(GradientBoostingClassifier(), param_grid, cv=ctrl)
estimators_result = grid.fit(x_train, y_train)<jupyter_output><empty_output><jupyter_text>Since the grid search takes so long to run, we'll quickly save a pickled version of the results in case we need to access them again later.<jupyter_code>dump(estimators_result, open('kickstarter-n_estimators.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>When plotting accuracy as a function of `n_estimators`, there is a clear trend. However, the minor gain in performance is definitely not worth the increased computational cost.<jupyter_code>plt.errorbar(x=np.arange(50, 500, 50), 
             y=estimators_result.cv_results_['mean_test_score'], 
             yerr=estimators_result.cv_results_['std_test_score'], 
             capsize=4)
plt.xlabel('Number of Estimators')
plt.ylabel('Average Test Accuracy')
plt.show()
grid = GridSearchCV(GradientBoostingClassifier(), dict(max_depth=np.arange(1, 10, 1)), cv=ctrl)
depth_result = grid.fit(x_train, y_train)
dump(depth_result, open('kickstarter-max_depth.pkl', 'wb'))
plt.errorbar(x=np.arange(1, 10, 1), 
             y=depth_result.cv_results_['mean_test_score'], 
             yerr=depth_result.cv_results_['std_test_score'], 
             capsize=4)
plt.xlabel('Max Estimator Depth')
plt.ylabel('Average Test Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>Neural Network Modeling<jupyter_code>?GradientBoostingClassifier<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10174992324898031, 'fraction_numerical': 0.013069602210429368, 'mean_word_length': 6.1345431789737175, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 117, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '660738', 'license': 'MIT', 'n_tokens_mistral': 7146, 'n_tokens_neox': 6528, 'n_words': 2404, 'path': 'projects/kickstarter-main.ipynb', 'repo_name': 'addisonklinke/portfolio', 'size': '22810'}