<jupyter_start><jupyter_text>Kickstarter - Meme Master Memoryby <NAME>, <NAME> and <NAME>  Business case: Mr. Brain invented a Meme Master Memory (a memory game with videos/gifs) and needs money for the implementation. He wants to do the project in 4 weeks, wants to get 100,000USD and wants to launch the campaign in summertime in the category photography.Under these circumstances, will the project be successful? Furthermore we want to recommend Mr. Brain different improvements to increase his chance for success. In addition we also want to predict his potential pledged amount.   Approach:* Business Understanding:    * Kickstarter is a Crowdfunding website that enables entrepreneurs to fund money for their business ideas.    * With regard to our business case we are looking for the features that might be relavant for the success of the project, especially with regard to the characteristics of our stakeholders project (e.g., product category, season)     * Evaluation metrics: It is in our stakeholders interest that we are confident in our prediction of success and that we avoid the case that our stakeholder puts a lot of effort in a project that is likely to fail. In other words we want to avoid a false positive prediction and that is why we choose precision as our main evlaution metric.* Data Collection     * For our data analysis and modeling we use a well known kaggle dataset. Our data science bootcamp teachers gave us the dataset but similar datasets can be found online.    * An overview of all the columns in the dataset can be found below.* EDA:    * see Jupyter Notebook "Kickstarter_EDA" for an EDA * Feature Engineering / Data Cleaning:    * see Jupyter Notebook "Kickstarter" for data preparation, cleaning and feature engineering * Predictive Modeling    * We will use a simple logistic regression, SVM and random forest to predict the success of the project. The classification models can be found in the Jupyter Notebook "Kickstarter"    * For the prediciton of the total amount that the stakeholder could pledge we use a polynomial regression, random forest and Adaboost. This is still work in progess. First models do not show a reasonable fit and further analyses might be needed.* Stakeholder Presentation    Data cleaningIn this notebook we will clean our Dataframe 'Kickstarter_merged.csv'. That means we will create new columns which we think are important as well as we will drop columns with unimportance. Overview new columns    * column 'blurbs' will be replaced with -> 'blurb_len_w'    * column 'slug' will be trplaced with -> 'slug_len_w',     * column 'category' will be replaced with -> 'parent_name'    * column 'launched_at' will be replaced with -> 'launched_month'    * new column duration of the crowdfunding    * new column preparation time (from created_at until launched_at    * column 'state_changed_at' will be replaced with -> 'state_changed_year', 'state_changed_month,     * new column pledged/backer    * column 'goal' will be converted in USD Overview dropped columns    * converted_pledged_amount    * creator    * currency     * currency_symbol    * currency_trailing_code    * current_curency    * disable_communication    * friends    * fx_rate    * id after using it for other transformations    * is_backing    * is_starrable    * is_starred    * location    * name     * permissions    * photo    * pledged    * profile     * slug    * source_url    * spotlight    * state_changed_at     * static_usd_rate     * urls    * usd_type            Overview dropped rows    * 8 rows with missing values in column 'blurbs'    * drop the duplicates    * drop rows with values 'suspended' and 'live' in column 'state'<jupyter_code># import packages
import pandas as pd
import numpy as np
import seaborn as sns
import time
import datetime as dt
import json
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn import svm
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.ensemble import AdaBoostRegressor
from sklearn.datasets import make_regression


import pylab as pl
import scipy.optimize as opt
from sklearn import preprocessing
import itertools
import warnings 
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Read in file.<jupyter_code>df = pd.read_csv('data/Kickstarter_merged.csv', index_col=0)<jupyter_output><empty_output><jupyter_text>Create new columns    Let´s start with creating new columns:  Length of Blurb in words:The blurb is the description of a kickstarter project. We assume that the length of the project description is an indicator of how well a project is described and presented and that a good presentation is relevant for success.<jupyter_code>def string_len_w(string):
    '''Return length of string (number of word, seperated by Space).'''
    string_str = str(string)
    string_list = string_str.split()
    string_len = len(string_list)
    return string_len
def add_blurb_len_w (df):
    '''Adding column that contains the length of the Blurb (words) and returns the updated Dataframe'''
    df['blurb_len_w'] = df.apply(lambda x: string_len_w(x['blurb']), axis=1)
    return df
df = add_blurb_len_w(df)<jupyter_output><empty_output><jupyter_text>Length of Slug in words:The slug is the addition in the URL and is equal to the project name. We think that a longer name is a better description of the project and more relevant keywords are easier to be found by web crawler (relevant for Search Engine Optimization). Therefore we think that longer slugs are associated with success.<jupyter_code>def string_len_slug_w(string):
    '''Returns length of string (number of words, seperated by "-").'''
    string_str = str(string)
    string_list = string_str.split("-")
    string_len = len(string_list)
    return string_len
def add_slug_len_w (df):
    '''Adding column that contains the length of the Slug (words) and returns the updated Dataframe'''
    df['slug_len_w'] = df.apply(lambda x: string_len_slug_w(x['slug']), axis=1)
    return df
df = add_slug_len_w(df)<jupyter_output><empty_output><jupyter_text>Category aka parent_name In this section we extract the category name out of information saved as a json type. We think certain categories (e.g. games) are more successfull than others (e.g. music).<jupyter_code>def add_parent_id(df):
    '''Extracts Parent ID out of the Category json and adds the Column to Dataframe. Returns updated Dataframe'''
    df['category_parent_id'] = pd.DataFrame([json.loads(df["category"][i]).get("parent_id") for i in range(df.shape[0])])
    return df
def add_category_id(df):
    '''Extracts category ID out of the Category json and adds the Column to Dataframe. Returns updated Dataframe'''
    df['category_id'] = pd.DataFrame([json.loads(df["category"][i]).get("id") for i in range(df.shape[0])])
    return df
def add_category_name(df):
    '''Extracts category name out of the Category json and adds the Column to Dataframe. Returns updated Dataframe'''
    df['category_name'] = pd.DataFrame([json.loads(df["category"][i]).get("name") for i in range(df.shape[0])])
    return df
def fill_na(df, column_name):
    '''Fill Missings with 0 as type integer. Returns updated dataframe. eg, for parent ID and pledged per backer'''
    df[column_name] = df[column_name].fillna(0).astype("int")
    return df
# Making a list based on entry in one category and if missing adds entry of another Column
def helper_list():
    '''Making a list based on entry in one category and if missing adds entry of another Column'''
    empty = []
    for i in range(df.shape[0]):
        if df["category_parent_id"][i] != 0:
            empty.append(df["category_parent_id"][i])
        else:
            empty.append(df["category_id"][i])
    return empty
# adds helper list as column to dataframe 
def add_list_as_column(df, column_name, list_name):
    '''Adds helper list as column to dataframe and retruns updated dataframe'''
    df[column_name] = pd.DataFrame(list_name)
    return df
def add_parent_name(df, column_name1, column_name2, dictionary):
    '''based on key value in a column, column with value is added as a column and updated dataframe is returned. 
    Example:
        parents_dict = {1: "Art", 3: "Comics", 6: "Dance", 7: "Design", 9: "Fashion", 10: "Food",
                11: "Film & Video", 12: "Games", 13: "Journalism", 14: "Music", 15: "Photography", 16: "Technology",
               17: "Theater", 18: "Publishing", 26: "Crafts"}
            df["parent_name"] = df["filled_parent"].apply(lambda x: parents_dict.get(x))'''
    df[column_name1] = df[column_name2].apply(lambda x: dictionary.get(x))
    return df
df = add_parent_id(df)
df = add_category_id(df)
df = add_category_name(df)
df = fill_na(df, 'category_parent_id')
empty = []
for i in range(df.shape[0]):
    if df["category_parent_id"][i] != 0:
        empty.append(df["category_parent_id"][i])
    else:
        empty.append(df["category_id"][i])
df = add_list_as_column(df, "filled_parent", empty)
df = add_parent_name(df, "parent_name", "filled_parent", {1: "Art", 3: "Comics", 6: "Dance", 7: "Design", 9: "Fashion", 10: "Food",
                11: "Film & Video", 12: "Games", 13: "Journalism", 14: "Music", 15: "Photography", 16: "Technology",
               17: "Theater", 18: "Publishing", 26: "Crafts"})
df["parent_name"]<jupyter_output><empty_output><jupyter_text>Month launchedIn the dataset we have a timestamp for the moment the project was created, launched and ended (deadline). We think that the month/season a project is launched influences its success.<jupyter_code>#funtion to extract the month out of the number
def extract_month(number):
    '''Extracts the month out of the number and returns the month'''
    gmtime = time.gmtime(number)
    return gmtime[1]
# Adding column with month the project was launched
def adding_month_launched(df):  
    '''Adding column with month the project was launched and returns the updated dataframe'''
    df["launched_month"] = df.apply(lambda x: extract_month(x["launched_at"]), axis=1)
    return df
df = adding_month_launched(df)<jupyter_output><empty_output><jupyter_text>DurationThe duration is the time period between the moment it was launched and the deadline. We created a feature by substracting the corresponding timestamps. We assume that the duration of a project influences its success.<jupyter_code>def duration(deadline, launched_at):
    '''Calculating difference between two timepoints and returns it in days'''
    duration = deadline - launched_at
    duration_complete = dt.timedelta(seconds=duration)
    return duration_complete.days
# Adding column with duration in days
def adding_duration(df):
    '''Adding column with duration in days and returns updated dataframe'''
    df["duration_days"] = df.apply(lambda x: duration(x["deadline"], x["launched_at"]), axis=1)
    return df
df = adding_duration(df)<jupyter_output><empty_output><jupyter_text>Preparation Similar to duration, the preparation is the period between the creation and the moment the poject is launched. Again we assume that a longer preparation might be relevant for success.<jupyter_code>def adding_preparation(df):
    '''Adding column with preparation in days and returns updated dataframe'''
    df["preparation"] = df.apply(lambda x: duration(x["launched_at"], x["created_at"]), axis=1)
    return df
df = adding_preparation(df)<jupyter_output><empty_output><jupyter_text>pledged/backer as "Reward Amount"The project owner can set a reward for a certain amount that is pledged. As an indicator of this amount we used the average amount pledged per backer. The reward size might influence how fast the goal is reached and consequently for the success of a project.<jupyter_code>def adding_pledged_per_backer(df):
    '''Adding column that is the averaged amount pledged per backer, returns updated dataframe'''
    df['pledged_per_backer'] = (df['usd_pledged'] / df['backers_count']).round(2)
    return df
df = adding_pledged_per_backer(df)<jupyter_output><empty_output><jupyter_text>There are cases in our dataset with no backers. Because we cannot devide by zeros we have to convert the resulted missing values to 0.<jupyter_code>df.pledged_per_backer = df.pledged_per_backer.fillna(0).astype("int")<jupyter_output><empty_output><jupyter_text>Coverting Goal to USDTo garantee comparability of different currencies we convert the goal to USD. The amount of the goal is directly related to the success. The lower the goal is set, the more likely it is to be reached.<jupyter_code>def usd_convert_goal(df, column_name, exchange_rate): 
    '''Converts a Column based on given exchange rate, rounds it to two decimal spaces  
    and returns updated dataframe, e.g. 
    df['goal'] = (df['goal'] * df['static_usd_rate']).round(2)'''
    df[column_name] = (df[column_name] * df[exchange_rate]).round(2)
    return df
df = usd_convert_goal(df, 'goal', 'static_usd_rate')
df['goal']<jupyter_output><empty_output><jupyter_text>Drop rowsWe drop few rows that have missing values in the column "blurbs" and duplicates. We checked for differences in the duplicated rows. The only column that differs, is the type of usd_type (domestic vs. international Dollars). Out of the duplicates we kept the latest because we assume that this would be the latest version.Futhermore, we drop one project that has a goal of 0 Dollar because this is not realistic.Because we want to predict the success of a project we decided to focus on the projects that have a certain state of success or failure. We therefore drop rows with the state "live", "canceled", "suspended".<jupyter_code>def drop_rows_missings(df, column_name):
    '''Drop rows with missing values in column, eg. Blurb. Retruns dataframe.'''
    df.dropna(subset = [column_name], inplace=True)
    return df
def drop_duplicates(df, column_name):
    '''Creating dataframe and dropping all duplicates, based on a column_name (eg, ID) 
    and keep the last ("newest") duplicate'''
    df = df.drop_duplicates(subset=['id'], keep='last')
    return df
# drop rows with values certain values in a dataframe and returns updated dataframe, eg 'suspended' and 'live' in column 'state'
def drop_rows_value (df, column_name, value):
    '''drop rows with values certain values in a dataframe and returns updated dataframe'''
    df = df.drop(df[df[column_name] == value ].index)
    return df
# drop 8 rows with missing values in column 'blurbs'
df = drop_rows_missings(df, 'blurb')
# creating dataframe and dropping all duplicates and keep the last ("newest") duplicate 
df = drop_duplicates(df, 'id')
df = drop_rows_value(df, 'state', 'suspended')
df = drop_rows_value(df, 'state', 'live')
df = drop_rows_value(df, 'state', 'canceled')
df = drop_rows_value(df, 'goal', 0)<jupyter_output><empty_output><jupyter_text>Drop Columns Now, we drop all columns that we assume not to be relevant.<jupyter_code>def drop_columns(df, list_columns):
    '''Drops columns in the list and returns updated datadrame'''
    df.drop(list_columns, axis=1, inplace=True)
    return df
df.columns
df = drop_columns(df, ['backers_count', 'blurb', 'category', 'converted_pledged_amount',
       'country', 'created_at', 'creator', 'currency', 'currency_symbol',
       'currency_trailing_code', 'current_currency', 'deadline',
       'disable_communication', 'friends', 'fx_rate', 'id',
       'is_backing', 'is_starrable', 'is_starred', 'launched_at', 'location',
       'name', 'permissions', 'photo', 'pledged', 'profile', 'slug',
       'source_url', 'spotlight', 'state_changed_at',
       'static_usd_rate', 'urls', 'usd_type', 'category_parent_id', 'category_id', 'category_name',
       'filled_parent', 'staff_pick'])<jupyter_output><empty_output><jupyter_text>Drop Rows and only keep relevant categories Our stakeholder's project would only fit into one of the following categories: "Games", "Art", "Photography", "Film & Video", "Design", "Technology". Therefore we will drop all rows with other categories.<jupyter_code>categories = ["Games", "Art", "Photography", "Film & Video", "Design", "Technology"]
df = df[df.parent_name.isin(categories)]<jupyter_output><empty_output><jupyter_text>Get DummiesFor modeling we convert the categories into dummies. Because we want to include the season in our model we converted the month_launched into 4 seasons and make dummies out of that new column. The old column we drop.<jupyter_code># convert the categorical variable parent_name into dummy/indicator variables
df_dum2 = pd.get_dummies(df.parent_name, prefix='parent_name')
df = df.drop(['parent_name'], axis=1)
df = pd.concat([df, df_dum2], axis=1)
# making a categorical variable for launched_month q1, q2, q3, q4 
df.loc[df['launched_month'] <  4, 'time_yr'] = 'q1'
df.loc[(df['launched_month'] >=  4) & (df['launched_month'] <  7), 'time_yr'] = 'q2'
df.loc[(df['launched_month'] >=  7) & (df['launched_month'] <  10), 'time_yr'] = 'q3'
df.loc[df['launched_month'] >  9, 'time_yr'] = 'q4'
df_dum3 = pd.get_dummies(df.time_yr, prefix='time_yr')
df = df.drop(['time_yr'], axis=1)
df = df.drop(['launched_month'], axis=1)
df = pd.concat([df, df_dum3], axis=1)<jupyter_output><empty_output><jupyter_text>Reset index<jupyter_code>df = df.reset_index()
df = df.drop(['index'], axis = 1)
df.head()<jupyter_output><empty_output><jupyter_text>Classification As a next step, we want to predict our stakeholder's success chances. To do so, we train a classification model based on our data. Because we want to use the current dataset for classification and regression, we rename/duplicate the dataframe at this point.<jupyter_code>df_c = df.copy()<jupyter_output><empty_output><jupyter_text>Our target value for the prediction is the column "state". We convert this column into binary integers.<jupyter_code>df_c['state'] = np.where(df_c['state'] == 'successful', 1, 0)<jupyter_output><empty_output><jupyter_text>For classification we don't need usd_pledged, because we would not know this in advance. 'usd_pledged' will be our target variable in the regression model.<jupyter_code>df_c = drop_columns(df_c, ['usd_pledged'])<jupyter_output><empty_output><jupyter_text>Training the Model Train-Test-SplitBefore we fit our model we have to split our data into test and train data.<jupyter_code>y = df_c.state
X = df_c.drop('state', axis=1)
# Train-test-split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# we have to define which columns we want to scale.
col_scale = ['goal', 'blurb_len_w', 'slug_len_w', 'duration_days', 'preparation', 'pledged_per_backer']<jupyter_output><empty_output><jupyter_text>Data standardizationTo bring all our numeric features on the same scale we standardize them.<jupyter_code># we have to define which columns we want to scale.
col_scale = ['goal', 'blurb_len_w', 'slug_len_w', 'duration_days', 'preparation', 'pledged_per_backer']
# Scaling with standard scaler
scaler = StandardScaler()
X_train_scaled_st = scaler.fit_transform(X_train[col_scale])
X_test_scaled_st = scaler.transform(X_test[col_scale])
# Concatenating scaled and dummy columns 
X_train_preprocessed_st = np.concatenate([X_train_scaled_st, X_train.drop(col_scale, axis=1)], axis=1)
X_test_preprocessed_st = np.concatenate([X_test_scaled_st, X_test.drop(col_scale, axis=1)], axis=1)
X_train<jupyter_output><empty_output><jupyter_text>Logistic RegressionWe start with a simple logistic regression. We fit the model and print the confusion matrix and the classification report.<jupyter_code>def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# fit model
lr = LogisticRegression()
lr.fit(X_train_preprocessed_st, y_train)
y_pred = lr.predict(X_test_preprocessed_st)
cnf_matrix = confusion_matrix(y_test, y_pred)
print(classification_report(y_test, y_pred))

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['failed','successful'],normalize= False,  title='Confusion matrix')<jupyter_output>precision    recall  f1-score   support

           0       0.73      0.61      0.66      7732
           1       0.70      0.80      0.74      8676

    accuracy                           0.71     16408
   macro avg       0.71      0.70      0.70     16408
weighted avg       0.71      0.71      0.71     16408

Confusion matrix, without normalization
[[4723 3009]
 [1767 6909]]<jupyter_text>Conclusion* For the evaluation of the model we look at the precision of the model. In this case we have a precision of 70 % (6909 true positive/(6909 true positive + 3009 false positive)* That means that 70% of all the projects that are predicted to be successful, in fact have a successful out* In the following we will use an ensemble algorithm to check if we can improve the precision of the model  Random Forrest<jupyter_code># Create the model with 100 trees
model = RandomForestClassifier(n_estimators=100, 
                               random_state=42, 
                               max_features = 'sqrt',
                               n_jobs=-1, verbose = 1)
# Fit on training data
model.fit(X_train_preprocessed_st, y_train)
y_pred = model.predict(X_test_preprocessed_st)
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, y_pred, labels=[0,1])
np.set_printoptions(precision=2)

print (classification_report(y_test, y_pred))

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['failed','successful'],normalize= False,  title='Confusion matrix')<jupyter_output>precision    recall  f1-score   support

           0       0.85      0.73      0.79      7732
           1       0.79      0.89      0.83      8676

    accuracy                           0.81     16408
   macro avg       0.82      0.81      0.81     16408
weighted avg       0.82      0.81      0.81     16408

Confusion matrix, without normalization
[[5624 2108]
 [ 963 7713]]<jupyter_text>* Compared to the logistic regression model, the precision improved to 79%.* Next we try to improve the random forest model by using a gridsearch tune the hyperparameters  Hyperparameter Tuning / Gridsearch<jupyter_code># Hyperparameter grid
param_grid = {
    'n_estimators': np.linspace(10, 200).astype(int),
    'max_depth': [None] + list(np.linspace(3, 20).astype(int)),
    'max_features': ['auto', 'sqrt', None] + list(np.arange(0.5, 1, 0.1)),
    'max_leaf_nodes': [None] + list(np.linspace(10, 50, 500).astype(int)),
    'min_samples_split': [2, 5, 10],
    'bootstrap': [True, False]
}
# Estimator for use in random search
estimator = RandomForestClassifier(random_state = 42)
# Create the random search model
rs = RandomizedSearchCV(estimator, param_grid, n_jobs = -1, 
                        scoring = 'roc_auc', cv = 3, 
                        n_iter = 10, verbose = 5, random_state=42)
# Fit 
rs.fit(X_train_preprocessed_st, y_train)
rs.best_params_
# Create the model with 100 trees
model = RandomForestClassifier(n_estimators=196, 
                               random_state=42, 
                               min_samples_split=10,
                               max_leaf_nodes=49,
                               max_features=0.7,
                               max_depth=17,
                               bootstrap=True,
                               n_jobs=-1, verbose = 1)
# Fit on training data
model.fit(X_train_preprocessed_st, y_train)
y_pred = model.predict(X_test_preprocessed_st)
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, y_pred, labels=[0,1])
np.set_printoptions(precision=2)
print (classification_report(y_test, y_pred))
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['failed','successful'],normalize= False,  title='Confusion matrix')<jupyter_output>precision    recall  f1-score   support

           0       0.88      0.69      0.77      7732
           1       0.77      0.92      0.84      8676

    accuracy                           0.81     16408
   macro avg       0.83      0.80      0.80     16408
weighted avg       0.82      0.81      0.81     16408

Confusion matrix, without normalization
[[5324 2408]
 [ 710 7966]]<jupyter_text>Conclusion* When we used the best hyperparameters found in our gridsearch our model shows a higher recall (92%). That means that out of 100 successful projects 92 are correctly classified.* But: the precision of the model decreased to 77%. For us this is the main focus because we do not want avoid that our clients project is classified to be successful but fails in the end. Therefore we will use our initial parameters for the prediciton.   Regression* As an additional service for our stakeholder we want to predict how much money he can pledge if his project is successful* To do so we will use a regression model* But first, we will have to do some modification to our dataset. We will drop all unsuccessful projects because projects only receive money in the case of success (pledged amount is higher than the goal).<jupyter_code># creaste new dataframe for regression
df_r = df.copy()<jupyter_output><empty_output><jupyter_text>Our target value for the prediction is the column "usd_pledged".<jupyter_code># drop unsuccessful projects
df_r = df_r.drop(df_r[df_r['state'] == "failed" ].index)<jupyter_output><empty_output><jupyter_text>For regression we don't need the column "state" anymore. So we drop the column.<jupyter_code>df_r = drop_columns(df_r, ['state'])<jupyter_output><empty_output><jupyter_text>Training the Model Train-Test-SplitBefore we fit our model we have to split our data into test and train data.<jupyter_code>y = df_r['usd_pledged']
X = df_r.drop('usd_pledged', axis=1)
# Train-test-split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Remove outlierBecause our client does not want to pledge a very small amount and also no very high amount, we will remove the extreme "outlier"<jupyter_code># we concatenate the X_train and y_train sets in order to remove the rows from both sets.
new = pd.concat([X_train, y_train], axis=1)

# define the borders for extreme values
lower = new['usd_pledged'].quantile(0.025)
upper = new['usd_pledged'].quantile(0.975)

# remove extreme values
new_2 = new[(new['usd_pledged'] < upper) & (new['usd_pledged'] > lower)]

# now we split again into y_train and X_train
y_train = new_2['usd_pledged']
X_train = new_2.drop('usd_pledged', axis=1)<jupyter_output><empty_output><jupyter_text>Scaling<jupyter_code># Scaling with standard scaler
scaler = StandardScaler()
X_train_scaled_st = scaler.fit_transform(X_train[col_scale])
X_test_scaled_st = scaler.transform(X_test[col_scale])
# Concatenating scaled and dummy columns 
X_train_preprocessed_st = np.concatenate([X_train_scaled_st, X_train.drop(col_scale, axis=1)], axis=1)
X_test_preprocessed_st = np.concatenate([X_test_scaled_st, X_test.drop(col_scale, axis=1)], axis=1)<jupyter_output><empty_output><jupyter_text>Polynomial RegressionWe start with a simple polynomial regression with second degree which is a linear regression. We fit the model and print the root mean squared error, mean percentage error, $R^2$.<jupyter_code># Compute mean absolute percentage error
def mean_absolute_percentage_error(y_train, y_pred): 
    y_train, y_pred = np.array(y_train), np.array(y_pred)
    return np.mean(np.abs((y_train - y_pred) / y_train)) * 100
poly_reg = PolynomialFeatures(degree=1)
X_poly = poly_reg.fit_transform(X_train_preprocessed_st)
pol_reg = LinearRegression()
pol_reg.fit(X_poly, y_train)

X_poly_test = poly_reg.fit_transform(X_test_preprocessed_st)
X_poly_test.shape
y_pred = pol_reg.predict(X_poly_test)

print("RMSE:", np.sqrt(mean_squared_error(y_test, y_pred)))
print("MAPE:", mean_absolute_percentage_error(y_test, y_pred))
print("R2:", r2_score(y_test, y_pred))<jupyter_output>RMSE: 95536.25607809285
MAPE: 552.0544006144306
R2: 0.2907157758680152<jupyter_text>If you look at the different errors, you can see that the root mean squared error is very high due to our goal value of approximately 20000 USD. This is also reflected in the mean absolute percentage error which indicates that the average predicted value is 552% off. This means that our model is improvable. We saw in our EDA that some features seem not linear e.g. duration and preparation. Therefore it´s better nor using a linear model. Let´s try the polynomial regression again with a higher degree and compare the results.<jupyter_code>poly_reg = PolynomialFeatures(degree=2)
X_poly = poly_reg.fit_transform(X_train_preprocessed_st)
pol_reg = LinearRegression()
pol_reg.fit(X_poly, y_train)

X_poly_test = poly_reg.fit_transform(X_test_preprocessed_st)
X_poly_test.shape
y_pred = pol_reg.predict(X_poly_test)

print("RMSE:", np.sqrt(mean_squared_error(y_test, y_pred)))
print("MAPE:", mean_absolute_percentage_error(y_test, y_pred))
print("R2:", r2_score(y_test, y_pred))<jupyter_output>RMSE: 135837.35824409494
MAPE: 276.948558905549
R2: -0.4339114023367132<jupyter_text>With degree 2 the root mean squared error got worse but the mean average percentage error improved. In our case the MAPE is more important because very small and very big values are relativized. Let´s also check if the MAPE gets better with degree 3.<jupyter_code>poly_reg = PolynomialFeatures(degree=3)
X_poly = poly_reg.fit_transform(X_train_preprocessed_st)
pol_reg = LinearRegression()
pol_reg.fit(X_poly, y_train)

X_poly_test = poly_reg.fit_transform(X_test_preprocessed_st)
X_poly_test.shape
y_pred = pol_reg.predict(X_poly_test)

print("RMSE:", np.sqrt(mean_squared_error(y_test, y_pred)))
print("MAPE:", mean_absolute_percentage_error(y_test, y_pred))
print("R2:", r2_score(y_test, y_pred))<jupyter_output>RMSE: 750303.1193302529
MAPE: 292.1948652455093
R2: -42.74791589655732<jupyter_text>As you can see the MAPE got worse again. That´s the reason we take the model with second degree.  AdaboostIn general using Ensemble Methods like Adaboost can improve regression results.<jupyter_code>X_train_preprocessed_st, y_train = make_regression(n_features=16, n_informative=2,
                       random_state=42, shuffle=False)

regr = AdaBoostRegressor(random_state=42, n_estimators=5)
regr.fit(X_train_preprocessed_st, y_train)

y_pred = regr.predict(X_test_preprocessed_st)
print("RMSE:", np.sqrt(mean_squared_error(y_test, y_pred)))
print("MAPE:", mean_absolute_percentage_error(y_test, y_pred))
print("R2:", r2_score(y_test, y_pred))<jupyter_output>RMSE: 117326.20707454011
MAPE: 106.0216597521679
R2: -0.0697293867372506
{'fraction_non_alphanumeric': 0.09374513042665254, 'fraction_numerical': 0.02250132452395051, 'mean_word_length': 4.555401662049862, 'pattern_counts': {'":': 0, '<': 137, '<?xml version=': 0, '>': 143, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '155040', 'license': 'MIT', 'n_tokens_mistral': 10441, 'n_tokens_neox': 9800, 'n_words': 3290, 'path': 'Kickstarter.ipynb', 'repo_name': 'Martina385/Kickstarter', 'size': '32109'}