<jupyter_start><jupyter_text>Kickstarter data analysis by <NAME> **Questions:**1. Which categories have the best and worst success rates?2. How does success rate relate to goal amount?3. Does duration of a project influence success rate?4. What features have the greatest influence on success?5. Can we predict success of current/future projects?      First, A Look at the Data<jupyter_code># load packages
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from IPython import display
%matplotlib inline
# load in csv file as dataframe
df = pd.read_csv('ks-projects-201801.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>This data frame has 15 columns and 378,661 entries.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 378661 entries, 0 to 378660
Data columns (total 15 columns):
ID                  378661 non-null int64
name                378657 non-null object
category            378661 non-null object
main_category       378661 non-null object
currency            378661 non-null object
deadline            378661 non-null object
goal                378661 non-null float64
launched            378661 non-null object
pledged             378661 non-null float64
state               378661 non-null object
backers             378661 non-null int64
country             378661 non-null object
usd pledged         374864 non-null float64
usd_pledged_real    378661 non-null float64
usd_goal_real       378661 non-null float64
dtypes: float64(5), int64(2), object(8)
memory usage: 43.3+ MB<jupyter_text>It looks like from the kaggle description that 'usd pledged' and usd_pledged_real are both conversions of the pledged amount to usd, but done by different users. 'usd pledged' was calculated via Kickstarter and 'usd_pledged_real' was calculated via the Kaggle user tonyplaysguitar using the Fixer.io API. Let's take a look at some of the rows to see if they match up.<jupyter_code>df[['country','pledged', 'usd pledged','usd_pledged_real']].head(15)<jupyter_output><empty_output><jupyter_text>Above, I can already see that 'usd pledged' done by Kaggle has some mistakes, so I'll drop it and go with 'usd_pledged_real'. In addition, I'll drop the 'pledged' and 'goal' columns since these are represented consistently as USD in other columns.<jupyter_code># drop unnecessary columns
df.drop(['usd pledged', 'pledged','goal'], axis=1, inplace=True)

# confirm (should be 12)
df.shape[1]<jupyter_output><empty_output><jupyter_text>I can also drop the ID and name columns since those will not be important in the analysis.<jupyter_code># drop ID column
df.drop(['ID','name'], axis=1, inplace=True)

# confirm (should be 11)
df.shape[1]
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>There are no null values in the dataset. Let's take a look at some descriptive statistics for the numerical variables.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>We can see that there are a lot of high outliers in the three numerical features above, based on the fact that their max values are much greater than the lower 75% of the data. Now let's take a closer look at the categorical variables.<jupyter_code>df.category.value_counts()
df.category.nunique()
df.main_category.value_counts()
df.main_category.nunique()<jupyter_output><empty_output><jupyter_text>There are 15 main categories and 159 sub categories for these kickstarter projects. I now want to take a look at the proportion of projects in each main category.<jupyter_code># convert value count to df
df_main_cat_counts = df.main_category.value_counts().reset_index()
df_main_cat_counts.columns = ['main_category','count']
df_main_cat_counts
# set grid style
sns.set_style('darkgrid')
# plot percent of projects in each main category
heights = df_main_cat_counts['count']/len(df) *100
ind = np.arange(0,15)
labels = df_main_cat_counts['main_category'].values.tolist()

ind = np.arange(1,16)
width=0.4

plt.figure(figsize=(12,5))
plt.bar(ind, heights, width)
plt.xticks(ind, labels, rotation='vertical', size=15)
plt.yticks(size=15)
plt.ylabel('%', size=15)
plt.title('Percent of projects in each main category', size=18)

# save image
plt.savefig('category_percents.png', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>We can see that Film & Video and Music were the most common main categories, while Dance and Journalism were the least common Let's take a look now at the state of the projects to evaluate how many were successful and how many failed.<jupyter_code>df.state.value_counts()
# disregard live, suspended, undefined, and canceled ones for now
df_completed = df[(df.state == 'successful') | (df.state == 'failed')]

perc_success = len(df_completed[df_completed.state == 'successful']) / len(df_completed) *100
perc_fail = 100 - perc_success

print('{:.2f}% of completed projects were successful.'.format(perc_success))
print('{:.2f}% of completed projects failed.'.format(perc_fail))<jupyter_output>40.39% of completed projects were successful.
59.61% of completed projects failed.<jupyter_text>Let's take a quick look at how the rest of the data breaks down by project state. I'll look at the median values since there seem to be a lot of outliers based on the descriptive statistics above.<jupyter_code>df_by_state = df.groupby('state').median()

df_by_state.head()<jupyter_output><empty_output><jupyter_text>Specifically paying attention to failed vs. successful, we can already see at first glance that the successful projects had a higher median for number of backers and pledged amount and a lower median for goal amount than the failed projects.<jupyter_code># plot pledged and goal medians for failed vs. successful projects
succ_heights = df_by_state.loc['successful'][['usd_pledged_real','usd_goal_real']].values.tolist()
fail_heights = df_by_state.loc['failed'][['usd_pledged_real','usd_goal_real']].values.tolist()

ind = np.arange(0,2)
labels = ['Pledged USD','Goal USD']

width = 0.3

plt.bar(ind, succ_heights, width, color='#419df4', label='Sucessful')
plt.bar(ind+width, fail_heights, width, color='#f49242', label='Failed')
plt.xticks(ind+0.1, labels)
plt.legend();<jupyter_output><empty_output><jupyter_text>Now I'll take a closer look at the country variable.<jupyter_code>df_completed.country.value_counts()<jupyter_output><empty_output><jupyter_text>The 'N,0"' value for country has 210 occurences. After googling what this means, I found that it most likely represents 'Null Island', a fictional island. Therefore, these values should be converted to null. First let me take a look at these entries.<jupyter_code>df_completed[df_completed.country == 'N,0"'].head()<jupyter_output><empty_output><jupyter_text>It looks like we can use the currency values to fill in the missing country names.<jupyter_code># get name of all distinct currencies in the null country entries
null_currencies = df_completed[df_completed.country == 'N,0"']['currency'].unique().tolist()
null_currencies<jupyter_output><empty_output><jupyter_text>Based on the list above, I can use most of the currencies to fill in the missing country data. Unfortunately, I cannot do this for the entries with euro listed as their currency, since euro is the currency for multiple countries.<jupyter_code># fill missing country values using currency (except for when currency is euro)
df_completed.country = np.where((df_completed.country == 'N,0"') & (df_completed.currency != 'EUR'), 
                                df_completed.currency.str[:-1], df_completed.country)
# convert remaining values of N,0" to null
df_completed.country = np.where(df_completed.country == 'N,0"', None, df_completed.country)
df_completed.country.value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>There are still 21 missing values for country since the missing entries with euro as their currency could not be determined. Luckily this is a very small percent of the total entries, so I will go ahead and drop these rows.<jupyter_code>df_completed = df_completed[df_completed.country.notnull()]
# confirm
df_completed.country.isnull().sum()<jupyter_output><empty_output><jupyter_text>Next I want to use the deadline and launched columns to create a duration column.<jupyter_code># convert deadline and launched columns from strings to datetime objects
df_completed.deadline = pd.to_datetime(df_completed.deadline)
df_completed.launched = pd.to_datetime(df_completed.launched).dt.date #drops time part
df_completed.launched = pd.to_datetime(df_completed.launched) #convert back to datetime object
    
# create duration column
df_completed['duration'] = df_completed.deadline - df_completed.launched #creates timedelta object
# convert from timedelta to number of days
df_completed['duration'] = df_completed['duration'].apply(lambda x: x.days)
# confirm
df_completed.duration.head()<jupyter_output><empty_output><jupyter_text>Finally I want to map the values in the state column to 1 for successful and 0 for failed.<jupyter_code># map successful to 1, failed to 0
df_completed.state = df_completed.state.map({'successful':1, 'failed':0})

# rename column
df_completed.rename({'state':'succeeded'}, axis=1, inplace=True)
df_completed.head()<jupyter_output><empty_output><jupyter_text>Explore Question 1: Which categories have the best and worst success rates? Now I'll take a look at how successes and failures break down by category.<jupyter_code>def plot_success_rate(df, col, xlabel=''):
    """
    Creates a table and plot of the success and failure rates grouped by 
    the specified column in the dataframe.
    
    INPUTS: dataframe, column to group by
    OUTPUTS: table of success rates and failure rates, sorted by success rate, plot of success rates
    """
    succ_rate = df.groupby(col).mean()['succeeded'] *100
    fail_rate = 100 - succ_rate
    
    results = pd.concat([succ_rate, fail_rate], axis=1)
    results.columns = ['success_rate', 'failed_rate']
    
    results = results.sort_values(by='success_rate', ascending=False)
    
    # plot success rate
    heights = results.success_rate.values.tolist()

    ind = np.arange(0,len(results))
    labels = results.index.values.tolist()

    width = 0.4

    plt.figure(figsize=(12,5))
    plt.bar(ind, heights, width)
    plt.xticks(ind, labels, rotation=45, size=15)
    plt.yticks(size=15)
    plt.ylabel('% of successful projects', size=15)
    plt.xlabel(xlabel, size=15)
    
    title = 'Success rate broken down by {}'.format(col)
    plt.title(title, size=18);

    return results
results = plot_success_rate(df_completed, 'main_category')
plt.savefig('category_success.png', bbox_inches='tight')

print(results)<jupyter_output>success_rate  failed_rate
main_category                           
Dance             65.435209    34.564791
Theater           63.796134    36.203866
Comics            59.141527    40.858473
Music             52.665491    47.334509
Art               44.889045    55.110955
Games             43.890467    56.109533
Film & Video      41.792127    58.207873
Design            41.592083    58.407917
Publishing        34.707526    65.292474
Photography       34.110847    65.889153
Fashion           28.284616    71.715384
Food              27.591367    72.408633
Crafts            27.052955    72.947045
Journalism        24.397300    75.602700
Technology        23.785582    76.214418<jupyter_text>The four categories that had the highest proportion of successful projects (greater than 50%) were Dance, Theater, Comics, and Music. All other categories had a higher percentage of failed projects than successful projects (i.e. less than a 50% success rate). Technology had the lowest proportion of successful projects, at about 24%. But what if categories with the lowest success rate end to have higher goal amounts and THAT is what affects the outcome more? Let's look at this by plotting the median goal amount for each category (I want to look at median instead of mean since there are a lot of high outliers for goal amount).<jupyter_code># create df of success rate for each category
succ_rate = df_completed.groupby('main_category').mean()['succeeded'] *100
succ_rate = succ_rate.reset_index().sort_values(by='succeeded', ascending=False)
# get list of categories in order
categories_ordered = succ_rate.main_category.values.tolist()
categories_ordered
# create df of median values grouped by category
median_by_cat = df_completed.groupby('main_category').median()
median_by_cat
# reorder the categories to the be same as success rate df
median_by_cat = median_by_cat.reindex_axis(categories_ordered, axis=0)
median_by_cat
# plot median goal amount per category
heights = median_by_cat.usd_goal_real.values.tolist()

ind = np.arange(0,len(median_by_cat))
labels = median_by_cat.index.values.tolist()

width = 0.4

plt.figure(figsize=(12,5))
plt.bar(ind, heights, width, color = '#f49242')
plt.xticks(ind, labels, rotation=45, size=15)
plt.yticks(size=15)
plt.ylabel('Median Goal Amount (USD)', size=15)
    
plt.title('Median USD goal amount by main category', size=18);
plt.savefig('category_goal.png', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>This looks like it could be the case for Technology, which does have the highest median goal amount seen in the graph above to the right (with the categories plotted in the same order as the graph on the left). However, this is not always the case - for example, Crafts has the lowest median goal amount but still has one of the lowest success rates.  Explore Question 2: How does success rate relate to goal amount? First I'll look at the median goal amounts for the successful and failed projects, and also plot goal against 'succeeded' on a category plot.<jupyter_code># group by succeeded and get median of goal
median_goal = df_completed.groupby('succeeded').median()['usd_goal_real'].reset_index()
median_goal.columns = ['succeeeded','median_goal_USD']
median_goal
# create category plot
sns.catplot(x="succeeded", y="usd_goal_real", data=df_completed);<jupyter_output><empty_output><jupyter_text>As found earlier, the median goal amount for the failed projects is almost twice as high as that of the successful projects. We can also see from the plot that the goal spread for the failed projects is much wider. Now I want to create a new column to bucket the goals into ranges. First I'll look again at the descriptive statistics for usd_goal_real so that I can choose appropriate bin values.<jupyter_code>df_completed.usd_goal_real.describe()
# specify bin values
bin_values = [0,5000,10000,15000,20000,25000,50000, df_completed.usd_goal_real.max()]

# cut goal amount into buckets based on bin values
df_completed['usd_goal_bucket'] = pd.cut(df_completed.usd_goal_real, bin_values)
# get value counts for goal bucket column
df_completed.usd_goal_bucket.value_counts()<jupyter_output><empty_output><jupyter_text>From these counts, we can see that the majority of the goal amounts are in the 0 to $5000 range. Now I can see how the success rate breaks down by these goal buckets.<jupyter_code>results = plot_success_rate(df_completed, 'usd_goal_bucket', xlabel='Goal range in USD')
plt.savefig('goal_success.png', bbox_inches='tight')

print(results)<jupyter_output>success_rate  failed_rate
usd_goal_bucket                                   
(0.0, 5000.0]               48.983384    51.016616
(5000.0, 10000.0]           39.468396    60.531604
(10000.0, 15000.0]          37.405491    62.594509
(15000.0, 20000.0]          32.546260    67.453740
(20000.0, 25000.0]          28.588001    71.411999
(25000.0, 50000.0]          25.202584    74.797416
(50000.0, 166361390.71]     13.077795    86.922205<jupyter_text>From the plot above, we can see that with increasing goal buckets, the proportion of successful projects decreases. The lowest bucket (0 to $5000) has almost a 50% success rate, while the highest bucket has only a 13% success rate.  Explore Question 3: Does duration of a project influence success rate?<jupyter_code># group by succeeded and get average duration
average_goal = df_completed.groupby('succeeded').mean()['duration'].reset_index()
average_goal.columns = ['succeeeded','avg_duration']
average_goal
# create category plot
sns.catplot(x="succeeded", y="duration", data=df_completed);<jupyter_output><empty_output><jupyter_text>The average durations for the failed and succeeded projects are similar, only about 3 days off from each other. From the plot above we can see that the data seems to be more uniform across duration lengths for both the failed and successful projects.<jupyter_code># get number of unique duration lengths
df_completed.duration.nunique()
# get value counts
df_completed.duration.value_counts()<jupyter_output><empty_output><jupyter_text>There are 92 unique durations in the dataset. Since some of these are very sparse, I will bucket these values.<jupyter_code>df_completed.duration.describe()
# specifiy bin values
bin_values = [0,14,20,25,30,40,50,df_completed.duration.max()]

# cut duration into buckets based on bin values
df_completed['duration_bucket'] = pd.cut(df_completed.duration, bin_values)
# get value counts for duration bucket column
df_completed.duration_bucket.value_counts()<jupyter_output><empty_output><jupyter_text>Above we can see that the majority of projects fell in the 25-30 day range for duration. Now let's see how success rate breaks down for each duration bucket.<jupyter_code>plot_success_rate(df_completed, 'duration_bucket', xlabel='Duration range in days')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08885979393180395, 'fraction_numerical': 0.045255308248420334, 'mean_word_length': 4.919137466307277, 'pattern_counts': {'":': 0, '<': 107, '<?xml version=': 0, '>': 107, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '575304', 'license': 'CNRI-Python', 'n_tokens_mistral': 5971, 'n_tokens_neox': 5243, 'n_words': 1813, 'path': 'Kickstarter_analysis.ipynb', 'repo_name': 'cnspatino/Kickstarter_Analysis', 'size': '17572'}