<jupyter_start><jupyter_text>Analysis of Crowdfunding Projects Data DescriptionKickstarter.com is a crowd funding website where creators post descriptions of projects they want to create, and individuals contribute some amount of personal money to finance the project. In return for funding, contributors typically get a wide range of “Backer Rewards”. The rewards differ depending on the type of project. For example, a book/video project will typically issue a digital copy of the final product to backers that donate above a given amount. The scope of projects can vary widely, from the initial factory runs of complex hardware products. One creator simply requested ten dollars so he could make some potato salad and instead raised $55,000 and threw a crazy party Data SourceWebrobots.io runs a monthly scrape of all Kickstarter projects available on the Kickstarter website. The resulting data are available as both csv and json files, and each package covers a number of projects on the website from the start date (April 21st 2009) until the scrape date. From April 2015, Kickstarter began limiting the number of search results for each category. As a result, the numbers of historical results displayed are limited. Webrobots started running scrapes in multiple sub-categories to capture more instances, and as a result, there are multiple entries for several projects. Dates of StudyThe scrape data that I'm working with was taken from Kickstarter.com in March of 2019. The scrape data includes both historical and current projects, so there's a wide variety of dates captured here. Data issues Inconsistent DatesKickstarter.com limits the number of search results shown for each category of project (ex. 'Film & Video', or 'Publishing'). Because of this limit and because the number of projects in each category varies based on their relative popularity, the active dates for projects across different categories are not all the same. As we'll see later, the popularity of the Kickstarter.com website fluctuates over time, so different launch dates can affect a project's chances of success Missing, incorrect or duplicate dataThe overall quality of the dataset is very good, with very few pieces of missing data. Scope and Goals Audience and Business NeedThis project is aimed at creators that are interested launching projects on Kickstarter's platform. Kickstarter campaigns are "all or nothing", meaning if a campaign doesn't reach the funding goal, all of the funds are returned to backers. For this reason, it's important for campaigns to carefully consider how their project is presented Previous Work There's been a lot of work published using Kickstarter datasets in the past. Most of the past work has focused on trying to predict whether a project will achieve their funding goal based on characteristics such as funding goal, project  category, and number of backers. While that's interesting from a technical/model optimization view, actionable information for creators is limited so far. Citations for previous work is given in Appendix B at the end of this notebook Project GoalIn light of the previous work done and the popularity of the dataset, this project attempts to approach this dataset from an original angle. The main question I'm trying to answer here is:* *Can we create a way for new creators to make their campaigns "look" like successful projects* This project aims to achieve that by running a segmentation analysis on attributes of successful and unsuccesful projects and try to uncover their differences  Data Cleaning and Manipulation JSON FormattingMany of columns in the original dataset were returned as JSON objects during the webrobots scrape. When exporting to a .csv file, and then importing into Pandas, the JSON formatting was lost. The cleaning script made for this project re-applied the JSON formatting, and then pulled the relevant information back out Interpreted DataSome columns were made with interpreted data. Time deltas were calculated between:* Campaign launch date and campaign creation date* Campaign launch date and campaign deadline* Campaign launch date and date funding goal was reachedNatural Language Processing was run on project titles and descriptions to better define project differences. The data from the NLP analysis was used to score project descriptions on their active phrasing and whatever other criteria I make up  Pruning and other cleaningThe remaining cleaning work has been selecting the data relevant to the project. The original dataset contained contained image url's, and duplicates of data available in other columns. These items were removed from the dataset used for analysis Date formats All of the columns that are time/date related are in epoch time, and I've left it that way to make things easier to do math External DataData pulled from IndieGoGo has been merged in with the Kickstarter dataset for comparison<jupyter_code>#Import libraries needed for analysis
import pandas as pd
import seaborn as sb
import matplotlib.pyplot as plt
import numpy as np
import datetime
import pdb
import time
import re
import os
import math as mt

url = 'https://tufts.box.com/shared/static/939cuad4z6ieo8njf3ya58a603q4enyh.csv'
if not os.path.exists('2019KickDataCleaned.csv'):
    df = pd.read_csv(url)
else:
    df = pd.read_csv('2019KickDataCleaned.csv', index_col = 0)

df.drop(columns = ['subcats'], inplace = True)
df = df.drop_duplicates(subset = ['id'])
len(df)<jupyter_output><empty_output><jupyter_text>Project statesThere are four different states listed for each project in the dataset:| State | Description||:---:|:---:||Live | Project is currently actively fundraising||Cancelled | Project was cancelled by the creators before the deadline | |Suspended | The project was suspended by Kickstarter for not meeting some criteria||Succesful| The project has met or exceeded the funding goal ||Failed| The project failed to meet their funding goal| Since this project aims to analyze factors that make projects successful, I've filtered the dataset to include only projects that are either successful or failed  Merge in External Data Alexa Site Traffic Ranking dataI snagged the Alexa global traffic ranking for Kickstarter.com for the last three years. Let's take a look and see if there's any correlation between the site's traffic ranking and the success rate of projects. I only had access to the last three years worth of ranking data, so some projects will be missing informationI'm using the launch date as a foreign key to match with Kickstarter's Alexa ranking, so I have to convert the timestamp to M/D/YYYY formatOnce the timestamps are converted, merge the Alexa ranking data in with the Kickstarter data<jupyter_code>url = 'https://tufts.box.com/shared/static/1xr291wybxfardn6cteqdmq08khic84t.csv' #Read the Alexa data
alexa = pd.read_csv(url, skiprows = 6) 

#Conver the launch date timestamp (in epoch time) to M/D/YYYY format
df['launch_date'] = df.apply(lambda x: time.strftime('%-m/%-d/%Y', time.localtime(x['launched_at'])), axis = 1)

#Merge the two dataframes together on the launch date column with a left join
df = df.merge(alexa, how = 'left', left_on = 'launch_date', right_on = 'Date')<jupyter_output><empty_output><jupyter_text>Convert Foreign Currencies to USD termsThere's a number of projects that originated outside of the US. For these projects, the funding goals are in a different currency. In order to clean up the dataframe for analysis, convert these into USD terms.Kickstarter.com was kind enough to include the prevailing currency exchange rate at the time of the project, so I'll use that for the conversion<jupyter_code># Multiply the goal by the fx rate
df['goal'] = df.apply(lambda x: (x['goal'] * x['fx_rate']), axis = 1) 
# Check that the merge was successful and did what I wanted
df.head()<jupyter_output><empty_output><jupyter_text>IndieGoGo DataWebrobots.io also runs a monthly scrape of the indiegogo website. Since there's already been a large amount of work done on Kickstarter data in the past, I want to try and make some unique analyses using a new dataset.I grabbed the most recent version of the indiegogo dataset, and now I'm going to merge it with my Kickstarter dataset<jupyter_code>#Start by taking a look at the indiegogo dataset
url = 'https://tufts.box.com/shared/static/on2cmdxc4vqpm2dutaaf7l8764ktq0u5.csv'
indie = pd.read_csv('https://tufts.box.com/shared/static/on2cmdxc4vqpm2dutaaf7l8764ktq0u5.csv')
#Drop columns that are unrelatd to the original kickstarter dataset
indie.drop(labels = ['bullet_point', 'category_url', 'clickthrough_url', 'image_url', 'is_pre_launch', 'offered_by', 
                     'price_offered', 'product_id', 'product_stage', 'project_type', 'price_retail', 
                     'perk_goal_percentage', 'tags'], axis = 1, inplace = True)
indie.head()<jupyter_output><empty_output><jupyter_text>Making the Indiegogo data look like the Kickstarter dataSince the two websites are unique, there's a lot of differences in how the datasets are structuredThe following few steps are manipulating the indiegogo data to look like the Kickstarter data Date ConversionFirst up, change the dates that are in a YY/MM/DD format and convert that into epoch time (the standard for the Kickstarter dataset)<jupyter_code>import datetime # Library for working with datetime information
indie.dropna(inplace = True) # Drop rows that are missing important information

# Define a function to strip out the extra information on the indiegogo timestamps, and then convert to epoch time
def epoch(target): 
    '''
    This function takes a date from the indiegogo dataset (read originally as a string), converts to a datetime object
    and then converts that date to epoch time
    '''
    target = target.split('T')
    target = datetime.datetime.strptime(target[0], '%Y-%m-%d')
    target = target.strftime('%s')
    return(target)

#Run the epoch function on the project close date and project open date columns
indie.close_date = indie.close_date.apply(epoch) 
indie.open_date = indie.open_date.apply(epoch)<jupyter_output><empty_output><jupyter_text>Renaming CategoriesThe category labels for indiegogo don't quite match up to Kickstarter categories. I've made a dictionary to convert the indiegogo categories to match Kickstarter categoriesSome of the categories in the indiegogo dataset didn't have an obvious counterpart in the kickstarter data, so I've left those in their original categories<jupyter_code>indieDictionary = { 'Home' : 'crafts',
                   'Web Series & TV Shows' : 'film & video',
                   'Photography' : 'photography',
                   'Health & Fitness' : 'health&fitness',
                   'Comics' : 'comics',
                   'Food & Beverages' : 'food',
                   'Music' : 'music',
                   'Dance & Theater' : 'dance',
                   'Podcasts, Blogs & Vlogs' : 'journalism' ,
                   'Video Games' : 'games',
                   'Art' : 'art',
                   'Fashion & Wearables' : 'fashion',
                   'Phones & Accessories' : 'technology',
                   'Wellness' : 'health&fitness',
                   'Camera Gear': 'cameragear',
                   'Travel & Outdoors': 'travel&outdoors',
                   'Film' : 'film & video',
                   'Productivity': 'productivity',
                   'Tabletop Games':  'games',
                   'Audio' : 'music',
                   'Culture' : 'culture',
                   'Human Rights' : 'human_rights',
                   'Environment' : 'environment',
                   'Writing & Publishing' :  'publishing',
                   'Transportation' : 'technology',
                   'Local Businesses' : 'local_businesses',
                   'Energy & Green Tech' : 'energy&green tech'}<jupyter_output><empty_output><jupyter_text>Column Names, Epoch Time Conversion, and some MathRename the columns of the indiegogo data to match the kickstarter column names. When the dates were converted to epoch time in the step above, the data was read as a string type. Convert those to intsFinally, calculate how long the fundraising campaigns were for each project<jupyter_code># Use the dictionary to re-categorize the indiegogo projects
indie.category = indie.category.map(indieDictionary) 

# Add an indiegogo tag column to separate kickstarter and indiegogo data
indie['source'] = 'indiegogo' 

# Rename each column header to match kickstarter
indie = indie.rename(index=str, columns={"close_date": "deadline", 
                              "funds_raised_amount": "usd_pledged", 
                              "open_date" : "launched_at",
                              "is_indemand":"spotlight", 
                              "tagline" : "blurb", 
                              "title":"name", 
                              "perks_claimed" : "backers_count",
                              "project_id": "id", 
                              "category" : "fullcats",
                              "source_url" : "urls"})

# Cast the epoch timestamps as ints instead of strings
indie.deadline = indie.deadline.astype(int) 
indie.launched_at = indie.launched_at.astype(int)

#Calculate how long each fundraising campaign was run for
indie['lauDeadDelta'] = indie.apply(lambda x: (x["deadline"] - x["launched_at"])/60/60/24, axis = 1)
indie['goal'] = indie.apply(lambda x: (mt.ceil(x['usd_pledged'] / (x['funds_raised_percent'] + 0.00000000001))), 
                            axis = 1)
indie['funds_raised_percent'] = indie.apply(lambda x: (x['funds_raised_percent'] * 100), axis = 1)<jupyter_output><empty_output><jupyter_text>Tagging IndieGoGo campaigns as successful or failedSince indiegogo creatores receive their funds regardless of how much the projects raise, the indiegogo dataset doesn't have a tag for the status of a projectTo fix that, I'll tag any project that reaches it's fundraising goal as successful and any that don't as failed<jupyter_code>inSuccess = pd.DataFrame()
inFailed = pd.DataFrame()

# First filter out all projects that raised 100% of their goal or more, then add a 'state' column and set that to
# successful
inSuccess = indie[indie['funds_raised_percent'] >= 100]
inSuccess['state'] = 'successful'

# Now do the same for projects that didn't raise 100% of their funding goals, and tag them as failed
inFailed = indie[indie['funds_raised_percent'] < 100]
inFailed['state'] = 'failed'

# Merge the two dataframes back together 
indie = inSuccess.merge(inFailed, how = 'outer', on = None)
del inSuccess
del inFailed
# Check that the above operations were successful
indie.head()<jupyter_output><empty_output><jupyter_text>Merge the two DataFrames together and deduplicateNow that the two datasets look alike, merge them together with an outer join (include only entries that in either dataset, but not both)Since the webscrape by webrobots searched by subcategory, projects can be listed multiple times if they had multiple subcategry tags. Use the drop_duplicates method from pandas to get rid of the extras<jupyter_code>merged = df.merge(indie, how = 'outer', on = None)
merged.reset_index(inplace = True)
merged.drop(labels = ['index', 'Unnamed: 0'], axis = 1, inplace = True)

del indie 
del df
# Drop duplicates from the dataset, use the 'id' column since its known to be unique only to each project
merged.drop_duplicates(subset = 'id', inplace = True)
merged.head()<jupyter_output>/Users/chrismay/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py:938: UserWarning: You are merging on int and float columns where the float values are not equal to their int representation
  'representation', UserWarning)
/Users/chrismay/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py:946: UserWarning: You are merging on int and float columns where the float values are not equal to their int representation
  'representation', UserWarning)<jupyter_text>Calculate the Funds Raised over Goal PercentKickstarter and Indiegogo give a different gauge of percent of funds raised. Standardize on giving the percent of goal for each project<jupyter_code>merged['funds_raised_percent'] = merged.apply(lambda x: (((x['usd_pledged']/ (x['goal'] + 0.0000001))*100)), axis = 1)
# Check the indiegogo data at the end of the dataset to make sure it was done right
merged.tail()<jupyter_output><empty_output><jupyter_text>Reducing Memory Usage of the DataFrameI've occassionally run into kernel issues with this notebook, which is most likely due to running out of memory. Optimize some of the datatypes to reduce working  Re-casting integer variablesInts have been automatically given a float or int64 datatype, downcast these to smaller variables to use less memory<jupyter_code>merged['backers_count'] = merged['backers_count'].astype(np.uint8)
merged['deadline'] = merged['deadline'].astype(np.uint32)
merged['goal'] = merged['goal'].astype(np.uint32)
merged['id'] = merged['id'].astype(np.uint32)
merged['launched_at'] = merged['launched_at'].astype(np.uint32)
merged['usd_pledged'] = merged['usd_pledged'].astype(np.uint32)
merged['lauDeadDelta'] = merged['lauDeadDelta'].astype(np.uint8)
merged['funds_raised_percent'] = merged['funds_raised_percent'].astype(np.float16)

merged.dtypes<jupyter_output><empty_output><jupyter_text>Changing object types to categoriesA few of the columns listed as object types are actually categorical variables. Columns that have a low number of unique values can be changed to categorical variables, where each category will be given an 8 bit integer value and then mapped to a much shorter array.<jupyter_code>merged['fullcats'] = merged['fullcats'].astype('category')
merged['staff_pick'] = merged['staff_pick'].astype('category')
merged['spotlight'] = merged['spotlight'].astype('category')
merged['source'] = merged['source'].astype('category')
merged['currency'] = merged['currency'].astype('category')
try:
    merged.drop(labels = ['Metric'], axis = 1, inplace = True)
except:
    pass
merged.dtypes<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>merged.describe()<jupyter_output><empty_output><jupyter_text>Data SkewFrom a first look at the descriptive statistics on the dataset, the data seems to be heavily skewed.The means and medians (50%) for "goal", and "usd_pledged", are off by an order of magnitude.I know there's a lot of different categories of projects, and some of the skew is probably due to different funding goals for different categories of projects (ex. A video publishing project needs a lot less money than the first run of a new hardware device)Knowing some of the skew comes from differences in project categories, let's take a closer look grouping by project category  Drop projects that raised no money at allLet's consider only projects that had at least one backer in the analysisThe indiegogo dataset has a few projects with fundraising goals over several million that didn't raise any money at all. Filter out those projects to make the remaining data easier to read<jupyter_code>merged = merged[(merged['backers_count'] > 0) & (merged['goal'] < 2000000)]<jupyter_output><empty_output><jupyter_text>Frequency of SuccessStart analyzing the data from the two datasets by taking a look at how success rates and funding goals differ across each category<jupyter_code>#Filter out projects that are live, canceled, or suspended
merged = merged[(merged['state'] == 'successful')|(merged['state'] =='failed')] 
suc = merged[(merged['state'] == 'successful')]
fSuc = len(suc)/len(merged)
print('The relative frequency of projects meeting their funding goal is', fSuc, 'or about', mt.floor(fSuc * 100), '%')<jupyter_output>The relative frequency of projects meeting their funding goal is 0.5887230220078954 or about 58 %<jupyter_text>Comparing the Money RaisedThe chart below compares average money pledged to projects in different categories against average raised in succesfully funded projects. It also gives a comparison of the medians for each categoryAs was seen in the descriptive statistics, there's a heavy skew here. Based on the median pledge totals, it seems like a lot of projects must raise almost no money at allThere's also a strange phenomenom with Indiegogo projects. A few categories (camera gear, environment and green tech, travel and outdoor, health and fitness in particular) are unique to indiegogo. It looks like there's some projects that raised incredible amounts of money that are skewing the averages<jupyter_code># Compare the money raised by succesfull projects against all projects in a certain category
money = pd.DataFrame()
grp = merged.groupby(['fullcats'])
sucGrp = suc.groupby(['fullcats'])
money['AvgRaised'] = grp.usd_pledged.agg(np.mean)
money['SuccessAvgRaised'] = sucGrp.usd_pledged.agg(np.mean)
money['MedianRaised'] = grp.usd_pledged.agg(np.median)
money['MedianSuccess'] = sucGrp.usd_pledged.agg(np.median)
#money.reset_index(inplace = True)

ax = money.plot(kind = 'bar', title='Crowdfunding Funds Raised by Project Category')
ax.set_xlabel("Category")
ax.set_ylabel("Funds Raised")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>Comparing pledge goalsThe same skew shows up in the fundraising goals for each project, there seems to be a few projects that have very high funding goals.What's interesting here is that most projects have significantly lower funding goals than the top projects. It's difficult to see, but it also seems there's a relatively small difference in the median funding goals between successful projects and all projects<jupyter_code># Delete old dataframes to reduce working memory
del money

# Compare the funding goals for successful projects in each category vs. the funding goals of all projects
goal = pd.DataFrame()
goal['avgGoal'] = grp.goal.agg(np.mean)
goal['sucAvgGoal'] = sucGrp.goal.agg(np.mean)
goal['medGoal'] = grp.goal.agg(np.median)
goal['sucMedGoal'] = sucGrp.goal.agg(np.median)

plt.close()
ax = goal.plot(kind = 'bar', title='Fundraising Goals by Category')
ax.set_xlabel("Category")
ax.set_ylabel("Fundraising Goal")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>Rates of Success across CategoriesCertain categories have much higher rates of success than others.Comics, publishing, dance, music, and theater all have success rates of 70% or moreAn interesting point is that these categories are tied to an individual creator, that have a dedicated fanbase they can draw fromAdditionally, projects that require a good to be produced (ie technology products, health and fitness equipment, productivity tools) generally score lowerThere's no binding rule for Kickstarter or indiegogo projects to produce the final product, all they require is that creators make a best faith effort. Since there's some inherent risks to these products being produced, it seems like consumers may be weighing that in their backing decisions<jupyter_code>del goal

plt.close()
catCounts = pd.DataFrame()
catCountAll = merged.groupby('fullcats').count()
catCountSuc = suc.groupby('fullcats').count()
catCounts['total_counts'] = catCountAll['source']
catCounts['success_counts'] = catCountSuc['source']
catCounts['successRate'] =  catCounts['success_counts']/catCounts['total_counts']
catCounts = catCounts.sort_values(by = ['successRate'], axis = 0, ascending = False)

ax = catCounts.successRate.plot(kind = 'bar', title='Rates of Success vs. Project Category')
ax.set_xlabel("Category")
ax.set_ylabel("Success Rate")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>Funding success across categoriesThe most popular categories generally have the lowest success rates, while things like comics, design photography, and games seem to perform better than average.These categories would likely typically have fanbases that closely follow the creators. Due to this, they have a reliable audience to draw from when try to source funds for their campaigns<jupyter_code># Look at the number of successfully funded projects in each category vs. the total number of projects
del catCounts

counts = pd.DataFrame()
counts['Total'] = grp.size()
counts['Successful'] = sucGrp.size()

plt.close()
ax = counts.plot(kind = 'bar', title='Comparison of Total # of Project and # of Successful Projects by Category')
ax.set_xlabel("Category")
ax.set_ylabel("Number of Projects")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>Success rate of projects above a certain amount raisedBecause of the large differences between the average funds raised and the median funds raised, it seems like there would be a large number of projects that raise hardly any money at all. Let's take a closer look project success rate versus the amount of money raised<jupyter_code>del counts

minRaised = [1, 25, 50, 100, 200, 300, 400, 500, 750, 1000, 2000, 5000]
lowGoals = [1, 25, 50, 100, 200, 300, 400, 500, 750, 1000, 2000, 5000]
succRate = []
goalRate = []
lowGoalSuccess = []
for raised in range(len(minRaised)):
    gotMoney = merged[(merged['usd_pledged'] >= minRaised[raised - 1]) & (merged['usd_pledged'] <= minRaised[raised])]
    gotMoneySuccess = gotMoney[(gotMoney['state'] == 'successful') | (gotMoney['state'] == 'finished')]
    if len(gotMoney) == 0:
        tempRate = 0
    else:
        tempRate = len(gotMoneySuccess) / len(gotMoney)
    succRate.append(tempRate)

for goals in range(len(lowGoals)):
    goalFilter = merged[(merged['goal'] <= lowGoals[goals]) & (merged['goal'] >= lowGoals[goals - 1])]
    lowSuccess = goalFilter[(goalFilter['state'] == 'successful') | (goalFilter['state'] == 'finished')]
    if len(lowSuccess) == 0:
        tempLowSuccess = 0
    else:
        tempLowSuccess = len(lowSuccess) / len(goalFilter)
    tempGoals = len(goalFilter) / len(merged)
    goalRate.append(tempGoals)
    lowGoalSuccess.append(tempLowSuccess)
    
graph = pd.DataFrame({'Dollars' : minRaised,
                     'Minimum_Raised' : succRate,
                     '%_of_Projects' : goalRate,
                     'Success_by_Goal' : lowGoalSuccess})
x = graph['Dollars']
y1 = graph['Minimum_Raised']
y2 = graph['%_of_Projects']
y3 = graph['Success_by_Goal']

plt.close()
plt.plot(x, y1, x, y2, x, y3)
plt.legend(('By Minimum Raised', 'Cumulative % of Project Goals < X', 'By Funding Goal'),
           loc='lower right')
plt.xlabel('Dollars')
plt.ylabel('Relative Frequency')
plt.title('Funding Goals and Success Rates')<jupyter_output><empty_output><jupyter_text>Funding Goals, Minimum Amount Raised, and Success RatesIn the plot above:'By Minimum Raised' Shows relative frequency of success in projects that raised at least the amounts on the x axis'Cumulative % of Project Goals' shows the % of projects that have funding goals less than the amounts on the x axis'By Funding Goal' shows relative frequency of successful projects that have goals lower than amounts on the x axisWhile a small % of projects have funding goals less than $1,000, projects that raised up to that amount have an outsized chance of meeting their funding goal. This idea seems to stress the importance of finding ambitious "first movers" from the general public to pledge support to projects and share with their friends. This supports the idea that project categories which have fanbases that closely follow project creators have a much better chance at reaching their funding goals. Let's take a closer look at the number of backers  Time between Project Creation and Project LaunchKickstarter allows creators to create a landing page for their projects before the project is officially launched. This time is for creators to raise awareness about the funding campaign, field questions from potential donors and tweak the campaign before making it liveThere's a clear relationship showing that a delay of between 20 and 30 days gives the most benefit<jupyter_code>del gotMoney
del gotMoneySuccess
del goalFilter
del lowSuccess
del graph

# Plot the delta between launch and creation date vs. success rate
# Plot the campaign length against success rate
launchCreate = [1, 5, 10, 15, 20, 25, 30, 40, 45, 50, 60]
launchSuccess = []
for i in range(len(launchCreate)):
    longTemp = merged[(merged['creLauDelta'] <= launchCreate[i]) & (merged['creLauDelta'] >= launchCreate[i - 1])]
    longTempSuccess = longTemp[(longTemp['state'] == 'successful') | (longTemp['state'] == 'finished')]
    if len(longTemp) == 0:
        tempRate = 0
    else:
        tempRate = len(longTempSuccess) / len(longTemp)
    launchSuccess.append(tempRate)

graph = pd.DataFrame({'creLauDelta' : launchCreate,
                     'Success_by_Delta' : launchSuccess})
x = graph['creLauDelta']
y1 = graph['Success_by_Delta']

plt.close()
plt.plot(x, y1)
plt.ylim(bottom=0)
plt.xlabel('Number of Days')
plt.ylabel('Relative Frequency of Success')
plt.title('Success Rates vs. Number of Days between Launch and Creation')<jupyter_output><empty_output><jupyter_text>Project Launch Date and Success Rate Along with that, projects being successfully funded seems to have dropped off slightly in the last few years<jupyter_code>del longTemp
del longTempSuccess

noOutliers = merged[merged['funds_raised_percent'] < 200]
x = noOutliers['launched_at']
y = noOutliers['funds_raised_percent']

plt.close()
plt.hexbin(x, y, gridsize=10)
plt.xlabel('Launch Date (in Epoch time)')
plt.ylabel('Percent Raised of Goal')
plt.title('Launch Date vs. Funds Raised Percent')<jupyter_output><empty_output><jupyter_text>Alexa Site Rank and Success RateLook at the success rate of campaign funding and the Alexa site rankingThere seems to be a slight increase in projects reaching their funding goals when Kickstarter is more popular than other websites in general<jupyter_code>noOutliers = merged[(merged['funds_raised_percent'] < 200)]
noOutliers.dropna(axis = 0, inplace = True)

x = noOutliers['kickstarter.com']
y = noOutliers['funds_raised_percent']

plt.close()
plt.figure()

plt.hexbin(x, y, gridsize=10)
plt.xlabel('Alexa Site Traffic Ranking')
plt.ylabel('Percent Raised of Goal')
plt.title('Alexa Site Ranking vs. Funds Raised Percent')<jupyter_output>/Users/chrismay/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy<jupyter_text>The Importance of Creators in Project Funding SuccessIn the previous section, there seemed to be a relationship between categories that have dedicated fanbases and an increaesd success rate in reaching funding goals.To further explore that relationship, make some comparisons grouping projects by the number of other projects a certain creator has launchedCount the number of projects that a certain creator ID has launched, and then add that as a column to the dataframe<jupyter_code>creaGroup = pd.DataFrame()
creators = pd.DataFrame()

plt.close()
# Group all projects by creator id and then merge that into the dataframe
creaGroup = merged.groupby('creator').count()
grouped = creaGroup.sort_values(by = ['backers_count'])
grouped = grouped.reset_index()
creators['creator'] = grouped['creator']
creators['no_projects'] = grouped['backers_count']
creators.head()
merged = merged.merge(creators, how = 'left', on = 'creator')<jupyter_output><empty_output><jupyter_text>Relationship between number of projects launched and success rateThere's a small subset of creators that have launched multiple projects. From the chart below, there's an increased rate of success as the number of projects launched by a particular creator increases. Note that this plot is shown by total number of projects, and does not consider the order in which projects were launched by a particular creator<jupyter_code>noProjects = [1, 2, 4, 8, 16, 32, 60, 72]
suc = merged[(merged['state'] == ('successful')) | (merged['state'] == ('finished'))]
multi = []
for i in noProjects:
    tempProj = merged[(merged['no_projects'] <= (i)) & (merged['no_projects'] >= (i - 1))]
    tempSucProj = suc[(suc['no_projects'] <= (i)) & (suc['no_projects'] >= (i - 1))]
    if len(tempProj) != 0:
        sucRate = len(tempSucProj) / len(tempProj)
    else:
        sucRate = 0
    multi.append(sucRate)

multiGraph = pd.DataFrame()
multiGraph['no_projects'] = noProjects
multiGraph['success_rate'] = multi
               
x = multiGraph['no_projects']
y1 = multiGraph['success_rate']

plt.close()
plt.plot(x, y1)
plt.ylim(bottom=0)
plt.xlabel('Number of Projects')
plt.ylabel('Relative Frequency')
plt.title('Number of Creator Projects and Success Rate')<jupyter_output><empty_output><jupyter_text>This is where I left off with the memory reduction Number of projects launched and percent of goal raisedThere also seems to be an increase in average % of goal raised as the number of projects increases. Above 15, the sample size for each category drastically decreases (ie the number of creators that launched more than 15 projects is very small.<jupyter_code># Compare the money raised by succesfull projects against all projects in a certain category
projects= pd.DataFrame()
proGroup = merged.groupby(['no_projects'])
sucProGrp = suc.groupby(['no_projects'])
#projects['Avg%'] = proGroup.funds_raised_percent.agg(np.mean)
#projects['SucAvg%'] = sucProGrp.funds_raised_percent.agg(np.mean)
projects['Median%'] = proGroup.funds_raised_percent.agg(np.median)
projects['SucMed%'] = sucProGrp.funds_raised_percent.agg(np.median)
#money.reset_index(inplace = True)

plt.close()
ax = projects.plot(kind = 'bar', title='Percent of Funds Raised by Number of Creator Projects')
ax.set_xlabel("Total Number of Projects by Creator")
ax.set_ylabel("Percent of Funds Raised")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>Number of Projects launched by each creator by categoryVisual and performing arts projects are most frequently have creators with a high number of projects. The median number of projects for each creator in each category is 1, so creators with multiple project history is a minority segment of the overall sample population<jupyter_code>create = pd.DataFrame()
createGrp = merged.groupby(['fullcats'])
create['no_projects'] = createGrp.no_projects.agg(np.mean)
create['no_projects_median'] = createGrp.no_projects.agg(np.median)
create.dropna(axis = 0, inplace = True)

plt.close()
ax = create.plot(kind = 'bar', title='Number of Projects per Creator by Category')
ax.set_xlabel("Category")
ax.set_ylabel("Avg. Number of Projects per Creator")
plt.figure(figsize=(600,300))<jupyter_output><empty_output><jupyter_text>The total number of projects vs. creators with multiple projectsThe next plot shows the number of projects in each category and compares that with the number of projects with creators who have launhed multiple proejcts.It's easy to see in this plot that most creators in this dataset have only launched one project<jupyter_code>createNumber = pd.DataFrame()
multi = merged[merged['no_projects'] > 1]
createGrp = merged.groupby(['fullcats'])
multiGrp = multi.groupby(['fullcats'])
createNumber['all_projects'] = createGrp.no_projects.size()
createNumber['multi_projects'] = multiGrp.no_projects.size()
createNumber.dropna(axis = 0, inplace = True)

plt.close()
ax = createNumber.plot(kind = 'bar', title='Number of Projects per Creator by Category')
ax.set_xlabel("Category")
ax.set_ylabel("Avg. Number of Projects per Creator")
plt.figure(figsize=(600,300))
createNoProjects = pd.DataFrame()
noProjects = merged.groupby(['no_projects'])
createNoProjects['funding_percent_avg'] = noProjects.funds_raised_percent.agg(np.mean)
createNoProjects['funding_percent_med'] = noProjects.funds_raised_percent.agg(np.median)
createNoProjects['staLauDelta_avg'] = noProjects.staLauDelta.agg(np.mean)
createNoProjects['staLauDelta_med'] = noProjects.staLauDelta.agg(np.median)
createNoProjects.reset_index(inplace = True)

x = createNoProjects['no_projects']
y1 = createNoProjects['funding_percent_avg'] = noProjects.funds_raised_percent.agg(np.mean)
y2 = createNoProjects['funding_percent_med'] = noProjects.funds_raised_percent.agg(np.median)

plt.close()
plt.plot(x,y2)
plt.legend(('Median Funding Percent', 'Median Funding Percent'), loc='upper right')
plt.xlabel('Number of Projects by Creator')
plt.ylabel('% of Goal Raised')
plt.ylim(bottom=0)
plt.ylim(top = 400)
plt.xlim(right=15)
plt.xlim(left=1)
plt.title('% of Goal Raised and Number of Projects by Creator')
createNoProjects['usd_pledged_avg'] = noProjects.usd_pledged.agg(np.mean)
createNoProjects['usd_pledged_med'] = noProjects.usd_pledged.agg(np.median)

x = createNoProjects['no_projects']
y1 = createNoProjects['usd_pledged_avg']
y2 = createNoProjects['usd_pledged_med'] 

plt.close()
plt.plot(x,y1,x,y2)
plt.legend(('Average Dollars Raised', 'Median Dollars Raised'), loc='upper right')
plt.xlabel('Number of Projects by Creator')
plt.ylabel('Dollars Raised')
plt.ylim(bottom=0)
plt.xlim(right=15)
plt.xlim(left=1)
plt.title('Dollars Raised and Number of Projects by Creator')
createNoProjects['staLauDelta_avg'] = noProjects.staLauDelta.agg(np.mean)
createNoProjects['staLauDelta_med'] = noProjects.staLauDelta.agg(np.median)

x = createNoProjects['no_projects']
y1 = createNoProjects['staLauDelta_avg']
y2 = createNoProjects['staLauDelta_med']

plt.close()
plt.plot(x,y1,x,y2)
plt.legend(('Average Days to Goal', 'Median Days to Goal'), loc='upper right')
plt.xlabel('Number of Projects by Creator')
plt.ylabel('Days to Goal')
plt.ylim(bottom=0)
plt.xlim(left=2)
plt.xlim(right=15)
#plt.ylim(ymax = 2000)
plt.title('Days to Goal and Number of Projects by Creator')<jupyter_output><empty_output><jupyter_text>Limiting the Results shown for Number of Projects by CreatorAbove 14 projects per creator, the number of unique sample points drops off sharply to just a few creators. Rather than view results for sample sizes of 1 or 2, the above graphics were limited to creators that made 15 projects or fewer to show a better representation of the population<jupyter_code>merged['no_projects'].value_counts()
fundRate = pd.DataFrame()
rateProjects = merged.groupby(['staLauDelta'])
fundRate['funding_percent_avg'] = rateProjects.funds_raised_percent.agg(np.mean)
fundRate['funding_percent_med'] = rateProjects.funds_raised_percent.agg(np.median)

fundRate.reset_index(inplace = True)

x = fundRate['staLauDelta']
y2 = fundRate['funding_percent_med']

plt.close()
plt.plot(x,y2)
plt.legend(('Median Funding Percent', 'Median Funding Percent'), loc='upper right')
plt.xlabel('Days to Reach Goal')
plt.ylabel('% of Goal Raised')
plt.ylim(bottom=0)

plt.xlim(right=40)
plt.xlim(left=1)
plt.title('% of Goal Raised and Days to Reach Goal')
fundRate['usd_pledged_avg'] = rateProjects.usd_pledged.agg(np.mean)
fundRate['usd_pledged_med'] = rateProjects.usd_pledged.agg(np.median)

x = fundRate['staLauDelta']
y1 = fundRate['usd_pledged_avg']
y2 = fundRate['usd_pledged_med']

plt.close()
plt.plot(x,y1,x,y2)
plt.legend(('Average USD Pledged', 'Median USD Pledged'), loc='upper right')
plt.xlabel('Days to Reach Goal')
plt.ylabel('USD Pledged')
plt.ylim(bottom=0)
plt.xlim(left=0)
plt.xlim(right=40)
plt.title('USD Pledged and Days to Reach Goal')<jupyter_output><empty_output><jupyter_text>What's Happening here?There's an odd abundance of projects that reach their funding goal either very close to, or on the deadline date.Kickstarter.com has a section of their homepage dedicated to projects whose deadlines are within a certain time limitWe don't have data for funds raised per day, but it seems plausible that the presence on the homepage plus the urgency of a project being close to its goal and end date motivates backers on the fence to contributeIt's also plausible that towards the end of a fundraising campaing, projects push more incentive messages to their target audience![Kickstarter's "Home Stretch"](https://i.imgur.com/rCG7hj0.png)  Effects of Staff Picks and Project Spotlights on Campaign SuccessAs mentioned above, Kickstarter highlights certain projects on their homepageThe dataset that I have includes "Staff Pick" tags, for projects that Kickstarter staff favorsThe spotlight tag is a little more ambiguous, but based on the success rate of spotlight projects, it seems like all successfully funded projects are spotlighted at some pointThe success rate for staff pick projects is almost 90%, and the rate for spotlighted projects is 100%<jupyter_code>staff = merged[(merged['staff_pick'] == True) & (merged['source'] == 'Kickstarter')]
staffSucc = staff[(staff['state'] == 'successful')]
spot = merged[(merged['spotlight'] == True) & (merged['source'] == 'Kickstarter')]
spotSucc = spot[(spot['state'] == 'successful')]
unstaff = merged[(merged['staff_pick'] == False) & (merged['source'] == 'Kickstarter')]
unstaffSucc = unstaff[unstaff['state'] == 'successful']
mergeKick = merged[merged['source'] == 'Kickstarter']
staffRate = len(staffSucc)/ len(staff)
spotRate = len(spotSucc)/len(spot)

print(staffRate, spotRate)
staffMoney = pd.DataFrame()
staffGrp = staff.groupby(['fullcats'])
noStaff = unstaff.groupby(['fullcats'])
staffMoney['avg_raised_staff_picks'] = staffGrp.usd_pledged.agg(np.mean)
staffMoney['no_staff_avg'] = noStaff.usd_pledged.agg(np.mean)
staffMoney['med_staff_picks'] = staffGrp.usd_pledged.agg(np.median)
staffMoney['no_staff_med'] = noStaff.usd_pledged.agg(np.median)

plt.close()
ax = staffMoney.plot(kind = 'bar', title='Crowdfunding Funds Raised by Project Category')
ax.set_xlabel("Category")
ax.set_ylabel("Funds Raised")
plt.figure(figsize=(600,300))
unstaff = merged[merged['staff_pick'] == False]
unstaffSucc = unstaff[unstaff['state'] == 'successful']
print('the number of successful projects that were not staff picks is ', len(unstaffSucc))
print('the number of successful projects that are staff picks is ', len(staffSucc))
kickOnly = merged[merged['source'] == 'Kickstarter']
print('the total number of projects is ', len(kickOnly))
staffMoney = pd.DataFrame()
staffMoney['StaffCounts'] = staffGrp.size()
staffMoney['NoStaffCount'] = noStaff.size()

plt.close()
ax = staffMoney.plot(kind = 'bar', title='# of Projects that are Staff Picks by Category')
ax.set_xlabel("Category")
ax.set_ylabel("Number of Projects")
plt.figure(figsize=(600,300))
allKick = mergeKick.groupby(['fullcats'])
noStaffCont = allKick.usd_pledged.agg(np.sum)
staffCont = staffGrp.usd_pledged.agg(np.sum)
contributions = pd.DataFrame()
contributions['staff_picks'] = staffCont / noStaffCont

plt.close()
ax = contributions.plot(kind = 'bar', title='Contributions of Staff Picks to Total Funds Raised (By Category)')
ax.get_legend().remove()
ax.set_xlabel("Category")
ax.set_ylabel("% of Total Funds Raised")
plt.figure(figsize=(600,300))
plt.close()
total = contributions.mean(axis = 0)
print('Kickstarter staff picks accounts for ', total, '% of all funds raised on average')<jupyter_output>Kickstarter staff picks accounts for  staff_picks    0.451525
dtype: float64 % of all funds raised on average<jupyter_text>Correlation between VariablesPandas has a method to do a pairwise correlation calculation for all pairs of columns that have numerical dataThe only semi-strong correlated pairs are between the total number of backers and the total USD pledged. There's some other weaker correlations here, but in reality there's probably too many confounding factors to give a strong correlation.Factors that are probably affecting the correlations- Differences in required funding goals to bring projects to fruition across categories- Varying scales of projects- Prior creator history- Individual backer tastes<jupyter_code>select = pd.DataFrame()
select['no_projects'] = merged['no_projects']
select['funds_raised_percent'] = merged['funds_raised_percent']
select['fullcats'] = merged['fullcats']
select['kickstarter.com'] = merged['kickstarter.com']
select['goal'] = merged['goal']
select['backers_count'] = merged['backers_count']
select['staLauDelta'] = merged['staLauDelta']
select['usd_pledged'] = merged['usd_pledged']
corr = select.corr()

plt.close()
sb.heatmap(corr, annot=True)
plt.title('Correlation Heatmap')<jupyter_output><empty_output><jupyter_text>Appendix A: Data Descriptions| Columns | Data Type | Description | |---|---|---||Backers Count| int64 | Total number of individual backers of a project|Blurb|object| The short description of the project|Category| list | A list of categories that the project has been tagged under|Country| string| Two character country code of the campaign’s headquarters|Created_at|int64| The creation date of the campaign in epoch time|Creator| string |A list of the creator’s name and their creator ID on Kickstarter.com|Currency| string| A two character string of the domestic currency of the project|Deadline| int64|The end date/time of the campaign in epoch time|Goal| int64|The funding goal of the project in native currency|Id| int64|The unique project ID|Launched_at| int64|Launch date of the campaign in epoch time|Location| string|The main location of the project|Name| object|The name of the project|Pledged| int64|Total amount pledged (in the project’s home currency)|Slug| string|An abbreviated description of the project|Spotlight|string|True/False categorical variable of whether the project was featured by Kickstarter|Staff_pick|string|True/False categorical variable of whether the project was a staff pick or not|State|string|Successful/Failed/Canceled/Active  categorical variable of the projects current status|State_changed_at| int64|Time/date that the project changed states, in epoch time|Static_usd_rate| int64|Static conversion rate between home currency and USD|Urls| string|Various URL’s associated with the project|Usd_pledged| int64|Total amount pledged, converted to USD  Appendix B: Previous Work and Citations[Good Audience](https://blog.goodaudience.com/kickstarter-projects-prediction-of-state-steps-for-a-beginner-analysis-f4630a50b7fe)•	Runs logistic regression, XGBoost, Random forests, LightGBM and an Ensemble method to try and predict whether a project will be successfully funded or not. Based on the Kaggle dataset (limited to 49k entries). Mainly focused on feature selection[Towards Data Science](https://towardsdatascience.com/predicting-the-success-of-kickstarter-campaigns-3f4a976419b9)•	Again predictions on successful funding, based on funding goals, category, rewards levels, and location data. Uses random forests, logistic regression, and kNearest Neighbors[Bentley University](https://www.bentley.edu/prepared/what-secret-perfect-kickstarter-campaign)•	Finds length of description and title are important•	Also the date of the launch important (winter is the best month)•	Higher average donations, shorter campaign durations<jupyter_code>merged.head()
alexa = merged.copy()
alexa.dropna(axis = 0, inplace = True)

x = alexa['launched_at']
y = alexa['kickstarter.com']

plt.close()
plt.scatter(x, y)
plt.legend(('Alexa Ranking', 'Alexa ranking'), loc='lower right')
plt.xlabel('Date (Epoch time)')
plt.ylabel('Alexa Ranking')
plt.ylim(bottom=1)
plt.title('Kickstarter.com Alexa Ranking over Time')
plt.close()
merged.columns
x = merged[(merged['funds_raised_percent'] < 200)]
y = x['funds_raised_percent']

plt.close()
plt.hist(y, bins = 100)
plt.xlabel('Percent of Goal Raised')
plt.ylabel('Number of Occurrences')
plt.title('Distribution of % of Goal Raised')
x = merged[merged['state'] == 'successful']
y = x['staLauDelta']

plt.close()
plt.hist(y, bins = 100)
plt.xlabel('Days to Reach Goal')
plt.ylabel('Number of Occurrences')
plt.xlim(left=0)
plt.xlim(right=60)
plt.title('Distribution of # of Days to Reach Goal')
from sklearn import preprocessing as pre
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
from sklearn.model_selection import GridSearchCV
mergeKick.columns
# Make a new dataframe with only the columns that we'll use for modeling
kickModel = pd.DataFrame()
kickModel['country'] = mergeKick['country']
kickModel['goal'] = mergeKick['goal']
kickModel['id'] = mergeKick['id']
kickModel['launched_at'] = mergeKick['launched_at']
kickModel['staff_pick'] = mergeKick['staff_pick']
kickModel['alexa_rank'] = mergeKick['kickstarter.com']
kickModel['creator_projects'] = mergeKick['no_projects']
kickModel['state'] = mergeKick['state']
kickModel['creLauDelta'] = mergeKick['creLauDelta']
kickModel['category'] = mergeKick['fullcats']

kickModel['staff_pick'] = kickModel.apply(lambda x: int(x['staff_pick'] == True), axis = 1)
kickModel['state'] = kickModel.apply(lambda x: int(x['state'] == 'successful'), axis = 1)
kickModel.head()
dummies1 = pd.get_dummies(kickModel['category'], columns = 'category')
dummies2 = pd.get_dummies(kickModel['country'], columns = 'country')
kickModel = pd.concat([kickModel, dummies1], axis = 1)
kickModel = pd.concat([kickModel, dummies2], axis = 1)
kickModel.drop(labels = ['category', 'country'], axis = 1, inplace = True)
kickModel.head()
kickModel = kickModel.fillna(value = 1000, axis = 0)
X = kickModel.loc[:, kickModel.columns != 'state']
y = kickModel.loc[:, kickModel.columns == 'state']

#Use a MinMax scaler to standardize features
scaler = MinMaxScaler(feature_range = (0,1))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
logreg = LogisticRegression()

scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))
cm = confusion_matrix(y_test, y_pred)
print(confusion_matrix)
logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09539740671087796, 'fraction_numerical': 0.011145204394734237, 'mean_word_length': 5.404158215010142, 'pattern_counts': {'":': 5, '<': 136, '<?xml version=': 0, '>': 133, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '104220', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 15636, 'n_tokens_neox': 14285, 'n_words': 5357, 'path': 'pezLyfe/ChrisMay_ProjectUpdate2_kickstarterAnalysis.ipynb', 'repo_name': 'fimofimo/applied_ds', 'size': '50515'}