<jupyter_start><jupyter_text>COSC 301 Milestone 2 <NAME> & <NAME> Data Analysis of Kickstarter Projects<jupyter_code>import pandas as pd
import seaborn as sns
import numpy as np
import sys
import os
import datetime
import matplotlib.pyplot as plt

# For how to import the scripts folder so that it can be used:
# https://stackoverflow.com/questions/53049195/importing-custom-module-into-jupyter-notebook
# https://stackoverflow.com/questions/49264194/import-py-file-in-another-directory-in-jupyter-notebook
sys.path.append('../scripts')

import project_functions
# relative import did not work for some reason
#from .. import project_functions

pathToData = '../../data/raw/ks-projects-201801.csv'

dfj = project_functions.load_and_process(pathToData, 'Jacob')
dfr = project_functions.load_and_process(pathToData, 'Rylan')<jupyter_output><empty_output><jupyter_text>Task 3 | Exploratory Data Analysis | Jacob The info function gives information on all the columns and there associated types. The result shows that the 'load_and_process' function of the data pipeline has worked correctly as the new **'stateInt'** (int64), **'duration'** (timedelta64)\[ns\], and **'durationInt'** (int 64) columns needed for my analysis have been created with the correct types.<jupyter_code>dfj.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 378657 entries, 0 to 378660
Data columns (total 16 columns):
 #   Column            Non-Null Count   Dtype          
---  ------            --------------   -----          
 0   Name              378657 non-null  object         
 1   category          378657 non-null  object         
 2   main_category     378657 non-null  object         
 3   currency          378657 non-null  object         
 4   deadline          378657 non-null  object         
 5   launched          378657 non-null  object         
 6   pledged           378657 non-null  float64        
 7   state             378657 non-null  object         
 8   backers           378657 non-null  int64          
 9   country           378657 non-null  object         
 10  usd_pledged_real  378657 non-null  float64        
 11  usd_goal_real     378657 non-null  float64        
 12  stateInt          378657 non-null  int64          
 13  duration          378657 non-null  timedelta[...]<jupyter_text>This shows how many outlier projects there are in the dataset, for this I am considering projects with duration greater than 90 days to be outliers since there are relatively few greater than that.<jupyter_code>outlierProjects = dfj[dfj['durationInt'] > 90]
print(f"There are {outlierProjects.shape[0]} projects in the dataset with duration greater 90 days out of a total of {dfj.shape[0]} projects")
dfj.describe().T<jupyter_output><empty_output><jupyter_text>These two plots below show the distribution of Kickstarter projects based on the duration of the project by count and probability. Interestingly, this shows that 1 month long projects are the majority of projects composing about 47% of all projects. 2 month long and 1.5 month long projects are the next most common.<jupyter_code>axes = sns.displot(data=dfj[dfj['durationInt'].between(0, 90)], x='durationInt', aspect=2.0, bins=50).set(title='Distribution of Kickstarter Projects by Count')
#axes = sns.displot(data=df, x='durationInt', aspect=2.0)
axes.set_axis_labels('Duration of Kickstarter Project', 'Count')
# https://seaborn.pydata.org/generated/seaborn.FacetGrid.html#seaborn.FacetGrid
axes.set(xticks=np.arange(0, 90, 5))


# Learned how to use 'stat' parameter to get bins by probability and not count from user miro from here:
# https://stackoverflow.com/questions/40092294/creating-a-matplotlib-or-seaborn-histogram-which-uses-percent-rather-than-count
axes = sns.displot(data=dfj[dfj['durationInt'].between(0, 90)], x='durationInt', aspect=2.0, bins=50, stat='probability').set(title='Distribution of Kickstarter Projects by Probability')
axes.set_axis_labels('Duration of Kickstarter Project', 'Probability')
axes.set(xticks=np.arange(0, 90, 5), yticks=np.arange(0,1.1,0.1))<jupyter_output><empty_output><jupyter_text>Here is another interesting plot which shows the distribution of the different possible states of a project. Since my research question is focused on how the duration of a project affects it success, I am going to focus on projects which have the states of 'failed/canceled' or 'successful'. The other states do not provide information that I need about the success so they will be filtered out.<jupyter_code>sns.countplot(x='state', data=dfj)
plt.title('Distribution of Project States')<jupyter_output><empty_output><jupyter_text>This graph here shows how the duration of projects is distributed by category which helps show that the category may affect my analysis and research question<jupyter_code># Learned how to change seaborn plot size from user 'student' here:
# https://stackoverflow.com/questions/31594549/how-do-i-change-the-figure-size-for-a-seaborn-plot
sns.set(rc={'figure.figsize':(16,8.27)})
box1 = sns.boxplot(data=dfj[dfj['durationInt'].between(0, 90)], y='durationInt', x='main_category')
box1.set(yticks=np.arange(0,90,5))
print()<jupyter_output><jupyter_text>Here's a pair plot that shows plots between duration, percent funded, and project goal amount. Interestingly as duration increases project funding goal seems to increase as well.<jupyter_code>dfWithOutliersRemoved = dfj[dfj['durationInt'].between(0, 90) & dfj['percentFunded'].between(0, 200) & dfj['usd_goal_real'].between(0, 1000000)].head(450)
print(dfWithOutliersRemoved.shape)
sns.pairplot(dfWithOutliersRemoved[['durationInt', 'usd_goal_real', 'percentFunded']], height=2, corner=True)<jupyter_output>(450, 16)<jupyter_text>Task 3 | Exploratory Data Analysis | Rylan<jupyter_code>dfr.shape<jupyter_output><empty_output><jupyter_text>First thing I did was take a look at the shape of my data, seeing I have six columns and 159,141 rows.<jupyter_code>dfr.head()<jupyter_output><empty_output><jupyter_text>Next I printed the head of the dataset to get a visualization of what the set of data lookslike.<jupyter_code>dfr.columns<jupyter_output><empty_output><jupyter_text>I also print out the categories to see what they are.<jupyter_code>dfr.nunique(axis=0)<jupyter_output><empty_output><jupyter_text>I then used nunique to see how many unique values I have finding that I have 15 categories and 3 different states which is as it should be from my processing step.<jupyter_code>dfr.describe().apply(lambda s: s.apply('{0:.5f}'.format))
#https://stackoverflow.com/questions/40347689/dataframe-describe-suppress-scientific-notation/47207283<jupyter_output><empty_output><jupyter_text>I then call describe for each possible category to see what the count, mean, and other relevant stats are<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='pledged',estimator=np.sum)
#https://stackoverflow.com/questions/31594549/how-do-i-change-the-figure-size-for-a-seaborn-plot
#https://www.youtube.com/watch?v=6GUZXDef2U0&ab_channel=DerekBanas<jupyter_output><empty_output><jupyter_text>This plot shows the total sum of pledges per category.<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='pledged',estimator=np.mean)<jupyter_output><empty_output><jupyter_text>This graph shows the average pledged amount per category.<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='plus_minus',estimator=np.sum)<jupyter_output><empty_output><jupyter_text>This plot shows the sum of plus-minus per category<jupyter_code>sns.countplot(x='category',data=dfr)<jupyter_output><empty_output><jupyter_text>This plot shows the amount of projects per category.<jupyter_code>dfr['category'].value_counts().plot.pie()
#https://www.dataforeverybody.com/matplotlib-seaborn-pie-charts/<jupyter_output><empty_output><jupyter_text>This plot is similar to the plot above but visualized differently.<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='goal',estimator=np.mean)<jupyter_output><empty_output><jupyter_text>Lastly this plot shows the sum of the plus-minus per category  Task 4 | Research Question Analysis | <NAME>'s research question: Does the funding duration of a Kickstarter project affect its success? --- One way to test this research question is to create a correlation matrix where the correlation between each pair of numeric variables in the dataset can be seen.<jupyter_code># Learned how to change seaborn plot size from user 'student' here:
# https://stackoverflow.com/questions/31594549/how-do-i-change-the-figure-size-for-a-seaborn-plot
sns.set(rc={'figure.figsize':(10,7)})

corr = dfj.corr()
sns.heatmap(corr, annot=True, cmap='winter')<jupyter_output><empty_output><jupyter_text>In this plot above, I am interested in the correlation between how funded the project was (percentFunded) and the duration of the project (durationInt). Here it shows a correlation of -0.024 which is a first indicator that the duration of the Kickstarter project may not affect its success. Here is another plot which helps show visually that duration of the project may actually be related to the success of the project. It shows that projects that have a really short duration have a smaller percent funded and the percent funded peaks around ~30 days and then gradually decreases until increasing again at 60 days. Interestingly we can see that the amount of projects increases around 100%, but before 100% the amount of projects is not as many. I think this may be because once a project is close to being funded (80%-90%) it may be that more backers will fund the project to push it to reach 100% since it is so close to reaching its goal.<jupyter_code>df2 = dfj[dfj['percentFunded'].between(0, 200) & dfj['durationInt'].between(0, 90)].head(50000)
sns.jointplot(data=df2, x='durationInt', y='percentFunded')<jupyter_output><empty_output><jupyter_text>Here is one of the main plots for this investigation. It shows the probability of success of the project versus how long the project's funding time was. It shows that there indeed is a variance in the success of projects based on how long they run. Interestingly, there are two main peaks for highest success. One at around ~10 days and another at ~66 days.<jupyter_code># Plot probability of success vs. project funding time length
df2 = project_functions.prepareDfForProbSuccessVSDuration(dfj)
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html?highlight=plot#pandas.DataFrame.plot
#df2.plot(kind='bar', x='formattedTimeIntervals', y='stateInt', ylabel='Probability of Success', xlabel='Duration of Project', ylim=(0,1), legend=False, title='Probability of Success vs. Project Funding Time Length')
ax = sns.barplot(data=df2, x='formattedTimeIntervals', y='stateInt')
plt.title('Probability of Success vs. Project Funding Time Length')
plt.xlabel('Duration of Project')
plt.ylabel('Probability of Success')
plt.yticks(np.arange(0,1.1,0.1))
# for help rotating x_tick labels:
# https://stackoverflow.com/questions/61368851/how-to-rotate-seaborn-barplot-x-axis-tick-labels
ax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right')
print()<jupyter_output><jupyter_text>This part gets the probability of success for projects that have a funding time in between 63 and 70 days and plots it by category. Here it shows that probability of success varies with the category, so the category may also affect how a project's success varies with funding duration.<jupyter_code>plt.figure(figsize=(8,5))

dfNew = project_functions.getProjectsWithDurationInTimeRange(dfj, 63, 70)
dfNew = project_functions.getProjectsAsFailedOrSuccessful(dfNew)
#print(dfNew.nunique(axis=0))
#print(f"mean: {dfNew['stateInt'].mean()}")
dfTest = dfNew.groupby('main_category')['stateInt'].mean().plot(kind='bar').set(title='Probability of Success for Projects with \nDuration Between 63 and 70 days by Category')
plt.yticks(np.arange(0, 1.1, 0.1))
plt.xlabel('Category')
plt.ylabel('Probability of Success')
print()<jupyter_output><jupyter_text>Here the plot is for projects between 0 and 7 days of funding duration. Here we see that fashion projects' probabilty of success has increased but crafts' has decreased, so category may affect how duration of funding affect's a project's success.<jupyter_code>plt.figure(figsize=(8,5))
dfNew2 = project_functions.getProjectsWithDurationInTimeRange(dfj, 0, 7)
dfNew2 = project_functions.getProjectsAsFailedOrSuccessful(dfNew2)
dfTest2 = dfNew2.groupby('main_category')['stateInt'].mean().plot(kind='bar').set(title='Probability of Success for Projects with \nDuration Between 0 and 7 days by Category')
plt.yticks(np.arange(0, 1.1, 0.1))
plt.xlabel('Category')
plt.ylabel('Probability of Success')
print()<jupyter_output><jupyter_text>The plot below takes the information learned above and shows that projects in different categories do vary slightly in their success relative to their funding duration with some going down when others go up. However overall they do actually follow a similar curve relative to funding duration.<jupyter_code>import matplotlib.pyplot as plt

# Learned how to change seaborn plot size from user 'student' here:
# https://stackoverflow.com/questions/31594549/how-do-i-change-the-figure-size-for-a-seaborn-plot
sns.set(rc={'figure.figsize':(16,8.27)})

dfByCategory = project_functions.prepareDfForProbSuccessVSDurationByCategories(dfj)

a = sns.barplot(data=dfByCategory, x='duration2', y='stateInt', hue='category2')
# for help rotating x_tick labels:
# https://stackoverflow.com/questions/61368851/how-to-rotate-seaborn-barplot-x-axis-tick-labels
a.set_xticklabels(a.get_xticklabels(), rotation=45, horizontalalignment='right')
# https://stackoverflow.com/questions/31632637/label-axes-on-seaborn-barplot
a.set(xlabel='Funding Time Length', ylabel='Probability of Project Success')

#plt.legend(loc='upper left')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title('Probability of Project Success vs Funding Duration By Category')<jupyter_output>Index(['stateInt', 'duration2', 'category2'], dtype='object')<jupyter_text>Based on the inital correlation matrix, it seemed as though there wasn't a relationship between the project funding duration and the probability of the project's success. **However, after ploting the probability of success vs project duration a relationship between the two variables was found, although non-linear**. Because the relationship is non-linear that may be why the correlation matrix did not show a relationship between the two variables. Based on the graphs, a project success peaks if it has a > 7 day < 30 days duration. This makes sense as if a project doesn't have too long to be funded it may incentivised a potential backer to back right away lessening the probability that they forget about the project. Also project success dips somewhat lower if it is < 7 days and this also makes sense. If a project only has a funding duration of 1 week less people are likely to be able to see it and the news of the project may not reach enough people. Lastly, projects in different categories had overall different probabilities of success, however they did follow similar curves in their probability of success relative to funding duration.  Task 4 | Research Question Analysis | <NAME>'s research question: Which categories have the most successful dollar spent? To answer this question the first thing I looked into what how much money was spent in each category. That produced these two graphs comparing the average pledge total and the sum pledged totals.<jupyter_code>sns.color_palette("husl", 8)
sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='pledged',estimator=np.sum)
plt.title('Sum of Pledge Totals per Category')
plt.ylabel('Pledge Sums')
plt.xlabel('Categories')
sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='pledged',estimator=np.mean)
plt.title('Pledge Averages per Category')
plt.ylabel('Pledge Averages')
plt.xlabel('Categories')<jupyter_output><empty_output><jupyter_text>With these graphs we are now able to tell how much money is put into each category. Technology having the highest average pledge total while design has the most money pledged. The issue with this is this does not tell us how much **successful** money was spent. In order to do that I created a Plus Minus category that was a positive pledge total if the campaign was a success, but was a negative pledge total if it was a failure. The graph below shows the sums of the Plus Minus category.<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='plus_minus',estimator=np.sum)
plt.title('Pledge Total Plus Minus per Category')
plt.ylabel('Pledge Total Plus Minus')
plt.xlabel('Categories')<jupyter_output><empty_output><jupyter_text>Looking at this it is easy to see that the largest amount of successful dollars is spent on Design, Games, Technology, and Film & Video in that order. Although, this does not tell us what would give the user the highest success chance in spending their money. If we look at the graphs below we will see that there is a disproportionate amount of campaigns per category that could change the outcome.<jupyter_code>dfr['category'].value_counts().plot.pie()
plt.title('Campagins per Category')
plt.ylabel('')<jupyter_output><empty_output><jupyter_text>By looking at the pie chart we can discern that the largest amount of Campaigns on Kickstarter are Film & Video, but I will also draw your attention to the fact that design has more campaigns total than technology does. So now if we look at the average of these pledges we can determine which category has the highest average Plus Minus, or amount of succesful dollars per campaign.<jupyter_code>sns.set(rc={'figure.figsize': (15,10)})
sns.barplot(data=dfr,x='category',y='plus_minus',estimator=np.mean)
plt.title('Average Plus Minus per Category')
plt.ylabel('Average Plus Minus')
plt.xlabel('Categories')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09867695700110253, 'fraction_numerical': 0.026901874310915105, 'mean_word_length': 5.579978237214363, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '538817', 'license': 'MIT', 'n_tokens_mistral': 5824, 'n_tokens_neox': 5331, 'n_words': 1841, 'path': 'analysis/submitted/Final.ipynb', 'repo_name': 'data301-2021-summer2/project-group46-project', 'size': '18160'}