<jupyter_start><jupyter_text>Linear Regression<jupyter_code>import statsmodels.api as sm #using statsmodels as opposed to sklearn due to regression output
    
model1 = sm.OLS(y1, X1)
results1 = model1.fit()
print(results1.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:       attendance_count   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 4.466e+30
Date:                Wed, 28 Feb 2018   Prob (F-statistic):               0.00
Time:                        14:21:41   Log-Likelihood:                 29598.
No. Observations:                1149   AIC:                        -5.919e+04
Df Residuals:                    1146   BIC:                        -5.918e+04
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
=======================================================================================
                          coef    std err  [...]<jupyter_text>What does this mean? ,Our model has estimated that there is a negative relationship (which is statistically significant) between the number of interested students and those that attend the events. This is somehwhat counter-intuitive, we have found that we would expect event attendance to *decrease* rather than increase as more people click interested.We have also found that there is a positive relationship between total invites and event attendance, aswell as a negative relationship between how many people do not reply and event attendance. These however are fairly intuitive conclusions.  Can we explain our results?Let's assume that the only people who could potentially come to these events are the ones invited on Facebook. So for each event, we have some part of the student body that is 'eligible' to come to these events. Within that body there are 2 subsets of people, those who are 'favourable' to the event, and those who aren't. Those who aren't are the no-replyers, they do not really ever consider going whereas the 'favourable' group is comprised of people who click attending and interested -- these are the people that who are actually thinking about going to these events. You can only choose to click either going or interested in an event on Facebook, and so it is sort of like a zero sum game -- the more interested people in this subgroup, the less can be 'going' and vice versa. What can we draw from this?**NEED TO ADD HERE**  Clustering<jupyter_code>import matplotlib.pyplot as plt
import matplotlib.image as mpimg
img=mpimg.imread('Small map.png')
imgplot = plt.imshow(img)
plt.show()<jupyter_output><empty_output><jupyter_text>We can use clustering now to understand whether or not holding events on/off campus has effects on event attendance. What we have done here is seperate the data into two groups, those events that are held on campus and those that aren't. We will then run through a similar process as above to determine whether geographical location has some significant effect on group attendance.<jupyter_code>zipdf = df.dropna(subset=['zip'], how='any') #dropping rows with no postcode
CampClass = np.where(zipdf['zip'] == 2006,1 ,0) #Creating an indicator vector
zipdf['Campus'] = CampClass #adding to existing dataframe"
X2 = zipdf[['interested_count','no_reply_count','total_invited_count', 'Campus']]
y2 = zipdf[['attendance_count']]
    
model2 = sm.OLS(y2, X2)
results2 = model2.fit()
print(results2.summary())
zipdf.to_csv('campus_data.csv')
import matplotlib.pyplot as plt
plt.hist(df['attendance_count'], bins='sqrt')  # bin choice via sqrt(n) provides something easy to understand visually
plt.title("How are attendance counts distributed?")
plt.show()<jupyter_output><empty_output><jupyter_text>| Size  | Frequency   |  |-------|-------------||0-86   |     422     |  |       |             ||87-172 |     107     ||       |             ||173-258|     38      ||       |             ||259-334|     27      | |       |             |Here, we have listed the frequencies of events that fit into the specific categories that          are arranged by size.\n",  What have we found?These results suggest that there is a small but statistically significant effect that stems from holding events on campus. Our model is telling us that events held on campus tend to have a *lower* average attendance than those that are not.The results seem to suggest that students do not prefer the convenience of on-campus events. Maybe the fact that the events *are* so **accesible** makes them a little less desireable, maybe people like to dress up, maybe people hate Manning Bar -- whatever it is it seems to be the case that off-campus events attract at least a few more people than the on-campus options. Note though, that a vast majority of events that are held by societies are relativley small. Most of our observations lie within that 0-86 individual attendance range. In other words, smaller events may exhibit different characterisitcs, and these effects may be a little different. In our next model we have controlled for this by only including data from the smallest bracket (0-86).<jupyter_code>small_df = zipdf[df.attendance_count < 87]

X2 = small_df[['interested_count','no_reply_count','total_invited_count','Campus']]
y2 = small_df[['attendance_count']]

model2 = sm.OLS(y2, X2)
results2 = model2.fit()
print(results2.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:       attendance_count   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 7.156e+30
Date:                Wed, 28 Feb 2018   Prob (F-statistic):               0.00
Time:                        14:09:37   Log-Likelihood:                 11866.
No. Observations:                 422   AIC:                        -2.372e+04
Df Residuals:                     418   BIC:                        -2.371e+04
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
=======================================================================================
                          coef    std err  [...]<jupyter_text>What have we found?If you're hosting a *small* event, there is no (statistical) difference between hosting the event on campus or off it. In short, if you're hosting something big, avoid spending time on campus, but if it's something small it could be worth keeping it local.  Natural Language Processing (NLP)<jupyter_code>SoldClass=np.zeros((len(df),1)) #creating indicator arrays
LimitClass=np.zeros((len(df),1))
for event in range(1, len(df)): #Looping through dataframe making sure we account for case sensitivity 
    if 'SOLD OUT' in df.loc[event,'description'] or 'Sold Out' in df.loc[event,'description'] or 'sold out' in df.loc[event,'description'] :
        SoldClass[event] = 1
for event in range(1, len(df)): #Looping through dataframe making sure we account for case sensitivity 
    if 'LIMIT' in df.loc[event,'description'] or 'Limit' in df.loc[event,'description'] or 'limit' in df.loc[event,'description'] :
        LimitClass[event] = 1
        
SoldClass = pd.DataFrame(SoldClass, columns=['Sold_Out']) #Turning arrays into dataframes so we can append
LimitClass = pd.DataFrame(LimitClass, columns=['Limit'])
df_NLP = pd.concat([SoldClass, LimitClass], axis=1) #important to set axis=1 so we concatenate horizontally
df = pd.concat([df,df_NLP], axis=1)<jupyter_output><empty_output><jupyter_text>Accessibility/ExclusivityEarlier, we brought up the accessibility of events as a potential driver/inhibitor of event attendance, and so we want to explore this further. Using Python, we were able to trawl through the descriptions of all the events we had data for. We then 'tagged' those events which included the words 'Sold Out' and 'Limit' in order to shed some light on whether the accesbility of events (not in terms of geography, but in terms of exclusivity') have some affect on attendance.<jupyter_code>X3 = df[['interested_count','no_reply_count','total_invited_count','Sold_Out','Limit']]
y3 = df[['attendance_count']]

model3 = sm.OLS(y3, X3)
results3 = model3.fit()
print(results3.summary())
X4 = df[['interested_count','no_reply_count','total_invited_count','Sold_Out']]
y4 = df[['attendance_count']]

model4 = sm.OLS(y4, X4)
results4 = model4.fit()
print(results4.summary())
X5 = df[['interested_count','no_reply_count','total_invited_count','Limit']]
y5 = df[['attendance_count']]

model5 = sm.OLS(y5, X5)
results5 = model5.fit()
print(results5.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:       attendance_count   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 1.942e+31
Date:                Wed, 28 Feb 2018   Prob (F-statistic):               0.00
Time:                        15:18:52   Log-Likelihood:                 30609.
No. Observations:                1149   AIC:                        -6.121e+04
Df Residuals:                    1145   BIC:                        -6.119e+04
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
=======================================================================================
                          coef    std err  [...]<jupyter_text>What have we found?There are *no* significant results when controlling for exclusivity in the data (we have even included slightly different models in order to confirm this). So, it seems there are no salient benefits of making an event more exclusive as opposed to hosting free events.<jupyter_code>df.to_csv('final_data_extra.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12461479198767335, 'fraction_numerical': 0.025327426810477657, 'mean_word_length': 2.7155635062611805, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '515898', 'license': 'MIT', 'n_tokens_mistral': 2874, 'n_tokens_neox': 2632, 'n_words': 1049, 'path': '.ipynb_checkpoints/USU societies notebook-checkpoint.ipynb', 'repo_name': 'ANOVAProjectUSYD/USYD-Societies-Events', 'size': '10384'}