<jupyter_start><jupyter_text>A4 - Unrequited Email - cs2951-<NAME>  My Personal Email Behavior Anecdotal: From an anecdotal perspective I certainly fall into the prioritizer category. [2]  This stems from early professional experience where I handled large volumes of email.  If you look at the work of Whittaker and Sidner [3] I fall into their category of frequent filers.  Two habits I have picked up in recent years are:Sending emails to myself from other accounts as a method to backup important data and/or projects.Utilizing Gmail’s star and important tag functionality as reminder that I need to go back and revisit an email.  Gmail account categorizes email to four distinct categories. (Primary, Social,Promotions, Forums)  Each email can be starred or tagged as priority.  A user selected priority tag helps teach Gmail that that type of email is important.  Gmail will not auto-assign a start but it can assign a priority tag.  I do not use any custom colors for stars or priority tags.  I wanted to use these as feature models but could not find how to extract them from using the IMAP protocol. Email Information/DataFiles_data_gathering/fetch-raw.py_data_gathering/fetch-raw-sent.pycalculateResponseTime.py bold = feature for linear regression (is numerical) - Index - generic index value for easy counting- MessageID- InReplyTo-  Date - full date and time converted to US/Eastern timezone-  Day - day of week-  Hour - hour of day-  Minute = minutes within the hour-  MinsSinceMidnight - total minutes since midnight- Flags - general email flags, to see if they are useful- Subject-  isReply-  isFwd- From- inFrom- To-  inTo-  inCc-  CcCount - how big was the CC field-  inBcc-  MultiPart-  BodyLen - length of message if plain/text-  BodyLenHTML - length of message with HTML.  Shows if message is long.-  impNames - Important Names (highlighted below)-  impTitles - Important Titles (highlighted below)-  impWords - Important Words (highlighted below)-  WordsTotal - total of all Important flagged above (highlighted below)I have so many initial features to test which ones are prevalent for predicting an email response.  Creativity was encouraged for the project.   Important WordsMy personal email is mainly used for messages with my family, music colleagues, prospective employers, or as a personal file back-up platform.  Hence I will read through each email to see if it contains a few keywords and/or names of family members.   Names: (20)Wallace,Watson,  Alison, Kyle, Mandy, Amanda, Mandy, Lexi, Alexa, Sandi, Lawrence, Larry, Roger, Margaret, Amy, Jason, Josh, Jenny, Birdwell, Nanny  Titles: (9)wife, brother, sister, mother, mom, dad, aunt, uncle, grandmother Words:  (13)Love, Wedding, Birthday, Child, Family, Professor, Adjunct, Manager,  Developer, Salary, Position, Mazda, Brown How to Assess Reply TimeFile: calculateResponseTime.pyOne note to make is all Date variables have been re-computed to in US Eastern time.  This will ensure accuracy pertaining to when I received the email and when I potentially sent a reply.Is From entry from received emails in the To field in sent emails?Is Date from received email before Date from sent emails?Other methods to consider:If message-ID from my received email is in InReplyTo then proceed.  Although you have to be careful with this since InReplyTo is not 100% reliable according to Zawinski. [4]  Shile Zawinksi’s look correct they from years ago.  I ran a test and it was incredibly consistent utilizing these variables.  I hand checked the results which was easy since I only replied to 419 emails over the years.  I believe for this assignment this method is trustworthy.<jupyter_code># imports
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

import warnings
warnings.filterwarnings('ignore')

from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler([0,1])

# this allows plots to appear directly in the notebook
%matplotlib inline

# read data into a DataFrame
data = pd.read_csv('_data_final2.csv', index_col=False)
data['DayNum'] = data.Day.map({'Sun':0, 'Mon':1, 'Tue':2, 'Wed':3, 'Thu':4, 'Fri':5, 'Sat':6}) #convert Day of week
#data.head()

print 'Original Shape: ', data.shape
data = data.loc[data['From'].str.contains('<EMAIL>')==False]
data = data.loc[data['From'].str.contains('<EMAIL>')==False]
data = data.loc[data['From'].str.contains('<EMAIL>')==False]
print 'After removal of emails sent by me: ', data.shape
print ''
dataRes = data.loc[data['resTime'] > 0]
print 'Shape of emails that I responded to: ', dataRes.shape
print 'Response time if I replied: ', dataRes.resTime.mean(), ' mins'
print ''
# print the header and shape of the DataFrame
print 'INITIAL FEATURES / HEADERS'<jupyter_output>Original Shape:  (17100, 69)
After removal of emails sent by me:  (16938, 69)

Shape of emails that I responded to:  (397, 69)
Response time if I replied:  895.317380353  mins

INITIAL FEATURES / HEADERS<jupyter_text>Features ModelWithin my features model any flag type of variable, such as is this message a reply, is record with a 1 if true and a 0 if not true.  This will help to count, categorize, and graph the frequency of these results since they are numerical.  From Jeff in class: Tons of equations that act like:> Compute weights through linear aggression:> weight(Last time you replied) + weight(response time) + weight() > = Prediction in mins, when you would reply Would I reply to Email?Out my email history ranging from April 1st,  2014 to the present I have only replied to roughly 2.34% of them.  That is 397 responses sent from 16938.  Another interesting thing to note is only 416 / 919 of my sent emails were were replies.  This equates to 45.2%.  Now on to what would trigger a response:Have I received emails before from this address?If yes and no reply, do not send replyElse if: start analyzing more  General Overview - My Activity + PatternsLast Time I Replied.  This takes into account what my recent activity over the past week has been.  What day of the week is it, and how likely am I to reply? (source sent email)- Sun - 62  - 6.7%- Mon - 159 - 17.3%- Tue - 168 - 18.3%- Wed - 151 - 16.4%- Thu - 170 - 18.5%- Fri - 156 - 17.0%- Sat - 54  - 5.9%As you can see from above my email usage dips considerably over the weekend.  These relationships will be modelled in my feature set.  Average Time is takes to reply.  The average time it takes me to reply is 895 minutes or 14.9 hours.  In general this seems about right.  To my knowledge I don’t recieve high priority emails for my personal email.  Much of it is family related or casual.  Things that sometimes warrant a careful response.   I can get a better idea of how this breaks down after I model my how previous responses using Mulitple Linear Regression.   Analysis of FeaturesLets start with some basic features and see which ones have a relationship to response time.  The goal here to trim down our feature set so we can utilize only the best features.  This will help with overfitting once we get to scikit learn and other platforms utilized to predict response time.  It will also improve the email responder's performance.<jupyter_code># visualize the relationship between the features and the response using scatterplots
fig, axs = plt.subplots(1, 2, sharey=False)

data = data.loc[data['resTime'] > 0]
dataMSM = data.loc[data['resTime'] < 10000] #trimming just for this for a better visual

dataMSM.plot(kind='scatter', x='MinsSinceMidnight', y='resTime', ax=axs[0], figsize=(16, 8))
#dataMSM.plot(kind='scatter', x='impNames', y='resTime', ax=axs[1]) #my wife's name
data.plot(kind='scatter', x='isReply', y='resTime', ax=axs[1]) 

data = data.loc[data['resTime'] < 30000] #remove outliers<jupyter_output><empty_output><jupyter_text>OutliersWhen you compare response time vs isReply you can see two obvious outliers.  I looked up these emails and they indeed receive responses.  However I was not exactly responding to the person in email.  I mainly used that old email as a way to track down and remember their email address.  I feel it is safe to remove these emails from future analysis in hopes I can improve my prediction and analysis.  This puts my range for responses roughly between 1 minute and 1 week.  Which given my history makes sense.<jupyter_code># this is the standard import if you're using "formula notation" (similar to R)
import statsmodels.formula.api as smf

# create a fitted model in one line
lm = smf.ols(formula='resTime ~ impNames + impTitles + impWords', data=data).fit()

# print the coefficients
print 'Basic Coefficients for impNames + impTitles + impWords (only impNames is good)'
print lm.params
print ''

# print the confidence intervals for the model coefficients
print 'P Values for the Rest of the Features'
print ''

lis=['DayNum','Hour','Minute','MinsSinceMidnight','isReply','isFwd','inCc','CcCount','inBcc','MultiPart','BodyLen','BodyLenHTML','impNames','impTitles','impWords','WordsTotal','Wallace','Watson','Alison','Kyle','Mandy','Amanda','Lexi','Alexa','Sandi','Lawrence','Larry','Roger','Margaret','Amy','Jason','Josh','Jenny','Birdwell','Nanny','wife','brother','sister','mother','mom','dad','aunt','uncle','grandmother','Love','Wedding','Birthday','Child','Family','Professor','Adjunct','Manager','Developer','Salary','Position','Brown']
for feature in lis:
    # create a fitted model in one line & print coeff
    lm = smf.ols(formula='resTime ~ '+feature, data=data).fit()
    pVal = lm.pvalues
    if pVal[1] < 0.05:
        print feature, 'is Significant 95%'
    elif pVal[1] < 0.1: 
        print feature, 'is Significant 90%'
    else:
        print feature, 'is not'
    print lm.pvalues
    print ''<jupyter_output>Basic Coefficients for impNames + impTitles + impWords (only impNames is good)
Intercept    649.430315
impNames     -56.908127
impTitles    293.915695
impWords      26.016958
dtype: float64

P Values for the Rest of the Features

DayNum is Significant 95%
Intercept    4.326635e-07
DayNum       3.863490e-02
dtype: float64

Hour is Significant 90%
Intercept    0.386556
Hour         0.066918
dtype: float64

Minute is Significant 90%
Intercept    0.052662
Minute       0.085333
dtype: float64

MinsSinceMidnight is Significant 90%
Intercept            0.471152
MinsSinceMidnight    0.055968
dtype: float64

isReply is Significant 95%
Intercept    0.000979
isReply      0.029272
dtype: float64

isFwd is not
Intercept    1.600950e-10
isFwd        5.060793e-01
dtype: float64

inCc is not
Intercept    1.052925e-10
inCc         5.636923e-01
dtype: float64

CcCount is not
Intercept    9.956891e-11
CcCount      3.639737e-01
dtype: float64

inBcc is not
Intercept    1.120154e-10
inBcc        6.505067e-[...]<jupyter_text>P Value Analysis - Pre Removal of OutliersHere are the values that had significant p values.  We will be working with these features from now on: Hour, Minute, MinsSinceMidnight, impTitles (important titles list), WordsTotalA surprising ommision is day of the week (dayNum) based upon my previous analysis.  Its p value was 0.104. However this analysis focuses on response time and when the email was recieved.  It does not focus solely on when an email is recieved.  Another surprising ommision is isReply.  This looks at whether the message sent to me was a reply.  Its p value was 0.10967.  It was very close.  If you think about a string of responses this makes sense.  Hypothetically you would respond to a reply because it is an on-going conversation.  However it is difficult to know when exactly does a conversation stop?  From important words: Alexa, Sandi, brother, sister, mother, momAlexa is my niece and Sandi is my mother. This is not surprising since these emails often relate to them visiting from Ireland and are often time sensitive in nature.  My mother and father also do not live where I do.  They are of an older generation and often find it easiest to communicate via email when they cannot call me.  However messages from my mother are usually more urgent, hence I am not surprised at the significance of mom and mother. After removal of Outliers:  ADD: DayNum, isReply, BodyLenHTML, Jason; REMOVE: WordsTotal, AlexaAfter the removal of the outliers the p values shifted.  It now shows day of week as having a significant value to 95% confidence.  This is great since this was one feature that looked important from my initial surveying of the data.  BodyLenHTML was an interesting addition.  It shows the length in characters of an emails HTML body.  Hence it indicates roughly how visually large an email would be.  This is just a rough estimate.  I figured advertising emails would include tons of HTML with pictures and fancy layouts.  Jason is my cousins name.  When we have family events he and I often coordinate through email.  So this is not a surpise on reflection.  It dropped my niece's name which was interesting. Forests Trees Analysis to assess Features (code below)<jupyter_code>#FROM: http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#example-ensemble-plot-forest-importances-py
feature_cols = ['DayNum','Hour','Minute','MinsSinceMidnight','isReply','impTitles','WordsTotal','Alexa','Sandi','brother', 'sister', 'mother', 'mom']
#feature_cols = ['MinsSinceMidnight']
feature_cols = lis

X = data[feature_cols]
y = data.resTime

forest = ExtraTreesClassifier()
forest = forest.fit(X, y)
importances = forest.feature_importances_  

model = SelectFromModel(forest, prefit=True)
X_new = model.transform(X)
X_new.shape  


##
std = np.std([tree.feature_importances_ for tree in forest.estimators_],axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

countF = 0
for f in range(X.shape[1]):
    #print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))
    print countF, feature_cols[f], ' (', importances[indices[f]], ')'
    countF += 1

# Plot the feature importances of the forest
plt.figure(figsize=(16, 8))
plt.title("Forest of Trees Feature Importances")
plt.bar(range(X.shape[1]), importances[indices], color="g", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices)
plt.xlim([-1, X.shape[1]])
plt.show()<jupyter_output>Feature ranking:
0 DayNum  ( 0.122346829665 )
1 Hour  ( 0.11515786169 )
2 Minute  ( 0.11078623268 )
3 MinsSinceMidnight  ( 0.0801239154864 )
4 isReply  ( 0.0757749843489 )
5 isFwd  ( 0.0498504314576 )
6 inCc  ( 0.0372683322847 )
7 CcCount  ( 0.0367055297648 )
8 inBcc  ( 0.0334180139249 )
9 MultiPart  ( 0.0260246548582 )
10 BodyLen  ( 0.0239782053757 )
11 BodyLenHTML  ( 0.022299384326 )
12 impNames  ( 0.0218195697235 )
13 impTitles  ( 0.0203681284836 )
14 impWords  ( 0.019129432152 )
15 WordsTotal  ( 0.0178726133379 )
16 Wallace  ( 0.0150591854664 )
17 Watson  ( 0.0127584009504 )
18 Alison  ( 0.0122974912583 )
19 Kyle  ( 0.0115408136118 )
20 Mandy  ( 0.0100455954276 )
21 Amanda  ( 0.00937173382753 )
22 Lexi  ( 0.00808982773491 )
23 Alexa  ( 0.00805276828088 )
24 Sandi  ( 0.00753999823333 )
25 Lawrence  ( 0.00726208511927 )
26 Larry  ( 0.00676979922728 )
27 Roger  ( 0.00592787595232 )
28 Margaret  ( 0.00579910945178 )
29 Amy  ( 0.00549929190807 )
30 Jason  ( 0.00454916712598 )
31 Josh  ([...]<jupyter_text>Forests of Trees AnalysisI ran this secondary analysis because as I was going through my iteration process I was getting vastly different results.  Instead of applying different features through pure brute force I wanted another method to assess the importance as features.The top five are clearly: - DayNum  (0.128029570047)- Hour  (0.114617828311)- Minute  (0.111912963833)- MinsSinceMidnight  (0.0989695882787)- isReply  (0.0806493766716)None of these are a surprise in any way.  MinsSinceMidnight is a derived value from Hour and Minute.  So it is not a surprise it scored high as well.There were other features that scored high in p value but not so good in the forests of trees such as BodyLenHTML, and especially brother, sister, mother.  Depending on their coefficients I may drop these from the feature list. Predicting Respone Time and more Feature Evaluation<jupyter_code># create X and y
feature_cols = ['DayNum','Hour','Minute','MinsSinceMidnight','isReply','isFwd','impTitles','BodyLenHTML','Jason','Sandi','brother', 'sister', 'mother', 'mom']

X = data[feature_cols]
y = data.resTime

# follow the usual sklearn pattern: import, instantiate, fit
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print 'Intercept: ', lm.intercept_
print ''
count = 0
for coeff in lm.coef_:    
    print feature_cols[count]
    print lm.coef_[count]
    print ''
    count += 1
    
#test prediction
# predict for a new observation
#'DayNum','Hour','Minute','MinsSinceMidnight','isReply','impTitles','WordsTotal','Alexa','Sandi','brother', 'sister', 'mother', 'mom'
pred = lm.predict([3, 13, 25, 805, 1, 0, 1, 2, 1, 0, 0, 0, 0, 1])

print pred[0], 'mins = ', pred[0]/60, 'hours' 

# calculate the R-squared
print 'R Squared: ', lm.score(X, y)<jupyter_output>Intercept:  199.997476956

DayNum
-114.181634835

Hour
-0.0810898406408

Minute
5.38168185324

MinsSinceMidnight
0.516291414562

isReply
281.333930106

isFwd
-108.737512452

impTitles
-159.150173694

BodyLenHTML
0.0616081093289

Jason
-249.105100596

Sandi
-451.46173114

brother
2242.79004674

sister
1041.89178631

mother
-20.6071638292

mom
881.62277243

1161.37968404 mins =  19.3563280674 hours
R Squared:  0.144179479839<jupyter_text>Coefficient AnalysisIt appears the 4 features that cause me to reply quickly are Day of the Week, isFwd, Important Titles flagged, "Sandi", and "Jason".  I would include "mother" in their as well.  None of these are a surpise.Hour, Day, MinsSinceMidnight did not have very significant coefficients despite being highly correlated to response times. I still feel these should be included in the final feature list because of the p values and placing in the forests of trees.  On the other hand BodyLenHTML scored low on the forests of tree and its coefficients provides little value.isReply, brother, sister, and mother's coefficients illustrate that if those are flagged they will increase response time.  I personally thought this would have been the opposite.  From the coefficients if any of these are true they can have a drastic effect on reply time. Scaling Data ResultsIn the everlong quest to improve my prediction analysis I decided to go analyze Scaling functions. (I actually did this very early on, I just decided to include here in the notebook) In the code below I iterated over various scaling mechanisms to try and improve my numbers.  I tried to use preprocessing.MaxAbsScaler() and the normal preprocessing.scale().  Here are the results: Before removal of OutliersBefore running either of these my response time for the test was 19 hours. Features = ['DayNum','MinsSinceMidnight', 'isReply']preprocessing.MaxAbsScaler() increased the response time to 19602 hours.preprocessing.scale(X) increased the response time to 4502 hours. After removal of OutliersBefore running either of these my response time for the test was 15.9 hours. Features = ['DayNum','MinsSinceMidnight', 'isReply']preprocessing.MaxAbsScaler() increased the response time to 13234 hours.preprocessing.scale(X) increased the response time to 3036 hours. SummaryI wanted to test again the differences when I removed the outliers.  These numbers after the removal look much better.  Thus is another reason to validate the decision to remove the outliers.  The differences between the scaled and non-scaled results seem illogical and not representative of a real world scenario.  Thus my original method of utilizing the raw data was used.   Removal of Features that are FlagsSo how can we improve our R squared value of 0.121?  What if we removed the features that are flags?  We could potentially only include these if they were flagged by the mailbot.  So if none of them were flagg how would this change our coefficients and R Squared value?<jupyter_code># create X and y
feature_cols = ['DayNum','Hour','impTitles','BodyLenHTML', 'isReply']


X = data[feature_cols]
y = data.resTime

# follow the usual sklearn pattern: import, instantiate, fit
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print 'Intercept: ', lm.intercept_
print ''
count = 0
for coeff in lm.coef_:    
    print feature_cols[count]
    print lm.coef_[count]
    print ''
    count += 1
    
#test prediction
# predict for a new observation
#'DayNum','Hour','Minute','MinsSinceMidnight','isReply','impTitles','WordsTotal','Alexa','Sandi','brother', 'sister', 'mother', 'mom'
pred = lm.predict([3, 8, 3, 1805, 1])

print pred[0], 'mins = ', pred[0]/60, 'hours' 

# calculate the R-squared
print 'R Squared: ', lm.score(X, y)<jupyter_output>Intercept:  -4.22283472987

DayNum
-4.41706425494

Hour
1.61256377728

impTitles
62.8495199602

BodyLenHTML
2.08458794126e-05

isReply
569.559711167

753.572380583 mins =  12.5595396764 hours
R Squared:  0.0374805273312<jupyter_text>So the R squared value went down to 0.0133.  If I added isFwd the R Squared goes up to 0.0134.  If I added isReply the R Squared goes up to 0.0374. When isReply is added here are the coefficients:- DayNum  -4.41706425494- Hour  1.61256377728- impTitles  62.8495199602- BodyLenHTML  2.08458794126e-05- isReply  569.559711167The inclusion of isReply ruins the other coefficients.  It dwarfs their coeffs to a level that are not as usable as when the other features were included.  Hence why it is important to include the other relevant features. Explanation for low R Squared ValuesAt first I was extremely worried that my R Squared values were so low despite selecting features with excellent p values, high placement on the forests of trees, and good coefficients.  I found an excellent resource that helps potentually explain why this might be okay.  Frost answers the question "Are Low R-squared Values Inherently Bad?" [5] [6]  > In some fields, it is entirely expected that your R-squared values will be low. For example, any field that attempts to predict human behavior, such as psychology, typically has R-squared values lower than 50%. Humans are simply harder to predict than, say, physical processes. > Furthermore, if your R-squared value is low but you have statistically significant predictors, you can still draw important conclusions about how changes in the predictor values are associated with changes in the response value. Regardless of the R-squared, the significant coefficients still represent the mean change in the response for one unit of change in the predictor while holding other predictors in the model constant. Obviously, this type of information can be extremely valuable.                 -Frost [5]What we are doing in essence is predicting human behavior.  Email response time is a human behavior.  It is still important to have a good R Squared value.  Hence why I compared R Squared values to see which feature would be most effective.   Feature ListWhile the R Squared values in general are poor I am going to with the set of features that fit into these general classifications:- Excellent P values (at least 90% significant)- High placement on the Forest of Trees- Good Coefficients from Linear Regression (provide value for prediction) DayNum, MinsSinceMidnight, isReply, isFwd, impTitles, Jason, Sandi, brother, sister, momI feel good about these features.  I went from initial 71 features down to 10.  5 out of these serve mainly as flags.  I felt including this flags is important.  The exclusion ruined the coefficients.  The data about these flags (Sandi, brother, sister, mom) fit with my percieved real life family situation.  Minute is an odd feature looking back on it.  It can only be between 0-60.  I am surprised it had such a good p value.  However this might tie into lunch breaks or watching TV.  The graph at the beggining of the notebook that featured MinsSinceMidnight against response time shows I respond to emails all throughout the day.  So I decided to just include MinsSinceMidnight since it is a derived variable and has the most significant coefficient.  Even if I don't actually need to respond to a reply, chances are I might.  It would be courteous to have my email responder contact someone.  I also decided to include day of the week (dayNum) because it was so close to being within 90% confidence.  It was 89.6%.  Like inReply I feel this just has too much real world utilily to ignore considering how close the p value was to being significant.These final features are what will be utilized by my email assistant and email responder applications.  It is okay that they are negative coefficients.  It helps to demonstrate the effect each can have:> In regression with multiple independent variables, the coefficient tells you how much the dependent variable is expected to increase when that independent variable increases by one, holding all the other independent variables constant. [7]If the model produces a negative prediction for response time then I will just count response time as 0 and feed it through the algorithm to determine a response and save a draft. Here are the Final Coefficients<jupyter_code># create X and y
feature_cols = ['DayNum','MinsSinceMidnight','isReply','isFwd','impTitles','Jason','Sandi','brother', 'sister', 'mom']

X = data[feature_cols]
y = data.resTime


# follow the usual sklearn pattern: import, instantiate, fit
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print 'Intercept: ', lm.intercept_
print ''
count = 0
for coeff in lm.coef_:    
    print feature_cols[count]
    print lm.coef_[count]
    print ''
    count += 1
    
#test prediction
# predict for a new observation
pred = lm.predict([2, 490, 0, 0, 1, 0, 1, 0, 0, 1])

print pred[0], 'mins = ', pred[0]/60, 'hours' 

# calculate the R-squared
print 'R Squared: ', lm.score(X, y)<jupyter_output>Intercept:  385.076172606

DayNum
-125.739028248

MinsSinceMidnight
0.540847777386

isReply
281.061248908

isFwd
-113.336662375

impTitles
-171.392653755

Jason
-269.282418585

Sandi
-449.941677809

brother
2269.60395303

sister
1053.62734551

mom
888.892840915

666.172036379 mins =  11.102867273 hours
R Squared:  0.131069632914
{'fraction_non_alphanumeric': 0.07586858006042296, 'fraction_numerical': 0.0618202416918429, 'mean_word_length': 4.5111342351716965, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '142459', 'license': 'Apache-2.0', 'n_tokens_mistral': 8892, 'n_tokens_neox': 7621, 'n_words': 3482, 'path': '_Main_Analysis.ipynb', 'repo_name': 'swallace21/A4', 'size': '26540'}