<jupyter_start><jupyter_text>Loading journal text entries<jupyter_code>users = pd.read_csv('../data/users.csv')
journals = pd.read_csv('../data/journals.csv')
# Dropping NAN entries in sentiment/journal/date
journals_sub = journals.dropna(subset=['sentiment', 'journal', 'date'])
journals_sub['sentiment_level'] = journals_sub[[
    'sentiment'
]].apply(lambda x: pd.cut(x, [-np.inf, -.5, .5, np.inf],
                          labels=['negative', 'neutral', 'positive']))<jupyter_output>/Users/afrah/miniconda3/envs/test/lib/python3.7/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """<jupyter_text>Feature Engineering<jupyter_code>journals_sub['date'] = pd.to_datetime(journals_sub['date'])
journals_sub['year'] = pd.Series(journals_sub['date'].dt.year,
                                 dtype='category')
journals_sub['day'] = journals_sub['date'].dt.day_name()
journals_sub['month'] = journals_sub['date'].dt.month_name()
journals_sub['month_trans'] = np.cos(
    (journals_sub['date'].dt.month - 6.0) / 6.0)
journals_sub['timeofday'] = journals_sub['date'].dt.hour
journals_sub['timeofday_trans'] = np.cos(
    (journals_sub['date'].dt.hour - 12.0) / 12.0)
plt.figure(figsize=(15, 8))
journals_sub.hist(figsize=(20, 10),
                  ylabelsize=15,
                  xlabelsize=15,
                  xrot=20,
                  bins=20,
                  edgecolor='white')<jupyter_output><empty_output><jupyter_text>To observe if there are any big fluctuations in sentiment levels across the years so that we can make reasonable assumptions about the distribution of the data.<jupyter_code>sns.countplot(x='year',
              hue='sentiment_level',
              data=journals_sub,
              orient='horizontal',
              edgecolor='white')<jupyter_output><empty_output><jupyter_text>Time of day effect on moodThis shows that numerous negative/positive entries are usually made at later/earlier hours in the day whereas neutral entries are made during mid-day. However as seen in the chart above the number of observations are also skewed between the different sentiment bins, which might indicate that "most entries" (and not sentiment-based associations) are made at earlier/later hours in the day rather than mid-day.<jupyter_code>plt.figure(figsize=(10, 8))
# plt.subplot(2,2,1)
ax = sns.boxplot(x="sentiment_level", y='timeofday_trans', data=journals_sub)
ax.set_xticklabels(ax.get_xticklabels(), rotation=20)

# plt.subplot(2,2,3)
# plt.scatter(x=np.arange(24), y= np.cos((np.arange(24)-12)/12))
# plt.axhline(y=0.75)<jupyter_output><empty_output><jupyter_text>Seasonal effectDo Seasons influence mood predictions? Especially in the context of seasonal affective disorders  Monthly fluctuations This is an attempt to see if there is any seasonality effect on mood. A classic example is Seasonal Affective Disorders which might indicate more negative sentiment entries during Winter Season. When binning by month and using "continuous" sentiment scores, there is a "dip" around January and "peak" around May, which is an interesting observation. When using "continuous" month transformed variable (cosine-transformation) and binning by sentiment scores (neutral/positive/negative) this trend is less obvious.<jupyter_code>plt.figure(figsize=(15, 15))
plt.subplot(3, 1, 1)
ax = sns.boxplot(x="sentiment_level", y='month_trans', data=journals_sub)
ax.set_xticklabels(ax.get_xticklabels(), rotation=20)

plt.subplot(3, 1, 2)
ax = sns.boxplot(y="sentiment", x='month', data=journals_sub)
ax.set_xticklabels(ax.get_xticklabels(), rotation=20)

plt.subplot(3, 1, 3)
plt.scatter(x=np.arange(12), y=np.cos((np.arange(12) - 6) / 6))<jupyter_output><empty_output><jupyter_text>Seasonal fluctuations<jupyter_code>Winter = ['December', 'January', 'February']
Spring = ['March', 'April', 'May']
Summer = ['June', 'July', 'August']
Fall = ['September', 'October', 'November']

seasons = dict(
    zip(Winter + Spring + Summer + Fall,
        ['Winter'] * 3 + ['Spring'] * 3 + ['Summer'] * 3 + ['Fall'] * 3))
journals_sub['season'] = journals_sub['month'].map(seasons)

seasons_color = dict(
    zip(Winter + Spring + Summer + Fall,
        ['skyblue'] * 3 + ['lightgreen'] * 3 + ['#FFD43B'] * 3 + ['Red'] * 3))<jupyter_output>/Users/afrah/miniconda3/envs/test/lib/python3.7/site-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  if __name__ == '__main__':<jupyter_text>Winter seems to have the lowest mood predictions; though the difference compared to other seasons seems not too big of an effect. This impact may be masked by the clumping of months together and seems more evident in the monthly binning of sentiment scores above.<jupyter_code>plt.figure(figsize=(15, 8))
ax = sns.boxplot(y="sentiment",
                 x='season',
                 data=journals_sub,
                 palette={
                     'Winter': 'skyblue',
                     'Fall': 'Red',
                     'Spring': 'lightgreen',
                     'Summer': '#FFD43B'
                 })
ax.set_xticklabels(ax.get_xticklabels())<jupyter_output><empty_output><jupyter_text>Content-based features Personal and Non-personal pronounsThis is to test the hypothesis that usage of pronouns might differ depending on mood where people may talk about themselves more often or less compared to other people depending on their mood/personality.<jupyter_code>def get_pprn(doc):
    inner_pprn = ('i', 'me', 'my', 'mine', 'myself')
    outer_pprn = ('we', 'you', 'your', 'yourself', 'yours', 'yourselves', 'he',
                  'him', 'his', 'himself', 'she', 'her', 'hers', 'herself',
                  'they', 'them', 'their', 'theirs', 'themselves')
    inner_count = 0
    outer_count = 0

    for token in doc:
        if token.text in inner_pprn:
            inner_count += 1
        elif token.text in outer_pprn:
            outer_count += 1
    return inner_count, outer_count<jupyter_output><empty_output><jupyter_text>These is a lot of variation in the length of the journal entry where longer journal entries are observed less often than shorter journal entries<jupyter_code>journals_sub['word_len'] = journals_sub['journal'].apply(lambda x: len(x.split(' ')))
journals_sub['word_len'].plot(kind='hist', logy=True, edgecolor='white')

# print(sum(journals_sub['word_len']>50))
plt.xlabel('# Words in Entry')
nlp = spacy.load("en_core_web_sm")
i_count_all = []
o_count_all = []

for entry in journals_sub['journal']:
    doc = nlp(entry.lower())
    i_count, o_count = get_pprn(doc)
    i_count_all.append(i_count)
    o_count_all.append(o_count)
journals_sub['inner_pprn'] = np.array(i_count_all)
journals_sub['outer_pprn'] = np.array(o_count_all)
journals_sub[
    'sum_pprn'] = journals_sub['outer_pprn'] + journals_sub['inner_pprn']
journals_sub[
    'inner_frac'] = journals_sub['inner_pprn'] / journals_sub['sum_pprn']
journals_sub['inner_frac'] = journals_sub['inner_frac'].fillna(0)
journals_sub[
    'outer_frac'] = journals_sub['outer_pprn'] / journals_sub['sum_pprn']
journals_sub['outer_frac'] = journals_sub['outer_frac'].fillna(0)
journals_sub['pprn_ratio'] = np.array(i_count_all) / (np.array(o_count_all) +
                                                      1)<jupyter_output><empty_output><jupyter_text>This figure shows that there is an association between sentiment, total pronouns used and the ratio of pronouns used to describe self vs. pronouns used to describe others (PPRN/Personal Pronoun Ratio Numeric). You can see this teased apart using the measure (Number of I,Me used/Total personal pronouns) where positive sentiments are seen more when this fraction is higher i.e. when people use alot of personal pronouns and when people talk about themselves more the sentiment is more positive.<jupyter_code>sns.boxplot(x='sentiment_level',
            y='inner_frac',
            data=journals_sub[journals_sub['sum_pprn'] > 10])
plt.ylabel('(Number of I,Me used/Total personal pronouns)')
plt.xlabel('')<jupyter_output><empty_output><jupyter_text>This effect is also there when looking at ratio of pronouns used to describe self vs. pronouns used to describe others (PPRN/Personal Pronoun Ratio Numeric). When we threshold the journal entries that use at least 10 personal pronouns and bin sentiment scores, the median PPRN is higher for positive journal entries<jupyter_code>sns.boxplot(x='sentiment_level',
            y='pprn_ratio',
            data=journals_sub[journals_sub['sum_pprn'] > 10])
plt.ylabel('Personal pronouns ratio (I,me vs. She/He/Them)')
plt.xlabel('')
stop_words = set(stopwords.words('english'))

filtered_sentences = journals_sub['journal'].apply(
    lambda x: ' '.join([i for i in WhitespaceTokenizer().tokenize(x)]))
corpus = journals_sub['journal']
vectorizer = TfidfVectorizer(stop_words=stopwords.words('english'),
                             token_pattern=r'(?u)\b[A-Za-z]+\b')
mat = vectorizer.fit_transform(corpus)<jupyter_output><empty_output><jupyter_text>Readability scores This effect of sentiment can also be observed using polysyllable count where people with positive sentiment scores tend to use more difficult/long polysyllable count words compared to those with negative sentiments.<jupyter_code>journals_sub['polysyllable_count'] = journals_sub['journal'].apply(
    lambda x: textstat.textstat.polysyllabcount(x))
journals_sub['difficult_words'] = journals_sub['journal'].apply(
    lambda x: textstat.textstat.difficult_words(x))
journals_sub['cl_index'] = journals_sub['journal'].apply(
    lambda x: textstat.textstat.coleman_liau_index(x))
sns.boxplot(x='sentiment_level',
            y=np.sqrt(journals_sub[journals_sub.sum_pprn>20].polysyllable_count),         
            data=journals_sub)
plt.ylabel('Polysyllable count')
plt.xlabel('')<jupyter_output><empty_output><jupyter_text>Users-frequency of posting There is also user behavior element in predictions where some users have logged more entries than others. For this analysis we dont have access to how long the user has been using the app which may be a contributing factor for this. Here we try to see what the median sentiment score looks like for users that log more journal entries than others.<jupyter_code>journals_sub['user'] = journals_sub['path'].apply(
    lambda x: re.sub('"', '',
                     x.split(',')[1].strip()))
# Number of journal entries varies per user
plt.figure(figsize=(10, 8))
plt.hist(journals_sub['user'].value_counts(), bins=20, edgecolor='white')
plt.yscale('log')
plt.xlabel('# Journal Entries')
plt.ylabel('# Users')<jupyter_output><empty_output><jupyter_text>As seen, the higher the number of entries by a user, the higher the median sentiment score of these journal entries. This can be due to (i) 'happier' users tend to log more entries/use the app more, (ii) users tend to be happier the longer they use the app, or (iii) sentiment scores are unreliable and these associations dont show any meaningful signals.<jupyter_code># Variance in sentiment scores may also be associated with # entries by a user
plt.hexbin(y=journals_sub.groupby('user').median()['sentiment'],
           x=journals_sub['user'].value_counts(),
           gridsize=20,
           cmap='Blues',
           bins='log')
plt.xlabel('# Entries by user')
plt.ylabel('Median(Sentiment)')
plt.colorbar()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12218187959131976, 'fraction_numerical': 0.010132567761546906, 'mean_word_length': 4.521678321678322, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '49261', 'license': 'MIT', 'n_tokens_mistral': 3891, 'n_tokens_neox': 3629, 'n_words': 1001, 'path': 'notebooks/Maslo journal entries data.ipynb', 'repo_name': 'HeyMaslo/AllTheFeels', 'size': '11843'}