<jupyter_start><jupyter_text>Identfying Depression with Machine Learning  Introduction (for more details please check my paper). We are going to begin with parsing the dataset. The dataset is RSDD ( Reddit Self-Diagnose Depression dataset)<jupyter_code>#First lets parse the training dataset
from Parser import *

#here we will get the dict of two datasets - one positive and negative. Negative is non-depressed and positive is depressed.
training_parser = Parser([DatasetTraining.POSITIVE,DatasetTraining.NEGATIVE])
training_data = training_parser.parse()<jupyter_output><empty_output><jupyter_text>Lets see the input of one post from the user  As we can see, we have a id ( which represents user's id), date, title and content of the reddit post. In this reimplementation we are going to only use content as an feature.<jupyter_code>user_post = training_data[DatasetTraining.POSITIVE][324]
print(user_post)<jupyter_output>[1mId[0m
train_subject4630

[1mDate[0m
2012-05-14 03:18:18

[1mTitle[0m


[1mContent[0m
K, American here. The anthropocentrism of science has formulated an inability to process without quantification. This is a fundamental flaw, and leads to the lovely fagfarts statement of knowledge being subjective, nothing is certain as true.<jupyter_text>Preprocessing step  Authors of the paper have only done Bag-of-words approach, so we are going to do the same.  Bag-of-words<jupyter_code>import numpy
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords
from Parser import *<jupyter_output><empty_output><jupyter_text>Initializing all the preprocessing tools<jupyter_code>vectorizer = TfidfVectorizer(stop_words = "english")<jupyter_output><empty_output><jupyter_text>Parsing the datasets and seeing the figures - like number of posts<jupyter_code>#lets again parse the training dataset and test dataset
training_parser = Parser([DatasetTraining.POSITIVE,DatasetTraining.NEGATIVE])
training_data = training_parser.parse()

test_parser = Parser([DatasetTest.POSITIVE,DatasetTest.NEGATIVE])
test_data = test_parser.parse()
#Training dataset 
#Positive users (number of depressed users)
print(len(training_data[DatasetTraining.POSITIVE]))

#Training dataset 
#Negative users (number of non-depressed users)
print(len(training_data[DatasetTraining.NEGATIVE]))

#Test dataset 
#Positive users (number of depressed users)
print(len(test_data[DatasetTest.POSITIVE]))

#Test dataset 
#Negative users (number of non-depressed users)
print(len(test_data[DatasetTest.NEGATIVE]))<jupyter_output>30934
264575
18783
218053<jupyter_text>Collecting the data into one textual list for preprocessing steps<jupyter_code>#initializing the lists for inputs and truth vectors
posts = []
truth = []
for post in training_data[DatasetTraining.POSITIVE][:2000]:
    content = post.title.strip()+post.content.strip()
    posts.append(content)
    truth.append(1)
    
for post in training_data[DatasetTraining.NEGATIVE][:2000]:
    content = post.title.strip()+post.content.strip()
    posts.append(content)
    truth.append(0)
    
for post in test_data[DatasetTest.POSITIVE][:500]:
    content = post.title.strip()+post.content.strip()
    posts.append(content)
    truth.append(1)

for post in test_data[DatasetTest.NEGATIVE][:500]:
    content = post.title.strip()+post.content.strip()
    posts.append(content)
    truth.append(0)<jupyter_output><empty_output><jupyter_text>Let's check whether the data matches<jupyter_code>#Number of posts 
print(len(posts))

#Number of truths
print(len(truth))<jupyter_output>5000
5000<jupyter_text>Now we have to transform the data... Let's use CountVectorizer for that.<jupyter_code>from sklearn.model_selection import train_test_split

matrix = vectorizer.fit_transform(posts).todense()
X, X_test, y, y_test = train_test_split(matrix, truth, test_size=0.25, random_state=42)
X_train,X_validation, y_train,y_validation = train_test_split(X,y,test_size = 0.33,random_state = 42)
print(len(X_train))
print(len(X_validation))
print(len(X_test))
print(len(y_train))
print(len(y_validation))
print(len(y_test))<jupyter_output>2512
1238
1250
2512
1238
1250<jupyter_text>Let's have a look what are the most common words for each class  Depressed users<jupyter_code>from wordcloud import WordCloud
import matplotlib.pyplot as plt
import re

depression_text = []
for post in training_data[DatasetTraining.POSITIVE]:
    content = post.title.strip()+post.content.strip()
    content = re.sub(r"http\S+", "", content)
    depression_text.append(content)

depression_words = ' '.join([text for text in depression_text])
wordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(depression_words)

plt.figure(figsize=(10, 7)) 
plt.imshow(wordcloud, interpolation="bilinear") 
plt.axis('off') 
plt.show()<jupyter_output><empty_output><jupyter_text>Control users<jupyter_code>import re
no_depression_text = []
for post in test_data[DatasetTest.NEGATIVE]:
    content = post.title.strip()+post.content.strip()
    content = re.sub(r"http\S+", "", content)
    no_depression_text.append(content)

no_depression_words = ' '.join([text for text in no_depression_text])
wordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(no_depression_words)

plt.figure(figsize=(10, 7)) 
plt.imshow(wordcloud, interpolation="bilinear") 
plt.axis('off') 
plt.show()<jupyter_output><empty_output><jupyter_text>Implementation of models from the Johns Hopkins paper  Decision trees  Cross-validation<jupyter_code>from sklearn.tree import DecisionTreeClassifier

#first we have to cross-validate to find the optimal tree depth
vocab_size = len(vectorizer.vocabulary_)
for i in range(1,vocab_size,500):
    clf = DecisionTreeClassifier(max_depth = i)
    clf.fit(X_train,y_train)
    print(clf.score(X_validation,y_validation),i)<jupyter_output>0.4991922455573506 1
0.592891760904685 501
0.6001615508885298 1001
0.5888529886914378 1501
0.5823909531502424 2001
0.5856219709208401 2501
0.5961227786752827 3001
0.5904684975767367 3501
0.5904684975767367 4001
0.5969305331179321 4501
0.5977382875605816 5001
0.5888529886914378 5501
0.6017770597738288 6001
0.5864297253634895 6501
0.5936995153473344 7001
0.5904684975767367 7501
0.5872374798061389 8001
0.5953150242326333 8501
0.5896607431340872 9001
0.5872374798061389 9501
0.5888529886914378 10001
0.6058158319870759 10501
0.6050080775444265 11001
0.6050080775444265 11501
0.5912762520193862 12001
0.5888529886914378 12501
0.5912762520193862 13001
0.5977382875605816 13501
0.598546042003231 14001<jupyter_text>Training and evaluation<jupyter_code>#the optimal depth was 10501 nodes
clf = DecisionTreeClassifier(max_depth = 10501)
clf.fit(X_train,y_train)
y_predicted = clf.predict(X_test)
print("Final score:",clf.score(X_test,y_test))
from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test, y_predicted, average='macro')<jupyter_output><empty_output><jupyter_text>Logistic Regression  Cross-validation<jupyter_code>from sklearn.linear_model import LogisticRegression

#Cross validation of logistic regression
for i in range(-5,6):
    clf = LogisticRegression(solver = "lbfgs",C = 2**i)
    clf.fit(X_train,y_train)
    print(clf.score(X_validation,y_validation),i)<jupyter_output>0.5452342487883683 -5
0.6203554119547657 -4
0.6663974151857835 -3
0.6583198707592892 -2
0.6615508885298869 -1
0.6672051696284329 0
0.6768982229402262 1
0.691437802907916 2
0.6873990306946688 3
0.6777059773828756 4
0.6680129240710824 5<jupyter_text>Training and evaluation<jupyter_code>from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(solver = "lbfgs",C = 2**2)
clf.fit(X_train,y_train)
y_predicted = clf.predict(X_test)
print(clf.score(X_test,y_test))
from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test, y_predicted, average='macro')<jupyter_output><empty_output><jupyter_text>Support vector machine  Cross-validation<jupyter_code>from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

#Cross validation of SVC C hyperparameter
for i in range(-5,6):
    clf = make_pipeline(StandardScaler(),SVC(C = 2**i,kernel = "linear"))
    clf.fit(X_train,y_train)
    print(clf.score(X_validation,y_validation),i)<jupyter_output>0.5872374798061389 -5
0.5872374798061389 -4
0.5872374798061389 -3
0.5872374798061389 -2
0.5872374798061389 -1
0.5872374798061389 0
0.5872374798061389 1
0.5872374798061389 2
0.5872374798061389 3
0.5872374798061389 4
0.5872374798061389 5<jupyter_text>Training and evaluation<jupyter_code>from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

clf = make_pipeline(StandardScaler(),SVC(C = 1,kernel = "linear"))
clf.fit(X_train,y_train)
y_predicted = clf.predict(X_test)
print(clf.score(X_test,y_test))
from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test, y_predicted, average='macro')<jupyter_output><empty_output><jupyter_text>Naive Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf.fit(X_train,y_train)
y_predicted = clf.predict(X_test)
print(clf.score(X_test,y_test))
from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test, y_predicted, average='macro')<jupyter_output><empty_output><jupyter_text>Ridge classifier  Cross-validation<jupyter_code>from sklearn.linear_model import RidgeClassifier

#Cross validation of regularization parameter
for i in range(-5,6):
    clf = RidgeClassifier(alpha = 2**i)
    clf.fit(X_train,y_train)
    print(clf.score(X_validation,y_validation),i)<jupyter_output>0.6470113085621971 -5
0.6615508885298869 -4
0.6639741518578353 -3
0.6680129240710824 -2
0.684168012924071 -1
0.6930533117932148 0
0.6898222940226171 1
0.6704361873990307 2
0.6615508885298869 3
0.6575121163166397 4
0.6672051696284329 5<jupyter_text>Training and evaluation<jupyter_code>from sklearn.linear_model import RidgeClassifier

clf = RidgeClassifier(alpha = 1)
clf.fit(X_train,y_train)
y_predicted = clf.predict(X_test)
print(clf.score(X_test,y_test))
from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test, y_predicted, average='macro')<jupyter_output><empty_output><jupyter_text>Future work? Deep learning and/or word embeddings?  In this part, we are going to experiment with word embeddings and simple neural networks  Adding pretrained word embeddings - Word2vec<jupyter_code>import gensim.models.keyedvectors as word2vec
import numpy as np

word2vec = word2vec.KeyedVectors.load_word2vec_format('/Users/ivanlovrencic/Personal/FER/GoogleNews-vectors-negative300.bin', binary=True)
from nltk import word_tokenize
from nltk.stem import WordNetLemmatizer 

lemmatizer = WordNetLemmatizer() 

X = []
y = []
for post in posts:
    vector = np.zeros(300)
    tokens = word_tokenize(post)
    
    for token in tokens:
        lemma = lemmatizer.lemmatize(token.lower())
        if(lemma in word2vec.vocab):
            vector += word2vec[lemma]
    X.append(vector)

for t in truth:
    y.append(t)

print(len(X))
print(len(y))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
X_train = np.array(X_train)
X_test = np.array(X_test)
y_train = np.array(y_train)
y_test = np.array(y_test)<jupyter_output><empty_output><jupyter_text>Simple neural network just to test the word embeddings<jupyter_code># TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

model = keras.Sequential([
    keras.layers.Input(shape = (300,)),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(optimizer = "adam",metrics=['accuracy'],loss = tf.keras.losses.Hinge())
model.fit(X_train,y_train,epochs = 30)
test_loss,test_acc = model.evaluate(X_test,y_test)<jupyter_output>52/52 [==============================] - 0s 850us/step - loss: 0.7425 - accuracy: 0.6806
{'fraction_non_alphanumeric': 0.09888304862023653, 'fraction_numerical': 0.11654073587385019, 'mean_word_length': 6.846005154639175, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '265835', 'license': 'MIT', 'n_tokens_mistral': 5264, 'n_tokens_neox': 4369, 'n_words': 902, 'path': '.ipynb_checkpoints/Identifying-Depression-with-Machine-Learning-checkpoint.ipynb', 'repo_name': 'ilovrencic/Identifying-Depression-with-Machine-Learning', 'size': '12176'}