<jupyter_start><jupyter_text>AI SATURDAYS DONOSTIA 2020 Regresión Indicador "LowMood" (Bajo Estado de Ánimo)     Proyecto Práctico Equipo FACEMOOD<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import linear_model
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.metrics import accuracy_score
import statsmodels.api as sm
from sklearn.model_selection import train_test_split
from regresion_functions import *
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Conjunto de Datos<jupyter_code>df = pd.read_csv('../processed-data/tidy_socialmedia_data.csv', index_col=0)
df.drop('full_count', axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Las 11 variables estudiadas (8 relacionadas a "depresión" y 3 relacionadas al uso de "Redes Sociales")<jupyter_code>df2 = df.iloc[:,12:]
df2.head()<jupyter_output><empty_output><jupyter_text>Estadísticas descriptivas de las 11 variables<jupyter_code>df2.describe()<jupyter_output><empty_output><jupyter_text>Medias por participante<jupyter_code>dfmean=df.groupby('Participant').mean()
dfmean=dfmean.iloc[:,6:]
dfmean=dfmean[["ASMU", "News", "PSMU", "Stress", "Hopeless", "Inferior", "LossOfInt", "Concentrat", "Loneliness", "Fatigue", "LowMood"]]
print(dfmean.head())
print("No. Filas/Columnas del Conjunto de Datos: {}".format(dfmean.shape))<jupyter_output>ASMU       News       PSMU     Stress   Hopeless   Inferior  \
Participant                                                                     
115091       16.792208  15.012987  32.883117  37.441558  21.389610  17.831169   
131183       28.254237  11.593220  45.203390  16.898305   0.406780   0.254237   
438907       27.040816  34.645833  44.595745  25.000000  14.591837  23.395833   
515070       37.826923  38.576923  25.711538  17.365385  10.500000  11.403846   
572172       40.545455  45.250000  36.852273  20.511364  17.431818  38.056818   

             LossOfInt  Concentrat  Loneliness    Fatigue    LowMood  
Participant                                                           
115091       47.675325   37.272727   23.285714  45.155844  20.454545  
131183       14.508475   19.508475    0.389831  36.288136   3.474576  
438907       50.979592   35.729167   21.833333  51.040816  26.285714  
515070       16.038462   32.903846   14.076923  34.865385  12.692308  
572172[...]<jupyter_text>Estadísticas descriptivas de las medias por participante<jupyter_code>dfmean.describe()<jupyter_output><empty_output><jupyter_text>Diagrama de Matriz para las Medias de las 11 Variables<jupyter_code>printMatrixDiagram(dfmean)   # Función definida en "regresion_functions"<jupyter_output><empty_output><jupyter_text>Correlaciones de Pearson para las Medias de las 11 Variables<jupyter_code>printPearsonCorrelations(dfmean)   # Función definida en "regresion_functions"<jupyter_output><empty_output><jupyter_text>Se observan correlaciones más significativas entre las siguientes variables:Fatigue vs ConcentratFatigue vs LossOfIntLowMood vs LonelinessLowMood vs InferiorLowMood vs HopelessLowMood vs StressLoneliness vs InferiorLoneliness vs Hopeless (90,3%, indicando "multicolinealidad")Loneliness vs StressConcentrat vs LossOfInt Inferior vs Hopeless (90,7%, indicando "multicolinealidad")Hopeless vs Stress  Regresión Lineal para las Medias: y = DeprMood, X = Demás Variables<jupyter_code>dfmean=df.groupby('Participant').mean()
dfmean=dfmean.iloc[:,6:]

label = dfmean.LowMood
dfmean.drop('LowMood', axis=1, inplace=True)
dfmean.drop('Hopeless', axis=1, inplace=True)   # Muy correlacionado con Loneliness & Inferior (>90% -> multicolinealidad)<jupyter_output><empty_output><jupyter_text>Proceso de eliminación de variables X que no contribuyen significativamente para explicar y<jupyter_code>resultsummary = pd.DataFrame(data={'iteration': [], 'intercept': [], 'RMSE_Training': [], 'RMSE_Testing': [],
                                   'R2_Training': [],'R2_Testing': [],'p_value_max':[],'removed_var':[]})

data_list_medias = calculateRegression(dfmean, label, resultsummary, alpha=0.15)   # Función definida en "regresion_functions"<jupyter_output>iteration  intercept  RMSE_Training  RMSE_Testing  R2_Training  R2_Testing  \
0        0.0     -1.628          3.254         3.815        0.915       0.885   
1        1.0     -1.623          3.254         3.814        0.915       0.885   
2        2.0     -1.438          3.256         3.809        0.915       0.885   
3        3.0     -1.359          3.265         3.785        0.915       0.887   
4        4.0     -1.281          3.299         4.111        0.913       0.866   
5        5.0     -0.341          3.347         4.132        0.910       0.865   
6        6.0     -0.634          3.366         4.009        0.909       0.873   

   p_value_max removed_var  
0        0.878  Concentrat  
1        0.671        PSMU  
2        0.628   LossOfInt  
3        0.189    Inferior  
4        0.308     Fatigue  
5        0.194        ASMU  
6        0.133           -  

Modelo Final
['Loneliness', 'Stress', 'News']
[0.78608676 0.17296278 0.03979688] -0.6337034002841335
RMSE of Linear Re[...]<jupyter_text>Análisis de Residuos Modelo Final<jupyter_code>fitt = data_list_medias[5]
standardized_residuals = data_list_medias[4]

residualAnalysis(fitt, standardized_residuals)   # Función definida en "regresion_functions"<jupyter_output>Estadística prueba normalidad Kolmogorov-Smirnov=0.133, pvalue=0.053

Probablemente Normal<jupyter_text>Se observan 2 outliers en el análisis de residuos del modelo final. Próximos pasos: Eliminar estos 2 outliers del conjunto de datos y volver a ajustar el modelo final.<jupyter_code># Eliminar Outliers

X_train = data_list_medias[0]
y_train = data_list_medias[2]
standardized_residuals = data_list_medias[4]

eliminacionoutliers = eliminateOutliers(X_train, y_train, standardized_residuals)   # Función definida en "regresion_functions"
# Volver a ajustar el Modelo Final sin Outliers

X_train = eliminacionoutliers[0]
y_train = eliminacionoutliers[1]
X_test = data_list_medias[1]
y_test = data_list_medias[3]

resultsummary = pd.DataFrame(data={'iteration': [], 'intercept': [], 'RMSE_Training': [], 'R2_Training': []})

data_list_medias_sinoutlier = repeatRegression(X_train, y_train, X_test, y_test, resultsummary)   # Función definida en "regresion_functions"<jupyter_output>iteration  intercept  RMSE_Training  R2_Training  R2_Testing  RMSE_Testing
0        0.0     -0.855          2.909        0.931       0.874         3.998

Modelo Final Sin Outlier
['Loneliness', 'Stress', 'News']
[0.78080015 0.17884387 0.03550915] -0.85504395813917
RMSE of Linear Regression Model with Training Data: 2.91
RMSE of Linear Regression Model with Testing Data: 4.00
R2 Coefficient for Linear Regression Model with Training Data: 0.931
R2 Coefficient for Linear Regression Model with Testing Data: 0.874<jupyter_text>Análisis de Residuos Modelo Final sin Outliers<jupyter_code>fitt = data_list_medias_sinoutlier[3]
standardized_residuals = data_list_medias_sinoutlier[2]

residualAnalysis(fitt, standardized_residuals)   # Función definida en "regresion_functions"<jupyter_output>Estadística prueba normalidad Kolmogorov-Smirnov=0.110, pvalue=0.172

Probablemente Normal
{'fraction_non_alphanumeric': 0.08766675741900354, 'fraction_numerical': 0.11570922951265995, 'mean_word_length': 3.4935779816513763, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '622705', 'license': 'MIT', 'n_tokens_mistral': 3131, 'n_tokens_neox': 2632, 'n_words': 575, 'path': 'scripts/Regresion_Low_Mood.ipynb', 'repo_name': 'henrycorazza/AISaturdays-depresion-rrss', 'size': '7346'}