<jupyter_start><jupyter_text>IEQ and Survey Response AnalysisExploring the relationship between the categorical survey responses from the EMA and the IEQ measurements from the Beacon<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Extreme IEQ's Affect on Mood ReportsDo more extreme measurements influence survey results?<jupyter_code>import os
import sys
sys.path.append('../')

from src.visualization import visualize

import pandas as pd
pd.set_option('display.max_columns', 200)
import numpy as np

from datetime import datetime, timedelta

import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.dates as mdates

from scipy import stats
from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Table of Contents 1. [Data Import](data_import)2. [Pre-Processing](pre_processing)3. [Inspection](inspection)4. [Analysis](analysis) ---  [Back to ToC](toc) Data ImportWe have two datasets to import:  EMAs Completed at HomeIntersection between GPS coordinates, home address, and the completion of the EMAs.<jupyter_code>ema = pd.read_csv("../data/processed/beiwe-ema_at_home_v2-ux_s20.csv",index_col="timestamp",parse_dates=["timestamp"],infer_datetime_format=True)
for column in ema.columns:
    if column != "beiwe":
        ema[column] = pd.to_numeric(ema[column])
ema["discontent"] = 3 - ema["content"]
ema.head()<jupyter_output><empty_output><jupyter_text>IEQ DataWe will be using all of the IEQ data to identify these extreme events.<jupyter_code>ieq = pd.read_csv('../data/processed/beacon-ux_s20.csv',index_col="timestamp",parse_dates=["timestamp"],infer_datetime_format=True)
ieq.drop(["beacon","redcap","pm1_number","pm2p5_number","pm10_number","pm1_mass","pm10_mass","no2","lux","co"],axis=1,inplace=True)
for column in ieq.columns:
    if column != "beiwe":
        ieq[column] = pd.to_numeric(ieq[column])
ieq.head()<jupyter_output><empty_output><jupyter_text>[Back to Data Import](data_import) ---  [Back to ToC](toc) Pre-Processing  Getting Periods with Extreme IEQ MeasurementsLooking for periods in the IEQ data that are above or below a certain threshold.<jupyter_code>def get_extreme(df_in,ieq_param,n_std=2,above=True):
    """
    Gets the periods of IEQ measurements for the given parameter above or below a certain threshold
    
    Parameters
    ----------
    df_in : DataFrame
        IEQ data for all participants
    ieq_param : string
        specifies which column in df_in to consider
    n_std : int or float
        number of standard deviations to consider for extreme
    above : boolean
        whether to look for measurements above or below the mean - if None, considers both
        
    Returns
    -------
    df_extreme : 
    """
    df = df_in.copy()
    df_extreme = pd.DataFrame()
    for pt in df["beiwe"].unique():
        df_pt = df[df["beiwe"] == pt]
        try:
            df_pt_ieq = df_pt[[ieq_param,"beiwe"]]
        except KeyError:
            print("IEQ parameter not given in dataframe")
            return
        
        avg = np.nanmean(df_pt_ieq[ieq_param])
        std = np.nanstd(df_pt_ieq[ieq_param])
        if above == True:
            df_pt_extreme = df_pt_ieq[df_pt_ieq[ieq_param] > avg + n_std*std]
        elif above == False:
            df_pt_extreme = df_pt_ieq[df_pt_ieq[ieq_param] < avg - n_std*std]
        else:
            df_pt_extreme = df_pt_ieq[(df_pt_ieq[ieq_param] < avg - n_std*std) | (df_pt_ieq[ieq_param] > avg + n_std*std)]
            
        df_agg = df_pt_extreme.reset_index()
        df_agg["start"] = df_agg["timestamp"]
        df_agg.drop("beiwe",axis="columns",inplace=True)
        df_agg = (
            df_agg.groupby(
                df_agg['timestamp'].diff().gt(pd.Timedelta(minutes=2)).cumsum()
            ).agg({
                'timestamp': 'last', ieq_param: 'mean','start': 'first'
            }).set_index('timestamp').rename_axis(index="end_time")
        )
        df_agg["beiwe"] = pt
        df_agg["extreme_minutes"] = (df_agg.index - df_agg["start"]).dt.total_seconds()/60
        df_agg["mean"] = avg
        df_agg["std"] = std
        df_agg.drop("start",axis="columns",inplace=True)
        df_extreme = df_extreme.append(df_agg)
    
    return df_extreme
extreme = get_extreme(ieq,"co2",above=True)<jupyter_output><empty_output><jupyter_text>Combining Extreme IEQ events with SurveysNow that we have periods with the extreme IEQ measurements, we can merge them with the surveys to get a sense of how the two relate.<jupyter_code>def combine_ieq_with_ema(ieq_in, ema_in, day_threshold=2):
    """
    Combines ieq and ema data based on timestamps of extremem events and ema submissions
    """
    ieq = ieq_in.reset_index()
    ema = ema_in.reset_index()
    df = pd.DataFrame()
    for pt in ieq_in["beiwe"].unique():
        ieq_pt = ieq[ieq["beiwe"] == pt]
        ema_pt = ema[ema["beiwe"] == pt]
        for event in ieq_pt["end_time"]:
            for submission in ema_pt["timestamp"]:
                delay = (submission - event).total_seconds() / 60 / 60 / 24
                if submission > event and delay < day_threshold:
                    ieq_pt["delay_days"] = delay
                    df = df.append(ieq_pt[ieq_pt["end_time"] == event].merge(ema_pt[ema_pt["timestamp"] == submission],on=["beiwe"]))
                    break
    
    df.drop_duplicates(subset=["timestamp"],keep="last",inplace=True)
    return df
combine_test = combine_ieq_with_ema(extreme,ema)<jupyter_output><empty_output><jupyter_text>EvaluatingNow that we have the link between the mood and the extreme IEQ events, we can loop through each of the IEQ parameters and see if the _extreme_ events lead to any differences.Some tests to evaluate the differences in the responses:* Mann-Whitney U-Test* Ordinal Regression<jupyter_code>def compare_mood_scores(extreme_in,ema_in,moods=["discontent","stress","sad","lonely","energy"],f=np.nanmean):
    """
    Compares the mood scores between the extreme and non-extreme cases
    """
    res = {"mean_normal":[],"mean_ext":[],"p":[]}
    extreme = extreme_in.copy()
    ema = ema_in.copy()
    normal_ema = ema[~ema.index.isin(extreme["timestamp"])]
    normal_ema = normal_ema[normal_ema["beiwe"].isin(extreme["beiwe"].unique())].sort_values("beiwe")
    print(f"Normal: \t{len(normal_ema)}\nExtreme:\t{len(extreme)}")
    for mood in moods:
        ext_mean = round(np.nanmean(extreme[mood]),2)
        norm_mean = round(np.nanmean(normal_ema[mood]),2)
        ext_std = round(np.nanstd(extreme[mood]),2)
        norm_std = round(np.nanstd(normal_ema[mood]),2)
        u, p = stats.mannwhitneyu(normal_ema[mood].values,extreme[mood].values)
        if p < 0.05:
            p = f"{round(p,3)}*"
        elif p < 0.1:
            p = f"{round(p,3)}**"
        else:
            p = f"{round(p,3)}"
        for key, val in zip(res.keys(),[(norm_mean,norm_std),(ext_mean,ext_std),p]):
            if len(val) == 2:
                res[key].append(f"{val[0]} ({val[1]})")
            else:
                res[key].append(val)
        #print(f"\tExtreme ({len(extreme[mood])}):\t{ext_response}\n\tNormal ({len(normal_ema[mood])}):\t{norm_response}")
        #print(f"\tStatistic:\t{round(p,3)}")
    print(pd.DataFrame(data=res,index=moods).to_latex())
compare_mood_scores(combine_test,ema)<jupyter_output>Normal: 	544
Extreme:	125
\begin{tabular}{llll}
\toprule
{} &  mean\_normal &     mean\_ext &       p \\
\midrule
discontent &  0.92 (0.89) &   1.07 (0.9) &   0.04* \\
stress     &  0.76 (0.83) &   0.9 (0.87) &  0.043* \\
sad        &  0.41 (0.73) &  0.38 (0.67) &   0.437 \\
lonely     &  0.43 (0.74) &  0.35 (0.68) &   0.119 \\
energy     &  2.06 (1.06) &  2.33 (0.89) &  0.003* \\
\bottomrule
\end{tabular}<jupyter_text>[Back to Pre-Processing](pre_processing) ---  [Back to ToC](toc) InspectionVarious functions and initial looks at the raw and pre-processed data. [Back to Inspection](inspection) ---  [Back to ToC](toc) Analysis   Effect of Extreme IEQ on Mood<jupyter_code>for ieq_param in ["co2","pm2p5_mass","temperature_c","rh"]:
    print(ieq_param.upper(),"\n")
    extreme = get_extreme(ieq,ieq_param,above=True)
    combined = combine_ieq_with_ema(extreme,ema)
    compare_mood_scores(combined,ema)<jupyter_output>CO2 

Normal: 	544
Extreme:	125
\begin{tabular}{llll}
\toprule
{} &  mean\_normal &     mean\_ext &       p \\
\midrule
discontent &  0.92 (0.89) &   1.07 (0.9) &   0.04* \\
stress     &  0.76 (0.83) &   0.9 (0.87) &  0.043* \\
sad        &  0.41 (0.73) &  0.38 (0.67) &   0.437 \\
lonely     &  0.43 (0.74) &  0.35 (0.68) &   0.119 \\
energy     &  2.06 (1.06) &  2.33 (0.89) &  0.003* \\
\bottomrule
\end{tabular}

PM2P5_MASS 

Normal: 	492
Extreme:	209
\begin{tabular}{llll}
\toprule
{} &  mean\_normal &     mean\_ext &       p \\
\midrule
discontent &  0.97 (0.91) &  0.95 (0.82) &   0.469 \\
stress     &  0.79 (0.86) &  0.79 (0.84) &   0.431 \\
sad        &  0.42 (0.76) &    0.4 (0.7) &   0.473 \\
lonely     &  0.45 (0.78) &  0.43 (0.71) &   0.478 \\
energy     &  2.07 (1.05) &  2.19 (1.02) &  0.07** \\
\bottomrule
\end{tabular}

TEMPERATURE_C 

Normal: 	477
Extreme:	89
\begin{tabular}{llll}
\toprule
{} &  mean\_normal &     mean\_ext &       p \\
\midrule
discontent &  0.92 (0.82) &  1[...]
{'fraction_non_alphanumeric': 0.1441615640777742, 'fraction_numerical': 0.034160489848533676, 'mean_word_length': 3.161823871256147, 'pattern_counts': {'":': 5, '<': 36, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '332193', 'license': 'MIT', 'n_tokens_mistral': 3553, 'n_tokens_neox': 3365, 'n_words': 709, 'path': 'notebooks/4.5.4-hef-beacon_ema-extreme_ieq_and_mood.ipynb', 'repo_name': 'intelligent-environments-lab/utx000', 'size': '9309'}