<jupyter_start><jupyter_text>For further analysis the dtasets need to be merged, the user id feature can be used to establish links between the datasets. I have also created a new feature for each patient, named percentage adherence, this indicates throughtout the treatment how what was the ration of medicines taken over prescribed dose. 0 indicating no medicines were taken and 1 indicates no doses were missed.<jupyter_code>bp_meds = pd.merge(bp, meds, on = ["user_id", "date"])
pdata = pd.merge(bp_meds, users, on = ["user_id"])
pdata=pdata.replace({'sbp': {0: pdata["sbp"].median()}}) 
pdata=pdata.replace({'dbp': {0: pdata["dbp"].median()}}) 

temp= pdata.groupby(by="user_id").mean().reset_index()
temp=temp.rename(columns = {'pill_detect':'pct_adh'}) # new feature
pdata = pd.merge(pdata, temp.iloc[:,[0, 3]], on="user_id")<jupyter_output><empty_output><jupyter_text>Data Statisitics.<jupyter_code>pdata.head() #this is how the data looks now<jupyter_output><empty_output><jupyter_text>Before proceeding with the analysis, understanding the distribution helps. From the distribution plots it can be observed that the blood pressure levels are fairly normal distributed.<jupyter_code>from scipy import stats
sns.set(rc={'figure.figsize':(10,8)})

sns.distplot(pdata.sbp, kde=True, fit=stats.gamma, label="Systolic Blood Pressure - sbp");
sns.distplot(pdata.dbp, kde=True , fit=stats.gamma, label="Diastolic Blood Pressure - dbp"); 

#fit - to approximate distributiom
plt.legend()<jupyter_output><empty_output><jupyter_text>The mean adherence level; that is, looking at all the data, overall adehrence is 37%.<jupyter_code>np.sum(pdata.pill_detect)/len(pdata.pill_detect) ## percentage of times pill was detected<jupyter_output><empty_output><jupyter_text>I was curious of the participants in the study, turns out male to female ratio is 1!<jupyter_code>users.groupby(by="gender").count().iloc[:,1].plot(kind="bar",figsize=(5,5))<jupyter_output><empty_output><jupyter_text>Does the day of week factor has any effect on adherence?  Turns out there is no noticeable effect.<jupyter_code>pdata["date"] = pd.to_datetime(pdata.date)
pdata["day"] = pdata["date"].dt.weekday_name
temp = pdata[pdata["pill_detect"] == True]
temp = temp.groupby(by="day").count().reset_index().iloc[:,1:2].plot(kind="bar",figsize=[7,7])
# So there is no noticeable effect of day on adherence<jupyter_output><empty_output><jupyter_text>Part - 1 Is there a relationship between medical adherence (i.e., how often somebody takes their medicine) and blood pressure ?   To answer the question above  is there a noticeable difference in mean bp levels for days when pills were taken vs Not?<jupyter_code>sns.factorplot(x="gender", y="sbp",
                    hue="pill_detect",
                    data=pdata, kind="box",
                    size=10, aspect=.7);<jupyter_output><empty_output><jupyter_text>From the plot we can see that the mean blood pressure levels are lower when the people are taking the pills vs not<jupyter_code>print(pdata[(pdata['pill_detect'] == True)].iloc[:,2:4].describe())
print(pdata[(pdata['pill_detect'] == False)].iloc[:,2:4].describe())
users["pct_adh"] = pdata["pct_adh"]
interaction_plot(pdata["pill_detect"], pdata["gender"], pdata["sbp"], func=np.mean,colors=['pink','blue'])<jupyter_output><empty_output><jupyter_text>From the interaction plot we can see that the percentage adherence factor that we created earlier has an impact on blood pressure, lower level are on higher end whereas high adherence has low mean  To gather more stastical evidence, we can use anova analysis and linear model with response variable as blood pressure.<jupyter_code>import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.graphics.api import interaction_plot, abline_plot
from statsmodels.stats.anova import anova_lm


interX_lm = ols("sbp ~ pct_adh", pdata.iloc[:,2:]).fit()
print(interX_lm.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                    sbp   R-squared:                       0.021
Model:                            OLS   Adj. R-squared:                  0.021
Method:                 Least Squares   F-statistic:                     5532.
Date:                Tue, 13 Mar 2018   Prob (F-statistic):               0.00
Time:                        02:49:41   Log-Likelihood:            -7.8725e+05
No. Observations:              255101   AIC:                         1.575e+06
Df Residuals:                  255099   BIC:                         1.575e+06
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>From the model sumary we can see that the mean adherence feature has a stastical impact on patients blood pressure levels. The magnitude of this impact can be upto 4.7 point change in blood pressure with a unit change in adherence.   We can also use t test to strengthen our analysis, with the hypothesis, are the population with lower than .5 adherence is different than than the population with adherence higher than .5<jupyter_code>low_adh = pdata[pdata['pct_adh'] < 0.5 ]['sbp']
high_adh = pdata[pdata['pct_adh'] > 0.5]['sbp']
stats.ttest_ind(low_adh, high_adh)
#stats.ttest_rel(pill_true, pill_false)<jupyter_output><empty_output><jupyter_text>and the answer is yes, the probability of both populations being similar is close to 0.   We could also study this as a series, where over time with resampling for each user. This will help us see how the blood pressure for a patient(patient with low adherence vs patient with high adherence) changes over time.<jupyter_code>#dc.pivot(columns='disease_header', index='medicine', values='disease').reset_index()
pv_data = pdata.pivot(columns="user_id", index="date", values="sbp").reset_index()

pv_data.columns = pv_data.columns.astype(str)
pv_data.set_index("date", inplace=True)
pv_data.index = pd.to_datetime(pv_data.index)

usr_meds = pd.merge(users, meds, on = ["user_id"]);
user_adh = usr_meds.groupby(by=["user_id"]).mean()
user_adh.sort_values(by="pill_detect")

low_adh = list(user_adh[user_adh["pill_detect"] < 0.4].head(200).index) #low adheremce level
high_adh = list(user_adh[user_adh["pill_detect"] > 0.7].head(200).index) #high adherence levels
high_adh = [str(i) for i in list(high_adh)]
low_adh = [str(i) for i in list(low_adh)]
pv_data = pv_data[3:-5] ## too many missing values in first and last few days 

sns.set(rc={'figure.figsize':(15,8)})

sns.tsplot([pv_data[i] for i in high_adh], time=pv_data.index, estimator=np.nanmean, color="green", ci=[90]) #high adherence >80
sns.tsplot([pv_data[i] for i in low_adh], time=pv_data.index, estimator=np.nanmean, color="red",  ci=[90]) #low adherence <30<jupyter_output><empty_output><jupyter_text>red - low adherence , green high adherence From the plot it is clear that pathent with high adherence have much better blood pressure levels at the end of the treatment than patients with low adherence. Even though both groups started at around same levels.<jupyter_code>import numpy as np; np.random.seed(22)
import seaborn as sns; sns.set(color_codes=True)

data = pdata[(pdata.user_id.isin(high_adh) )].iloc[:,1:3]
data1 = pdata[(pdata.user_id.isin(low_adh) )].iloc[:,1:3]
data.set_index('date').plot(color ="green", kind="hist") ##Skewed to left - which is good
data1.set_index('date').plot(color ="red", kind="hist") ##Sekwed to right<jupyter_output><empty_output><jupyter_text>Another way to understand this is using distributions of low and hight adherence individuals. We can clearly see that the high adherence distributions is left skewed - most of the values are on the lower side. Low adherence distribution is exact opposite.  Part  - 2 Users has about 2 months of data, and some of those users have probably changed their medical adherence during that span of  this time. Do users that increase their  frequency see a change in blood pressure ? If so, how much does it change?  To answer this question, we need a much different representation of our data. I decided, to go with weekly adherence, this indicates, how many times a person has taken their medicine over the prescribed dosage. For each day past 7 days adherence was calculated and used as a feature. The intention is that this number will indicate recent adherence and can be used to see the effect<jupyter_code>temp = pdata.groupby(by="user_id")['pill_detect'].apply(lambda x: x.shift().rolling(min_periods=1,window=7).mean()).reset_index(name="weekly_adh")
pdata["weekly_adh"] = temp["weekly_adh"]<jupyter_output><empty_output><jupyter_text>Now to understand, if it has an impact and what is the magnitude of this impact, a linear model was created, where the coefficient of the weekly adherence feature would indicate the impact.<jupyter_code>import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.graphics.api import interaction_plot, abline_plot
from statsmodels.stats.anova import anova_lm


interX_lm = ols("sbp ~ weekly_adh", pdata.iloc[:,2:]).fit()
print(interX_lm.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                    sbp   R-squared:                       0.028
Model:                            OLS   Adj. R-squared:                  0.028
Method:                 Least Squares   F-statistic:                     7341.
Date:                Tue, 13 Mar 2018   Prob (F-statistic):               0.00
Time:                        03:01:12   Log-Likelihood:            -7.7268e+05
No. Observations:              250692   AIC:                         1.545e+06
Df Residuals:                  250690   BIC:                         1.545e+06
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>The weekly adherence has a significant impact (p-val ~ 0), and magnitude of this impact is upto 3 units in systolic blood pressure.<jupyter_code>sns.set(rc={'figure.figsize':(18,12)})
#f, ax = plt.subplots(figsize=(5, 6))
sns.jointplot(x='weekly_adh', y='sbp',  data=pdata, size=10,kind="reg")<jupyter_output><empty_output><jupyter_text>The plot suggests an inverse relation betwen weekly adherence and systolic blood pressure. Higher adherence results in lower sbp levels.<jupyter_code>sns.pairplot(pdata, x_vars=["weekly_adh", "pct_adh"], y_vars=["sbp"],
             hue="pill_detect", size=12, aspect=.8, kind="reg");
temp = pdata[(pdata["bmi"] < 25) & (pdata["bmi"] > 18)].groupby("user_id").mean().reset_index()
sns.jointplot(x='bmi', y='pct_adh',  data=temp, size=10,kind="reg")
interX_lm = ols("weekly_adh ~ gender", pdata.iloc[:,2:]).fit()
print(interX_lm.summary())

# Males have a better medical adherence level , but only by 7%. With statistical support
pv_data = pdata.pivot(columns="user_id", index="date", values="sbp").reset_index()
pv_data.columns = pv_data.columns.astype(str)
pv_data = pv_data[7:] #ignore first 7 days 
low_adhw = list(pdata[pdata["weekly_adh"] < 0.3].head(200).index) #low adheremce level
high_adhw = list(pdata[pdata["weekly_adh"] > 0.85].head(200).index) #high adherence levels

high_adhw = [str(i) for i in list(high_adh)]
low_adhw= [str(i) for i in list(low_adh)]

sns.set(rc={'figure.figsize':(12,6)})
## high vs low adherence groups 
sns.tsplot([pv_data[i] for i in high_adhw], time=pv_data.index, estimator=np.nanmean, color="green", ci=[80]) #high adherence >80
sns.tsplot([pv_data[i] for i in low_adhw], time=pv_data.index, estimator=np.nanmean, color="red",  ci=[80]) #low adherence <30
all_users = [str(i) for i in list(users.user_id)]
print(set(all_users) - set(pv_data.columns))
#all_users.remove('102951')<jupyter_output>{'102951'}<jupyter_text>trends in adherence<jupyter_code>pdata2 = pdata

pdata2.sbp = (pdata2.sbp - pdata2.sbp.min())/(pdata2.sbp.max() - pdata2.sbp.min() )
pv_data0 = pdata2.pivot(columns="user_id", index="date", values="sbp").reset_index()
pv_data0 = pv_data0[7:]
pv_data0.columns = pv_data0.columns.astype(str)
pv_data0.set_index("date", inplace=True)
pv_data0.index = pd.to_datetime(pv_data0.index)

all_users = [str(i) for i in list(pv_data0.columns)]
pv_data1 = pdata2.pivot(columns="user_id", index="date", values="weekly_adh").reset_index()
pv_data1 = pv_data1[7:]
pv_data1.columns = pv_data1.columns.astype(str)
#pv_data1.set_index("date", inplace=True)
pv_data1.index = pd.to_datetime(pv_data1.index)

sns.set(rc={'figure.figsize':(9,6)})
sns.tsplot([pv_data1[i] for i in all_users], time=pv_data1.index, estimator=np.nanmean, color="green",  ci=[95]) #overall bp
sns.tsplot([pv_data0[i] for i in all_users], time=pv_data1.index, estimator=np.nanmean, color="blue",  ci=[95]) 
#resample and reduce the window to 3 for adherence 
temp = pdata.groupby(by="user_id")['pill_detect'].apply(lambda x: x.shift().rolling(min_periods=1,window=3).mean()).reset_index(name="weekly_adh")
pdata["weekly_adh"] = temp["weekly_adh"] 
pdata2 = pdata

pdata2.sbp = (pdata2.sbp - pdata2.sbp.min())/(pdata2.sbp.max() - pdata2.sbp.min() )
pv_data0 = pdata2.pivot(columns="user_id", index="date", values="sbp").reset_index()
pv_data0 = pv_data0[7:]
pv_data0.columns = pv_data0.columns.astype(str)
pv_data0.set_index("date", inplace=True)
pv_data0.index = pd.to_datetime(pv_data0.index)

all_users = [str(i) for i in list(pv_data0.columns)]
pv_data1 = pdata2.pivot(columns="user_id", index="date", values="weekly_adh").reset_index()
pv_data1 = pv_data1[7:]
pv_data1.columns = pv_data1.columns.astype(str)
#pv_data1.set_index("date", inplace=True)
pv_data1.index = pd.to_datetime(pv_data1.index)
sns.set(rc={'figure.figsize':(12,6)})
sns.tsplot([pv_data1[i] for i in all_users], time=pv_data1.index, estimator=np.nanmean, color="green",  ci=[95]) #overall bp
sns.tsplot([pv_data0[i] for i in all_users], time=pv_data1.index, estimator=np.nanmean, color="blue",  ci=[95]) 
sns.tsplot([pv_data[i] for i in high_adhw], time=pv_data.index, estimator=np.nanmean, color="green", ci=[80]) #high adherence >80
sns.tsplot([pv_data[i] for i in low_adhw], time=pv_data.index, estimator=np.nanmean, color="red",  ci=[80]) #low adherence <30<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1442989634478996, 'fraction_numerical': 0.02284506273867976, 'mean_word_length': 4.326916091536505, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '141902', 'license': 'Apache-2.0', 'n_tokens_mistral': 5063, 'n_tokens_neox': 4734, 'n_words': 1279, 'path': 'EDA_Sensor_Data.ipynb', 'repo_name': 'rpwr021/data-challenge', 'size': '14664'}