<jupyter_start><jupyter_text>About the data: The data is from an ongoing cardiovascular study on residents of the town of Framingham, Massachusetts.  The data includes more than 4000 participants and 15 columns of medical and behavioral parameters and the target column is TenYearCHD. About "TenYearCHD",The classification goal is to predict whether the patient has a 10-year risk of future coronary heart disease (CHD).<jupyter_code>url= 'https://raw.githubusercontent.com/YogevLD/DS_test/main/CHD_preprocessed%20(1).csv'
heart_diseases= pd.read_csv(url)
heart_diseases.head(5)<jupyter_output><empty_output><jupyter_text>We will checking the data:- Treat missing values (if needed)- Treat data types (if needed)- Treat column names (if needed)  Treat missing values Check if there are missing values:<jupyter_code>pd.DataFrame(heart_diseases.isnull().sum())<jupyter_output><empty_output><jupyter_text>Fixing data types Check if any of the data types need to be fixed:<jupyter_code>pd.DataFrame(heart_diseases.dtypes)<jupyter_output><empty_output><jupyter_text>- Creates two new columns that will hold categorical rather than Boolean values for the male and TenYearCHD columns,in order to make it more readable to display them.<jupyter_code>heart_diseases["male/female"]=heart_diseases["male"].replace({1:"male",0:"female"})
heart_diseases["Chance of getting sick"]=heart_diseases["TenYearCHD"].replace({1:"Yes",0:"No"})
heart_diseases.head(4)<jupyter_output><empty_output><jupyter_text>Understanding the data.   How many male and female in the data?<jupyter_code>pd.DataFrame(heart_diseases['male/female'].value_counts())
heart_diseases['male/female'].value_counts().plot.pie(label="male/ female",autopct='%1.1f%%')<jupyter_output><empty_output><jupyter_text>Study the differences between males and females:<jupyter_code>heart_diseases_male = heart_diseases[heart_diseases.male==1].copy()
heart_diseases_male.head(2)
heart_diseases_female = heart_diseases[heart_diseases.male==0].copy()
heart_diseases_female.head(2)<jupyter_output><empty_output><jupyter_text>Who smokes more- Men or women?<jupyter_code>ax=heart_diseases.pivot_table("cigsPerDay","male/female",aggfunc=sum).plot(kind="bar",colormap="GnBu_r")
ax.set_title('Cigarettes per day for men and women',fontsize=16)
ax.set_ylabel("Cigarettes per day",fontsize=12)
ax.set_xlabel("Male/Female")<jupyter_output><empty_output><jupyter_text>First conclusion.- We can learn an important lesson - men smoke more than women, this is despite the fact that there are almost 15% more women in the data and that reinforces our claim about who smokes more.  How does the amount of cigarettes that people smoke during their lives varies with time?- Since the most abudent data is about 40 and 50 years old individuals, we separated this groups into two such that the groups are from 30-50 and 50-70 years old. This will maintain a balanced size groups.<jupyter_code>fig,axes = plt.subplots(figsize=(20,5),ncols=2)
McigsPerDay= pd.cut(heart_diseases_male["cigsPerDay"],[-1,10,20,30,40,50,60])
FcigsPerDay= pd.cut(heart_diseases_female["cigsPerDay"],[-1,10,20,30,40,50,60])

Mage = pd.cut(heart_diseases_male['age'], [30,50,70])
Fage = pd.cut(heart_diseases_female['age'], [30,50,70])

sns.countplot(ax=axes[0],data= heart_diseases_male,x=McigsPerDay,hue=Mage,palette="colorblind")
axes[0].set_title('Male',fontsize=18)
axes[0].legend(loc="upper right",title="Age",fontsize=14)
axes[0].set_ylabel("Count of people",fontsize=14)
axes[0].set_xlabel("Cigarettes per day",fontsize=14)
axes[0].set(ylim=(-1,1000))
sns.countplot(ax=axes[1],data=heart_diseases_female,x=FcigsPerDay,hue=Fage,palette="colorblind")
axes[1].set_title('Female',fontsize=18)
axes[1].legend(loc="upper right",title="Age",fontsize=14)
axes[1].set_ylabel("Count of people",fontsize=14)
axes[1].set_xlabel("Cigarettes per day",fontsize=14)
axes[1].set(ylim=(-1,1000))
plt.show()<jupyter_output><empty_output><jupyter_text>Second conclusion.- Although older individuals smoke more 1-10 cigerettes per day, overall they smoke less than younger individuals does. In addition, it can be seen that in the category of 1-10 cigerette per day women smoke more than double the amount that men does.       - In order to rule out confounding effects as a result of poor health that is more likely at older age, we only use data of 30-40 years individuals.<jupyter_code>age=pd.cut(heart_diseases_female['age'],[30,40])
heart_diseases_female.pivot_table("cigsPerDay",["male/female",age],"Chance of getting sick")
age2=pd.cut(heart_diseases_male['age'],[30,40])
heart_diseases_male.pivot_table("cigsPerDay",["male/female",age2],"Chance of getting sick")<jupyter_output><empty_output><jupyter_text>- For women to be classify as high risk for heart diseases they need to smoke double the amount that men do. This raise the question of the importance of gender effect on chances to suffer from heart diseases.  Who has a higher risk of developing heart disease in the next ten years, men or women?<jupyter_code>fig,axes = plt.subplots(figsize=(15,5),ncols=2)

heart_diseases_male['Chance of getting sick'].value_counts().plot.pie(ax=axes[0],autopct="%1.1f%%", title = "TenYearCHD- Chance of getting sick-Male." )
heart_diseases_female['Chance of getting sick'].value_counts().plot.pie(ax=axes[1],autopct="%1.1f%%", title ="TenYearCHD- Chance of getting sick-Female")
plt.show()<jupyter_output><empty_output><jupyter_text>Third conclusion.- It can be seen that men have a higher risk of developing heart disease in the next ten years yet further research is needed to examine this issue.  We want to see the distribution of cholesterol and glucose values among men and women, in order to better learn about the population in the data..<jupyter_code>fig,axes= plt.subplots(figsize=(20,5),ncols=2)
sns.boxplot(ax=axes[0],y='glucose',data=heart_diseases_male,color='red')
axes[0].set_title('Glucose-male',fontsize=16)
sns.boxplot(ax=axes[1],y='glucose',data=heart_diseases_female,color='red')
axes[1].set_title('Glucose-female',fontsize=16)
plt.show()
fig,axes= plt.subplots(figsize=(20,5),ncols=2)
sns.boxplot(ax=axes[0],y='totChol',data=heart_diseases_male,color='yellow')
axes[0].set_title('Cholesterol-male',fontsize=16)
axes[0].set(ylim=(100,600))
sns.boxplot(ax=axes[1],y='totChol',data=heart_diseases_female,color='yellow')
axes[1].set_title('Cholesterol-female',fontsize=16)
plt.show()<jupyter_output><empty_output><jupyter_text>- We can see that the distribution of the tested values is a bit unusual since the average values of Glucose is normal, however there are a lot of outliers.- Although the presented Cholesterol levels seems slightly high, since the subjects are American citizens who suffer from 6% higher Cholesterol than Western countries citizens, these values are normal. Fourth conclusion.- This finding leads us to another conclusion which can be taken to other disease related research- most of the population is healthy. Although values such as Cholesterol and Glucose  can be abnormal it does not necessarily mean individual are sick since there can still be many other variables that affect the chances of getting sick.  This conclusion makes it very difficult to predict who would be sick and who would not be sick.  Because of the previous conclusion we raised:- Using several models we first tried to predict who would be sick and who would not be sick. We find that the accuracy of the models is high, about 90% prevision overall. However no model was able to predict an individuals who has a risk of suffering a heart disease.- We then realized that no model can predict patients because there are almost no patients (something we mentioned earlier) thus the model will constantly predict that the patient are healthy, therefore the model is accurate 90% of the times.- We therefore cut from our data 82% of healthy the patients in order to create a 50/50 sensible ratio of patients vs healthy.   This is the ratio of patients vs healthy before we cut the data.<jupyter_code>pd.DataFrame(heart_diseases['TenYearCHD'].value_counts())<jupyter_output><empty_output><jupyter_text>Here we cut 82% of the healthy patients and redefine our data.<jupyter_code>drop_idx = heart_diseases[heart_diseases['TenYearCHD']==0].sample(frac=0.82,random_state=6).index
heart_diseases_new=heart_diseases.drop(drop_idx).copy()<jupyter_output><empty_output><jupyter_text>This is the ratio of patients vs healthy after we cut the data.<jupyter_code>pd.DataFrame(heart_diseases_new['TenYearCHD'].value_counts())<jupyter_output><empty_output><jupyter_text>Define the features and target of the model. Split the data into 30% test and 70% training.<jupyter_code>features = ['male','age', 'totChol', 'BPMeds','glucose','cigsPerDay','diabetes','sysBP','diaBP']
X = heart_diseases_new[features]
y = heart_diseases_new['TenYearCHD']
X_train, X_test, y_train, y_test = sk.model_selection.train_test_split(X, y, test_size=0.3, random_state=23)<jupyter_output><empty_output><jupyter_text>- Choose a random forest model.- Fit the model.- Make prediction.- Check the accuracy of the model.<jupyter_code>from sklearn.ensemble import RandomForestClassifier

random_f = RandomForestClassifier(max_depth=10,n_estimators=1000,min_samples_leaf=20,random_state=23, criterion='gini')
random_f.fit(X_train, y_train)
predrf = random_f.predict(X_test)
acc = sk.metrics.accuracy_score(predrf, y_test)
print("The accuracy of our model:",acc,"!")<jupyter_output>The accuracy of our model: 0.6851851851851852 !<jupyter_text>- Check by matrix In some cases we were right and in some cases we were wrong.<jupyter_code>sk.metrics.plot_confusion_matrix(random_f, X_test, y_test, cmap=plt.cm.Blues)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10911716171617161, 'fraction_numerical': 0.024236798679867985, 'mean_word_length': 7.513608428446005, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 68, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '564202', 'license': 'CC0-1.0', 'n_tokens_mistral': 3358, 'n_tokens_neox': 3086, 'n_words': 904, 'path': 'projects-spring-2021/N.ipynb', 'repo_name': 'YuLe33/data-analytics', 'size': '9696'}