<jupyter_start><jupyter_text>Predicting heart disease using machine learningThis notebook looks into using various Python-based machine learning and data science libraries in an attempt to build a machine learning model capable of predicting whether or not someone has heart disease based on their medical attributes.The approach we are following:1. Problem Definition2. Data3. Evaluation4. Feature5. Modelling6. Experimentation  1. Problem DefinitionIn a statement,> Given clinical parameters about a patient, can we predict whether or not they have heart disease?  2. Data ContextThis database contains 76 attributes, but all published experiments refer to using a subset of 14 of them. In particular, the Cleveland database is the only one that has been used by ML researchers tothis date. The "goal" field refers to the presence of heart disease in the patient. It is integer valued from 0 (no presence) to 4. ContentAttribute Information:1. age2. sex3. chest pain type (4 values)4. resting blood pressure5. serum cholestoral in mg/dl6. fasting blood sugar > 120 mg/dl7. resting electrocardiographic results (values 0,1,2)8. maximum heart rate achieved9. exercise induced angina10. oldpeak = ST depression induced by exercise relative to rest11. the slope of the peak exercise ST segment12. number of major vessels (0-3) colored by flourosopy13. thal: 3 = normal; 6 = fixed defect; 7 = reversable defectThe names and social security numbers of the patients were recently removed from the database, replaced with dummy values. One file has been "processed", that one containing the Cleveland database. All four unprocessed files also exist in this directory.To see Test Costs (donated by <NAME>), please see the folder "Costs" AcknowledgementsCreators:Hungarian Institute of Cardiology. Budapest: <NAME>, M.D.University Hospital, Zurich, Switzerland: <NAME>, M.D.University Hospital, Basel, Switzerland: <NAME>, M.D.V.A. Medical Center, Long Beach and Cleveland Clinic Foundation: <NAME>., Ph.D.Donor:<NAME> (aha '@' ics.uci.edu) (714) 856-8779 InspirationExperiments with the Cleveland database have concentrated on simply attempting to distinguish presence (values 1,2,3,4) from absence (value 0).See if you can find any other trends in heart data to predict certain cardiovascular events or find any clear indications of heart health.  3. Evaluation> If we can reach 95% accuracy at predicting whether or not a patient has heart disease during the proof of concept, we'll pursue the project,  4. Features**Create data dictionary**1. age - age in years2. sex - (1 = male; 0 = female)3. cp - chest pain type    * 0: Typical angina: chest pain related decrease blood supply to the heart    * 1: Atypical angina: chest pain not related to heart    * 2: Non-anginal pain: typically esophageal spasms (non heart related)    * 3: Asymptomatic: chest pain not showing signs of disease4. trestbps - resting blood pressure (in mm Hg on admission to the hospital) anything above 130-140 is typically cause for concern5. chol - serum cholestoral in mg/dl    * serum = LDL + HDL + .2 * triglycerides    * above 200 is cause for concern6. fbs - (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)    * '>126' mg/dL signals diabetes7. restecg - resting electrocardiographic results    * 0: Nothing to note    * 1: ST-T Wave abnormality        * can range from mild symptoms to severe problems        * signals non-normal heart beat    * 2: Possible or definite left ventricular hypertrophy        * Enlarged heart's main pumping chamber8. thalach - maximum heart rate achieved9. exang - exercise induced angina (1 = yes; 0 = no)10. oldpeak - ST depression induced by exercise relative to rest looks at stress of heart during excercise unhealthy heart will stress more11. slope - the slope of the peak exercise ST segment    * 0: Upsloping: better heart rate with excercise (uncommon)    * 1: Flatsloping: minimal change (typical healthy heart)    * 2: Downslopins: signs of unhealthy heart12. ca - number of major vessels (0-3) colored by flourosopy    * colored vessel means the doctor can see the blood passing through    * the more blood movement the better (no clots)13. thal - thalium stress result    * 1,3: normal    * 6: fixed defect: used to be defect but ok now    * 7: reversable defect: no proper blood movement when excercising14. target - have disease or not (1=yes, 0=no) (= the predicted attribute)  4.1 Preparing the toolsWe're going to use Pandas, Matplotlib and Numpy for data analysis and manipulation<jupyter_code># Import all the tools we need

# Regular EDA (exploratory data analysis) and plotting libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

# Models from Scikit-Learn
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier

# Model Evaluation
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import plot_roc_curve<jupyter_output><empty_output><jupyter_text>4.2 Load data<jupyter_code>df = pd.read_csv("heart-disease.csv")
df<jupyter_output><empty_output><jupyter_text>4.3 Exploratory Data Analysis (EDA)The goal is to find out more about the data and become a subject matter expert on the data set we are working with.1. What question we are trying to solve?2. What kind of data do we have and how do we treat different types?3. What's missing from the data and how should we deal with it?4. Where are the outliers and why should we care about them?5. How can we add, change or remove features to get more out of the data?|<jupyter_code>df.head()
# How many each class there are
df['target'].value_counts()
df['target'].value_counts().plot(kind='bar', color=['salmon', 'lightblue']);
df.info()
# Are there missing values?
df.isna().sum()
df.describe()<jupyter_output><empty_output><jupyter_text>4.3.1 Heart Disease frequency according to sex<jupyter_code>df.sex.value_counts()
# Compare target column with sex column
pd.crosstab(df.target, df.sex)
# Create a plot of crosstab
pd.crosstab(df.target, df.sex).plot(kind='bar',
                                    figsize=(10, 6),
                                    color=['salmon', 'lightblue'])
plt.title("Heart Disease Frequency for Sex")
plt.xlabel('0 = No Disease, 1 = Disease')
plt.ylabel("Amount")
plt.legend(['Female', 'Male'])<jupyter_output><empty_output><jupyter_text>4.3.2 Age vs Max Heart Rate for Heart Disease<jupyter_code># Create another figure
plt.figure(figsize=(10,6))

# Start with positve examples
plt.scatter(df.age[df.target==1], 
            df.thalach[df.target==1], 
            c="salmon") # define it as a scatter figure

# Now for negative examples, we want them on the same plot, so we call plt again
plt.scatter(df.age[df.target==0], 
            df.thalach[df.target==0], 
            c="lightblue") # axis always come as (x, y)

# Add some helpful info
plt.title("Heart Disease in function of Age and Max Heart Rate")
plt.xlabel("Age")
plt.legend(["Disease", "No Disease"])
plt.ylabel("Max Heart Rate");
# Check the distribution of the ahe column with a histogram
df.age.plot.hist();<jupyter_output><empty_output><jupyter_text>4.3.3 Heart Disease Frequency per Chest Pain Type<jupyter_code># Create a new crosstab and base plot
pd.crosstab(df.cp, df.target).plot(kind="bar", 
                                   figsize=(10,6), 
                                   color=["lightblue", "salmon"])

# Add attributes to the plot to make it more readable
plt.title("Heart Disease Frequency Per Chest Pain Type")
plt.xlabel("Chest Pain Type")
plt.ylabel("Frequency")
plt.legend(["No Disease", "Disease"])
plt.xticks(rotation = 0);<jupyter_output><empty_output><jupyter_text>4.3.4 Correlation between independent variables<jupyter_code># Find the correlation between our independent variables using a correlation matrix
corr_matrix = df.corr()
corr_matrix 
corr_matrix = df.corr()
plt.figure(figsize=(15, 10))
sns.heatmap(corr_matrix, 
            annot=True, 
            linewidths=0.5, 
            fmt= ".2f", 
            cmap="YlGnBu");<jupyter_output><empty_output><jupyter_text>5. Modelling<jupyter_code># Split data into X and y
X = df.drop('target', axis=1)
y = df['target']
X
y
# Split data into train and test sets
np.random.seed(42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
X_train
y_train
X_test
y_test<jupyter_output><empty_output><jupyter_text>5.1 Model choicesNow we've got our data prepared, we can start to fit models. We'll be using the following and comparing their results.1. Logistic Regression - LogisticRegression()2. K-Nearest Neighbors - KNeighboursClassifier()3. RandomForest - RandomForestClassifier()<jupyter_code># Put models in a dictionary
models = {'LogisticRegression' : LogisticRegression(),
          'KNN': KNeighborsClassifier(),
          'RandomForest' : RandomForestClassifier()}

# Create a function to fit and score models
def fit_and_evaluate(models, X_train, X_test, y_train, y_test):
    """
    Fits and eveluate givien machine learning models.
    models : a dict of differetn Scikit-Learn machine learning models
    X_train : training data (no labels)
    X_test : testing data (no labels)
    y_train : training labels
    y_test : test labels
    """
    # Set random seed
    np.random.seed(42)
    
    # Make a dictionary to keep model scores
    model_scores = {}
    for name, model in models.items():
        # Fit the model to the data
        model.fit(X_train, y_train)
        # Evaluate the model and append its score to model_scores
        model_scores[name] = model.score(X_test, y_test)
    return model_scores
    
model_scores = fit_and_evaluate(models, X_train, X_test, y_train, y_test)
model_scores<jupyter_output>C:\Users\wangs\Documents\OnlineCourses\MachineLearningBootcamp\Heart-Disease-CLassification\env\lib\site-packages\sklearn\linear_model\_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(<jupyter_text>5.2 Model Comparison Visualization<jupyter_code>model_compare = pd.DataFrame(model_scores, index=["accuracy"])
model_compare.T.plot.bar();<jupyter_output><empty_output><jupyter_text>6. ExperimentationNow we have the baseline model but we want to improve it. Let's look at the following:* Hyperparameter tuning* Feature importance* Confusion matrix* Cross-Validation* Precision* Recall* F1 Score* Classification Report* ROC Curve* Area Under the Curve (AUC)  6.1 Hyperparameter Tuning (by hand)<jupyter_code># Let's tune KNN
train_scores = []
test_scores = []

# Create a list of different values for n_neighbors
neighbors = range(1, 21)

# Setup KNN instance
knn = KNeighborsClassifier()

# Loop through different n_neighbors
for i in neighbors:
    knn.set_params(n_neighbors=i)
    # Fit the tuned model
    knn.fit(X_train, y_train)
    train_scores.append(knn.score(X_train, y_train))
    test_scores.append(knn.score(X_test, y_test))
train_scores
test_scores
# Scores Visualization for KNN Scores
plt.plot(neighbors, train_scores, label="Train score")
plt.plot(neighbors, test_scores, label="Test score")
plt.xticks(np.arange(1, 21, 1))
plt.xlabel("Number of neighbors")
plt.ylabel("Model score")
plt.legend()

print(f"Maximum KNN score on the test data: {max(test_scores)*100:.2f}%")<jupyter_output>Maximum KNN score on the test data: 75.41%<jupyter_text>The KNN model does not perform well enough: discarded  6.2 Hyperparameter Tuning (by RandomizedSearchCV)We're going to tune:* LogisticRegression()* RandomForestClassifier()<jupyter_code># Create a hyperparameter grid for LogisticRegression
log_reg_grid = {'C' : np.logspace(-4, 4, 20),
                'solver' : ['liblinear']}
# Create a hyperparameter grid for RandomForesatClassifier
rf_grid = {"n_estimators": np.arange(10, 1000, 50),
           'max_depth': [None, 3, 5, 10], 
           'min_samples_split': np.arange(2, 20, 2),
           'min_samples_leaf' : np.arange(1, 20, 2)}

# Tuning using RandomizedSearchCV
# logisticRegression
np.random.seed(42)
rs_log_reg = RandomizedSearchCV(LogisticRegression(),
                                param_distributions=log_reg_grid,
                                cv=5,
                                n_iter=20,
                                verbose=2)
rs_log_reg.fit(X_train, y_train)

# Find best params
rs_log_reg.best_params_
# Check the tuned model score
rs_log_reg.score(X_test, y_test)
# RandomForestClassifier
np.random.seed(42)
rs_rf = RandomizedSearchCV(RandomForestClassifier(),
                           param_distributions=rf_grid,
                           cv=5,
                           n_iter=20,
                           verbose=2)
rs_rf.fit(X_train, y_train)
rs_rf.best_params_
rs_rf.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>LogisticRegression works best: RandomForest discarded  6.3 Hyperparameter Tuning (by GridSearchCV)<jupyter_code># Different hyperparameters for our LogistocRegression model
log_reg_grid = {"C": np.logspace(-4, 4, 30),
                "solver": ["liblinear"]}
gs_log_reg = GridSearchCV(LogisticRegression(),
                          param_grid=log_reg_grid,
                          cv=5,
                          verbose=2)
gs_log_reg.fit(X_train, y_train)
gs_log_reg.best_params_
gs_log_reg.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>7. Model Evaluation* ROC curve and AUC score* Confusion Matrix* Classification Report* Precision* Recall* F1 Score<jupyter_code># Make predictions with our tuned model
y_preds = gs_log_reg.predict(X_test)
y_preds
y_test<jupyter_output><empty_output><jupyter_text>7.1 ROC curve and AUC score<jupyter_code># Plot ROC curve and calculate AUC
plot_roc_curve(gs_log_reg, X_test, y_test)<jupyter_output>C:\Users\wangs\Documents\OnlineCourses\MachineLearningBootcamp\Heart-Disease-CLassification\env\lib\site-packages\sklearn\utils\deprecation.py:87: FutureWarning: Function plot_roc_curve is deprecated; Function :func:`plot_roc_curve` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: :meth:`sklearn.metric.RocCurveDisplay.from_predictions` or :meth:`sklearn.metric.RocCurveDisplay.from_estimator`.
  warnings.warn(msg, category=FutureWarning)<jupyter_text>7.2 Confusion Matrix<jupyter_code># Confusion matrix
sns.set(font_scale=1.5) # Increase font size
 
def plot_conf_mat(y_test, y_preds):
    """
    Plots a confusion matrix using Seaborn's heatmap().
    """
    fig, ax = plt.subplots(figsize=(3, 3))
    ax = sns.heatmap(confusion_matrix(y_test, y_preds),
                     annot=True, # Annotate the boxes
                     cbar=False)
    plt.xlabel("Predicted label") # predictions go on the x-axis
    plt.ylabel("True label") # true labels go on the y-axis 
    
plot_conf_mat(y_test, y_preds)<jupyter_output><empty_output><jupyter_text>7.3 Classification Report<jupyter_code>print(classification_report(y_test, y_preds))<jupyter_output>precision    recall  f1-score   support

           0       0.89      0.86      0.88        29
           1       0.88      0.91      0.89        32

    accuracy                           0.89        61
   macro avg       0.89      0.88      0.88        61
weighted avg       0.89      0.89      0.89        61<jupyter_text>7.4 Evaluation Metrics Using Cross-Validation<jupyter_code># Calculate evaluation metrics using cross-validation
def cv_eval_metrics(model, X, y):
    """
    Calculate accuracy, precision, recall and f1 using 5 fold cross-validation
    model = the model used
    X = features
    y = target
    """
    scorings = ["accuracy", "precision", "recall", "f1"]
    scores = {}
    for i in range(len(scorings)):
        scores[scorings[i]] = np.mean(cross_val_score(model, X, y, cv=5, scoring=scorings[i]))
    return scores

# Create a new classifier with best parameters
clf = LogisticRegression(C=0.20433597178569418,
                         solver="liblinear")
scores = cv_eval_metrics(clf, X, y)
scores
# Visualize cross-validated metrics
cv_metrics = pd.DataFrame({"Accuracy": scores["accuracy"],
                           "Precision": scores["precision"],
                           "Recall": scores["recall"],
                           "F1": scores["f1"]},
                          index=[0])

cv_metrics.T.plot.bar(title="Cross-validated classification metrics",
                      legend=False);<jupyter_output><empty_output><jupyter_text>8. Feature ImportanceWhich feature contributed most to the outcomes of the model and how did they contribute?<jupyter_code># Fit an instane of LogisticRegression
clf = LogisticRegression(C=0.20433597178569418,
                         solver="liblinear")
clf.fit(X_train, y_train)
# Check coef_
clf.coef_
# Match coef's of features to columns
feature_dict = dict(zip(df.columns, list(clf.coef_[0])))
feature_dict
# Visualize feature importance
feature_df = pd.DataFrame(feature_dict, index=[0])
feature_df.T.plot.bar(title="Feature Importance", legend=False);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09001191083886337, 'fraction_numerical': 0.020985763711644262, 'mean_word_length': 3.9005002779321845, 'pattern_counts': {'":': 7, '<': 79, '<?xml version=': 0, '>': 84, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '418418', 'license': 'MIT', 'n_tokens_mistral': 5730, 'n_tokens_neox': 5261, 'n_words': 1742, 'path': 'Heart-Disease-Classification.ipynb', 'repo_name': 'chaggysen/Heart-Disease-Classification', 'size': '17687'}