<jupyter_start><jupyter_text>Exploratory Data Analysis Assignment - Haberman DatasetSource: Kaggle From Haberman Dataset, we have data collected over the span of 12 years between 1958 to 1970.There are three features which includes age of the patient, year of operation, number of positive axillary nodes.There is one predictor value which is Survival status (Survival status 1 = patient survived more than 5 years/longer ,2 = patient dies with in 5 years) Objective:To predict whether the patient will survive after 5 years or not based upon the age,year of opertaion,axillary nodes.<jupyter_code>#importing librararies 

import warnings
warnings.filterwarnings("ignore")

import pandas as pd              # Pandas extensively used for loading datasets  
                                 #  and creating data frames from dataset
import seaborn as sns            # Data visualization library based upon matplotlib
import matplotlib.pyplot as plt  # Python 2D plotting library
import numpy as np               # NumPy is the fundamental package for scientific 
                                 # computing with Python 

#Loading the dataset using pandas library 
Haberman_df = pd.read_csv('haberman.csv',header=None, names=['Age','year_of_operation',
                            'positive_lymph_nodes','survival_status_after_5_years']) 
Haberman_df.head()             # prints heads nodes of DataFrame

Haberman_df['survival_status_after_5_years'] = \
Haberman_df['survival_status_after_5_years'].map({1:"yes", 2:"no"})
Haberman_df['survival_status_after_5_years'] = \
Haberman_df['survival_status_after_5_years'].astype('category')
Haberman_df.head()<jupyter_output><empty_output><jupyter_text>- Survival status denoted as numerical values in the dataset, for better understanding changed from numeric values to string category type.<jupyter_code>Haberman_df.shape  #Prints dimension of DataFrame in terms of rows and columns
Haberman_df.columns  # prints columns of DataFrame
Haberman_df.info()
Haberman_df.describe()<jupyter_output><empty_output><jupyter_text>Observations:- From above data, average (mean) survival rate of patients age is around 52(50% percentile).- Patients are between min age of 30 and max age of 83.- Max lymph nodes are 52 and minimum 0<jupyter_code>Haberman_df["survival_status_after_5_years"].value_counts()<jupyter_output><empty_output><jupyter_text>- From the above Data looks like imbalanced, patients who survived more than 5 years are 225.- patients who die with in 5 years are 81.  Univariate Analysis - Probability Density Function<jupyter_code>sns.FacetGrid(Haberman_df,hue='survival_status_after_5_years',size = 5)\
   .map(sns.distplot,'Age')\
   .add_legend()
plt.ylabel('Probability Density')
plt.show()
sns.FacetGrid(Haberman_df,hue='survival_status_after_5_years',size = 5)\
   .map(sns.distplot,'year_of_operation')\
   .add_legend()
plt.show()
sns.FacetGrid(Haberman_df,hue='survival_status_after_5_years',size = 5)\
   .map(sns.distplot,'positive_lymph_nodes')\
   .add_legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Observations:- The above PDF's Describes probability density of survival status of patients based upon age, year of operation,   positive_axillary_nodes.- features are overlapped massively.- from PDF of positive_lymph_nodes, more than 50% of patients survived who had lymph_nodes between 0 to 5 and around   11-12% of patients who had more than 5 nodes.  Cumulative distribution function<jupyter_code>counts, bin_edges = np.histogram(Haberman_df["Age"],bins = 10,density=True)

pdf = counts/(sum(counts))
print(pdf)
print(bin_edges)
cdf = np.cumsum(pdf)
plt.plot(bin_edges[1:],pdf)
plt.plot(bin_edges[1:],cdf)

plt.title("PDF and CDF of Age")
plt.xlabel("Age")
plt.ylabel("")
plt.show()
counts, bin_edges = np.histogram(Haberman_df["year_of_operation"],
                                 bins=10,density=True)

print(counts)
print(bin_edges)
pdf = counts/sum(counts)
print(pdf)
cdf = np.cumsum(pdf)
print(cdf)
plt.plot(bin_edges[1:],pdf)
plt.plot(bin_edges[1:],cdf)
plt.xlabel('year_of_operation')
plt.title("PDF and CDF of year_of_operation")
plt.show()
counts, bin_edges = np.histogram(Haberman_df["positive_lymph_nodes"],
                                 bins=10,density=True)
print(counts)
print(bin_edges)
pdf = counts/sum(counts)
print(pdf)
cdf = np.cumsum(pdf)
print(cdf)

plt.plot(bin_edges[1:],pdf)
plt.plot(bin_edges[1:],cdf)

plt.show()<jupyter_output>[0.14831574 0.0188537  0.01131222 0.00502765 0.00565611 0.00125691
 0.00062846 0.         0.00062846 0.00062846]
[ 0.   5.2 10.4 15.6 20.8 26.  31.2 36.4 41.6 46.8 52. ]
[0.77124183 0.09803922 0.05882353 0.02614379 0.02941176 0.00653595
 0.00326797 0.         0.00326797 0.00326797]
[0.77124183 0.86928105 0.92810458 0.95424837 0.98366013 0.99019608
 0.99346405 0.99346405 0.99673203 1.        ]<jupyter_text>Box Plots and Whiskers:<jupyter_code># Used to get 25th,50th,75th percentile value
sns.boxplot(x='survival_status_after_5_years',y='Age',data=Haberman_df)
plt.show()
sns.boxplot(x='survival_status_after_5_years',y='year_of_operation',
            data=Haberman_df)
plt.show()
sns.boxplot(x='survival_status_after_5_years',y='positive_lymph_nodes',
            data=Haberman_df)
plt.show()<jupyter_output><empty_output><jupyter_text>Observations:- From the above positive_lymph_nodes Box plot ,we can see the patients survived who had lymph_nodes between 0 to 5.- features are overlapped massively.  Violin Plots-  Violin plots are combination of box plots and Probability density function<jupyter_code>sns.violinplot(x='survival_status_after_5_years',y='Age',
               data=Haberman_df,size=8)
plt.show()
sns.violinplot(x='survival_status_after_5_years',y='year_of_operation',
               data=Haberman_df,size=8)
plt.show()
sns.violinplot(x='survival_status_after_5_years',y='positive_lymph_nodes',
               data=Haberman_df,size=8)
plt.show()<jupyter_output><empty_output><jupyter_text>Observations:- From the above positive_lymph_nodes Violin plot ,we can see the patients survived who had lymph_nodes between 0 to 5.  Multi-Variate Analysis - 2D Scatter Plots<jupyter_code>sns.set_style("whitegrid")
sns.FacetGrid(Haberman_df, hue="survival_status_after_5_years", size=5)\
   .map(plt.scatter, "Age", "year_of_operation")\
   .add_legend()
plt.show()
sns.set_style("whitegrid")
sns.FacetGrid(Haberman_df, hue="survival_status_after_5_years", size=5)\
   .map(plt.scatter, "Age", "positive_lymph_nodes")\
   .add_legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Multi-Variate Analysis - Pair Plots<jupyter_code>sns.set_style("whitegrid")
sns.pairplot(Haberman_df,hue='survival_status_after_5_years',size=5)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1049502006838115, 'fraction_numerical': 0.05470492046974877, 'mean_word_length': 5.116363636363636, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '604449', 'license': 'MIT', 'n_tokens_mistral': 2672, 'n_tokens_neox': 2283, 'n_words': 531, 'path': 'Exploratory_Data_Analysis_Assignment/Exploratory_Data_Analysis_Assignment.ipynb', 'repo_name': 'shivakv9/ML-and-AI', 'size': '6727'}