<jupyter_start><jupyter_text>Data Science Unit 1 Sprint Challenge 1 Loading, cleaning, visualizing, and analyzing dataIn this sprint challenge you will look at a dataset of the survival of patients who underwent surgery for breast cancer.http://archive.ics.uci.edu/ml/datasets/Haberman%27s+SurvivalData Set Information:The dataset contains cases from a study that was conducted between 1958 and 1970 at the University of Chicago's Billings Hospital on the survival of patients who had undergone surgery for breast cancer.Attribute Information:1. Age of patient at time of operation (numerical)2. Patient's year of operation (year - 1900, numerical)3. Number of positive axillary nodes detected (numerical)4. Survival status (class attribute)-- 1 = the patient survived 5 years or longer-- 2 = the patient died within 5 yearSprint challenges are evaluated based on satisfactory completion of each part. It is suggested you work through it in order, getting each aspect reasonably working, before trying to deeply explore, iterate, or refine any given step. Once you get to the end, if you want to go back and improve things, go for it!  Part 1 - Load and validate the data- Load the data as a `pandas` data frame.- Validate that it has the appropriate number of observations (you can check the raw file, and also read the dataset description from UCI).- Validate that you have no missing values.- Add informative names to the features.- The survival variable is encoded as 1 for surviving >5 years and 2 for not - change this to be 0 for not surviving and 1 for surviving >5 years (0/1 is a more traditional encoding of binary variables)At the end, print the first five rows of the dataset to demonstrate the above.<jupyter_code># TODO

import pandas as pd

# Load the data into a pandas dataframe
# Add informative names to the features

column_headers = ['age_at_operation', 'year_of_operation', 'positive_aux_nodes', 'survival_status']
df = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data', header=None, names=column_headers)
df.head()

# 1. Age of patient at time of operation (numerical) 
# 2. Patient's year of operation (year - 1900, numerical) 
# 3. Number of positive axillary nodes detected (numerical) 
# 4. Survival status (class attribute) 
# -- 1 = the patient survived 5 years or longer 
# -- 2 = the patient died within 5 year
# Validate that it has the appropriate number of observations (you can check the raw file, and also read the dataset description from UCI).

# According to UCI, there should be 306 observations and 3 numerical attributes
# and one "class attribute"

df.shape
# Validate that you have no missing values.

# There are no missing values listed below, and this is confirmed by the 
# abstract on UCI.

df.isnull().sum()
# The survival variable is encoded as 1 for surviving >5 years and 2 for not - 
# change this to be 0 for not surviving and 1 for surviving >5 years (0/1 is a 
# more traditional encoding of binary variables)

d = {2:0}
df.survival_status = df.survival_status.replace(d)
df.survival_status.head(10)
# At the end, print the first five rows of the dataset to demonstrate the above.
df.head()
df.survival_status.value_counts()<jupyter_output><empty_output><jupyter_text>Part 2 - Examine the distribution and relationships of the featuresExplore the data - create at least *2* tables (can be summary statistics or crosstabulations) and *2* plots illustrating the nature of the data.This is open-ended, so to remind - first *complete* this task as a baseline, then go on to the remaining sections, and *then* as time allows revisit and explore further.Hint - you may need to bin some variables depending on your chosen tables/plots.<jupyter_code># TODO

#create at least 2 tables

# Summary statistics

df.describe()
time_bins = pd.cut(df['age_at_operation'], 5)
ct = pd.crosstab(df['survival_status'], df['binned_age'])
ct
# Histogram of positive auxiliary nodes

import matplotlib.pyplot as plt
import seaborn as sns

sns.set()

fig, ax = plt.subplots(figsize=(16,8))
sns.distplot(df['positive_aux_nodes'], color=sns.xkcd_rgb["cherry red"], bins=15);
fig, ax = plt.subplots(figsize=(16,8))
sns.distplot(df['age_at_operation'], color=sns.xkcd_rgb["bright sky blue"], bins=15);
fig, ax = plt.subplots(figsize=(16,8))
sns.distplot(df['year_of_operation'], color=sns.xkcd_rgb["lime green"], bins=10);
# Pair plot

sns.pairplot(df, kind='reg')
#!pip install pandas==0.23.4
time_bins = pd.cut(df['age_at_operation'], 8)  # 5 equal-sized bins
pd.crosstab(df['survival_status'], time_bins, normalize='columns')
time_bins.value_counts()
time_bins = pd.cut(df['year_of_operation'], 5)  # 5 equal-sized bins
pd.crosstab(df['survival_status'], time_bins, normalize='columns')
time_bins.value_counts()

bins = [0,1,2,3,4,5,6,8,10,41.6]
time_bins = pd.cut(df['positive_aux_nodes'], bins)  # 5 equal-sized bins
pd.crosstab(df['survival_status'], time_bins, normalize='columns')
time_bins.value_counts()
plt.scatter(df.survival_status, df.positive_aux_nodes)
plt.title('Scatter plot pythonspot.com')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
g = sns.lmplot(x="year_of_operation", y="survival_status",
               truncate=True, data=df, height=8, aspect=2)

# Use more informative axis labels than are provided by default
g.set_axis_labels("Patient's year of operation", "Survival status");
g = sns.lmplot(x="age_at_operation", y="survival_status",
               truncate=True, data=df, height=8, aspect=2)

# Use more informative axis labels than are provided by default
g.set_axis_labels("Age of patient at time of operation", "Survival status");
g = sns.lmplot(x="positive_aux_nodes", y="survival_status", x_bins=[0,1,2,3,4,5,10,45],
               truncate=True, data=df, height=8, aspect=2)

# Use more informative axis labels than are provided by default
g.set_axis_labels("Number of positive axillary nodes detected", "Survival status");
g = sns.lmplot(x="age_at_operation", y="positive_aux_nodes", hue="survival_status",
               truncate=True, data=df, height=8, aspect=2)

# Use more informative axis labels than are provided by default
g.set_axis_labels("Age of patient at time of operation", "Number of positive axillary nodes detected");
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
for idx, feature in enumerate(list(df.columns)[:-2]):
    sns.boxplot( x='survival_status', y=feature, data=df, ax=axes[idx])
plt.show()<jupyter_output><empty_output><jupyter_text>Part 3 - Analysis and InterpretationNow that you've looked at the data, answer the following questions:- What is at least one feature that looks to have a positive relationship with survival?There doesn't appear to be any features that have a positive relationship with survival rates. It's possible that there may have been a slight bump in survival rates starting in 1966, but there's not enough data to confidently make that assertion.- What is at least one feature that looks to have a negative relationship with survival?The number of positive auxiliary nodes detected seems to have a strongly negative relationship with survival status (i.e., the more nodes detected, the less likely the patient is to survive). The age of the patient at the time of the operation also seems to have a neagtive relationship with survival.- How are those two features related with each other, and what might that mean?Age at operation doesn't appear to have a strong relationship with the number of positive auxiliary nodes detected. The number of positive auxiliary nodes detected seemed to have the strongest relationship with survival rates of five years or longer. The vast majority of values for this feature were clustered between zero and five, and you can observe a marked decline in survival rates as you progress through those valuesAnswer with text, but feel free to intersperse example code/results or refer to it from earlier.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0868470736312146, 'fraction_numerical': 0.016614222781623663, 'mean_word_length': 5.15015479876161, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '614429', 'license': 'MIT', 'n_tokens_mistral': 2411, 'n_tokens_neox': 2218, 'n_words': 985, 'path': 'Stephen_Sinclair_DS_Unit_1_Sprint_Challenge_1.ipynb', 'repo_name': 'SMSinclair/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '7945'}