<jupyter_start><jupyter_text>Titanic Data AnalysisBy <NAME>  IntroductionThe purpose of this project is to build an analysis from the Titanic Survivors data set provided by Kaggle using Pandas and Numpy dependencies within Python. [(Kaggle: Titanic)](https://www.kaggle.com/c/titanic) This data set contains demographics and passenger information from 891 of the 2224 passengers and crew on board the Titanic. In this data investigation I will focus in analysing the factors that made it more likely for a person to survive on the Titanic. Our established hypothesis is that higher class passengers, females, and younger people had a higher chance of surviving.   Data InvestigationI will begin by browsing the data structure and understanding what variables are included in the data, particularly the meaning and type of each variable and if necessary convert variables into format that is fit for data analysis.Once I am able to understand variables, I will explore and understand my dataset by answering useful questions using panda and matplotlib to bring together useful insights into graphs. I will also use descriptive statistics to define correlation between survivors and nonsurvivors. The variables that showed evident difference between two groups are likely to have had an impact on chance of survival.  Loading DataTo begin I will import pandas for data structure analisis and matplotlib for ploting graphs using python and load data from csv file.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pylab as plt
%matplotlib inline
data = pd.read_csv("titanic_data.csv")<jupyter_output><empty_output><jupyter_text>*What is the size of my data set?* From my evaluation I can confirm that the size of our data set has 891 rows and 12 columns by calling the following:<jupyter_code>#Find length of data
#len(data)
# Look at the shape of the dataFrame object, and it has 891 rows and 12 columns.
data.shape<jupyter_output><empty_output><jupyter_text>*What variables are within data structure?*<jupyter_code># Build data structure sample. 
data.head()<jupyter_output><empty_output><jupyter_text>Below I have attached description to each variable. | Variable Name |  Variable Description | |---------------|-----------------------||PassengerID| Passenger ID||   Survived  |  Survival (0 = No;1 = Yes)||Pclass|Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)||Name|Name||Sex|Sex||Age|Age||SibSp| Number of Siblings/Spouses Aboard||Parch|Number of Parents/Children Aboard||Ticket|Ticket Number||Fare|Passenger Fare||Cabin|Cabin||Embarked|Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)| Now that I am able to understand our variables, I could dive into evaluating if we have any missing fields within our data set.<jupyter_code>data.count()
# Check whether the columns to be analyzed having any missing values, how many missing values each column has.
def null_count(data):
    data_null = pd.isnull(data)
    data_null_true = data[data_null == True]
    null_count = len(data_null_true)
    return null_count
data.apply(null_count) # Let's check how many missing values each column has.<jupyter_output><empty_output><jupyter_text>*Discussion: *I have establish that there are missing fields within our data (Age, Cabin, and Embarked). Missing a lot of Age data does affect our age evaluation, however, a possible solution would be to replace empty fields with an estimated average.Missing the vast majority of Cabin data is be a bit problematic when evaluating how far away from possible safety this cabin was located. However, for the sake of this evaluation I will continue to analysis without omitting any data.  *Plotting Data*  *Plotting Passenger Survival Data**How many passengers survived?*<jupyter_code>data['Survived'].value_counts().plot(kind='bar')
plt.title("Survival Count")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Count")
#Find Survival count
data['Survived'].value_counts()
#Find Normalized Survival Percentage
data['Survived'].value_counts(normalize=True).plot(kind='bar')
plt.title("Survival Percentage")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Percentage")
#Find Survival Percentage
data['Survived'].value_counts()*100/len(data)<jupyter_output><empty_output><jupyter_text>Discussion: From my evaluation of this sample data set, I find that about 38% (342) survived and 62% (549 did not survive.   *Plotting Male/Female Distribution Data**How many male and female are counted within data set?**What was the survival distribution based on Sex?*<jupyter_code>#Find Sex count
data['Sex'].value_counts()
#Plot Sex Count
data['Sex'].value_counts().plot(kind='bar',
                                color=['c','m'])
plt.title("Sex Dristribution")
plt.xlabel("Sex (Male/Female)")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>Discussion: There are far more male passengers than female passengers in the data.<jupyter_code>fig = plt.figure()
fig1 = fig.add_subplot(1,2,1)
plt.xlabel("Female Age Range")
plt.ylabel("Passenger Count")
data[data['Sex']=="female"]['Age'].hist(bins=15)
fig2 = fig.add_subplot(1,2,2)
data[data['Sex']=="male"]['Age'].hist(bins=15)
plt.xlabel("Male Age Range")
plt.ylabel("Passenger Count")
# Male Survival Distribution
data[data['Sex'] == 'male']['Survived'].value_counts()
data[data['Sex'] == 'male']['Survived'].value_counts()*100/len(data)
data[data['Sex'] == 'male']['Survived'].value_counts().sort_index().plot(kind='bar', color=['c'])
plt.title("Male Survival Distribution")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Percentage")
# Female Survival Distribution
data[data['Sex'] == 'female']['Survived'].value_counts()
data[data['Sex'] == 'female']['Survived'].value_counts()*100/len(data)
data[data['Sex'] == 'female']['Survived'].value_counts().sort_index().plot(kind='bar', color=['m'])
plt.title("Female Survival Distribution")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Count")
data.groupby('Sex')['Survived'].mean() # Derive Positive Survival between female/male.<jupyter_output><empty_output><jupyter_text>*Discussion:*Proportionately more males died relative to females. Only 19% where male passengers and 74% of the 342 survivors where female passengers. Historically, "women and children first" is a phrase that was first coined on the British HMS Birkenhead, which sunk in 1852. Titanic, which also departed from Britain, likely encountered the same informal policy, handed down culturally through the British seas.  *Plotting Passenger Age Distribution **What are the lowest and largest Age values?**What was the survival distribution based on Age?*<jupyter_code>#Find lowest and Largest Age value 
data['Age'].min(), data['Age'].max()
# Take a closer look at the distribution of the age of passengers. It has a wide range of age span.
data['Age'].describe()<jupyter_output><empty_output><jupyter_text>*Discussion:*From my evaluation I find that the youngest was probably a baby just a few months old with a value converted to decimal. The age distributions for male and female passengers on board are similar. Most passengers are adults below 40s and there are also a few children and senior citizens on abroad.<jupyter_code># Define an age group for different ranges of age.
def age_group(age):
    if age < 18:
        return "young" 
    if 18 <= age < 60:
        return "adult"
    if age >= 60:
        return "senior"  
data["age_group"] = data["Age"].apply(age_group)
data.groupby(["age_group"])['Survived'].mean()<jupyter_output><empty_output><jupyter_text>*Discussion:* 53% of the 342 survivors seem to have been young (age<18) passengers.*<jupyter_code>#Positive Survival/Age Distribution
#data[data['Survived'] == 1]['Age'].value_counts()
#Negative Survival/Age Distribution
#data[data['Survived'] == 0]['Age'].value_counts()
data.plot(kind='scatter', x='Survived', y='Age', alpha=.5)
plt.title("Survival Age Distribution Scattered Box Plot")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Percentage")
data.boxplot(column=["Age"],by=["Survived"])
plt.title("Survival Age Distribution")
plt.xlabel("Yes/No (1/0)")
plt.ylabel("Percentage")
data[data['Survived'] == 1]['Age'].value_counts().sort_index().plot(kind='bar')
plt.title("Positive Survival/Age Distribution")
plt.xlabel("Age")
plt.ylabel("Count")
bins = [0, 10, 20, 30, 40, 50, 60, 70, 80]
data['AgeBin'] = pd.cut(data['Age'], bins)<jupyter_output><empty_output><jupyter_text>*Discussion:* Based on previous graph it is hard to find correlation related to survival based on age. Since there is very large range of ages from our data set I would have to organize age ranges in bins to better understand visualization.<jupyter_code>#Positive Survival/Age Distribution
data[data['Survived'] == 1]['AgeBin'].value_counts()
data[data['Survived'] == 1]['AgeBin'].value_counts().sort_index().plot(kind='bar')
plt.title("Positive Survival/Age Distribution")
plt.xlabel("AgeBin")
plt.ylabel("Count")
#Negative Survival/Age Distibution
data[data['Survived'] == 0]['AgeBin'].value_counts()
data[data['Survived'] == 0]['AgeBin'].value_counts().sort_index().plot(kind='bar')
plt.title("Negative Survival/Age Distribution")
plt.xlabel("AgeBin")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>*Discussion:*When comparing survivors and deaths based on age, the mean age for survivors was 28.164948 years and for deaths 29.705314 years. It's clear that there were a lot more younger survivors. Although I am able to visualize well the age distribution between survivors and nonsurvivors passengers it is still hard to correlate any relationship. It is possible that missing 177 Age fields has affects our overall analysis. Best if we had more data.  Plotting Class Distribution*What are the general demographics and passengers distibution by class (gender,age etc)?**How where passengers distributed between class within data set?**What was the survival distribution based on Class?*<jupyter_code>#Passenger Count by Class Distribution
data['Pclass'].value_counts()
data['Pclass'].value_counts().sort_index().plot(kind='bar')
plt.title("Passenger Class Distribution")
plt.xlabel("Class")
plt.ylabel("Count")
# Average age by class
data.groupby('Pclass')['Age'].mean() 
data.groupby('Pclass')[['Age']].describe()
plt.subplot2grid
for x in [1,2,3]:
        data.Age[data.Pclass == x].plot(kind="kde")
plt.title("Class/Age Distribution")
plt.xlabel("Age")
plt.ylabel("Density")
plt.legend(("1st", "2nd", "3rd")) 
data.boxplot(column=["Age"],by=["Pclass"])
plt.title("Age by Cabin Class Boxplot")
plt.xlabel("Cabin Class")
plt.ylabel("Age Range")<jupyter_output><empty_output><jupyter_text>*DIscussion*Based on my initial findings, on average first class passengers are older than the second class, and the ones in second class are older than the those in the third class.<jupyter_code>data.groupby('Pclass')[['Survived']].describe()
data.groupby('Pclass')[['Sex']].describe()
data.groupby('Pclass')[['Age']].describe()
data.groupby('Pclass')[['Parch']].describe()
data.groupby('Pclass')[['SibSp']].describe()
data.groupby('Pclass')['Survived'].sum() 
data.groupby('Pclass')['Survived'].mean()<jupyter_output><empty_output><jupyter_text>*Discussion:*It seems that first class passengers had a much higher chance of survival than the second class, also the second class has a higher rate of survival than the third class.<jupyter_code>#Positive Class Survival Count
data[data['Survived'] == 1]['Pclass'].value_counts().sort_index()
#Positive Class Survival Percentage
data[data['Survived'] == 1]['Pclass'].value_counts()*100/len(data)
data[data['Survived'] == 1]['Pclass'].value_counts().sort_index().plot(kind='bar')
plt.title("Positive Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")
#Negative Class Survival Count
data[data['Survived'] == 0]['Pclass'].value_counts().sort_index()
#Negative Class Survival Percentage
data[data['Survived'] == 0]['Pclass'].value_counts()*100/len(data)
data[data['Survived'] == 0]['Pclass'].value_counts().sort_index().plot(kind='bar')
plt.title("Negative Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")
# Male Positive Survical Class Count
data[(data['Sex'] == 'male') & (data['Survived'] == 1)]['Pclass'].value_counts().sort_index()
data[(data['Sex'] == 'male') & (data['Survived'] == 1)]['Pclass'].value_counts().sort_index().plot(kind='bar', color=['c'])
plt.title("Male Positive Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")
#Male Negative Survival Class Count
data[(data['Sex'] == 'male') & (data['Survived'] == 0)]['Pclass'].value_counts().sort_index()
data[(data['Sex'] == 'male') & (data['Survived'] == 0)]['Pclass'].value_counts().sort_index().plot(kind='bar', color=['c'])
plt.title("Male Negative Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")
# Female Positive Survival Class Count
data[(data['Sex'] == 'female') & (data['Survived'] == 1)]['Pclass'].value_counts().sort_index()
data[(data['Sex'] == 'female') & (data['Survived'] == 1)]['Pclass'].value_counts().sort_index().plot(kind='bar', color=['m'])
plt.title("Female Positive Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")
#Female Negative Survival Class Count
data[(data['Sex'] == 'female') & (data['Survived'] == 0)]['Pclass'].value_counts().sort_index()
data[(data['Sex'] == 'female') & (data['Survived'] == 0)]['Pclass'].value_counts().sort_index().plot(kind='bar', color=['m'])
plt.title("Female Negative Survival Class Count")
plt.xlabel("Class")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>*Discussion:*The results indicate that 41.7% of the passengers on the ship were third class and died. From the first class passengers, 63% survived. 47% of the second class passengers survived. A mere 24% of the third class passengers survived. Based on initial glance, it appears that there is a correlation between survival rate and class.  Plotting Embark Data*What was passenger embark distribution?<jupyter_code>#Embarked Distribution
data['Embarked'].value_counts()
#Embarked Count Distribution
data['Embarked'].value_counts().sort_index().plot(kind='bar')
plt.title("Embarked Distribution")
plt.xlabel("Embarked")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>Plotting Fare/Age Data<jupyter_code>#Fare Age Distribution
data.plot(kind='scatter', x='Fare', y='Age', alpha=.5)
plt.title("Fare/Age Distibution Scattered Box Plot")
plt.xlabel("Fare")
plt.ylabel("Age")
#Fare Age Distribution
data.plot(kind='scatter', x='Survived', y='Fare', alpha=.5)
plt.title("Fare/Survived Distibution Scattered Box Plot")
plt.xlabel("Fare")
plt.ylabel("Age")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1423957116349392, 'fraction_numerical': 0.01223283623118686, 'mean_word_length': 6.675105485232067, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 67, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '103414', 'license': 'MIT', 'n_tokens_mistral': 4816, 'n_tokens_neox': 4330, 'n_words': 1377, 'path': 'P2_DAND/Titanic Data Analysis.ipynb', 'repo_name': 'sanchezm1288/P1_DANDa', 'size': '14569'}