<jupyter_start><jupyter_text>Sample code from tutorial<jupyter_code># This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

# import numpy as np # linear algebra
# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)


# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

# import os
# for dirname, _, filenames in os.walk('/kaggle/input'):
#     for filename in filenames:
#         print(os.path.join(dirname, filename))

# Any results you write to the current directory are saved as output.

# # Let's read some data 
# train_data = pd.read_csv('/kaggle/input/titanic/train.csv')
# test_data = pd.read_csv('/kaggle/input/titanic/test.csv')
# # Let's see what is in the training sets 
# train_data.head()
# test_data.head()

# Below is the example code from the tutorial 

# # We first looked at the survival rate between female and male 
# women = train_data.loc[train_data.Sex=='female']['Survived']
# rate_women=sum(women)/len(women)
# print('% of women who survived:', rate_women)
# men = train_data.loc[train_data.Sex=='male']['Survived']
# rate_men = sum(men)/len(men)
# print('% of men who survived:', rate_men)

# # Creating the first random forest model!
# from  sklearn.ensemble import RandomForestClassifier

# y = train_data['Survived']
# features = ['Pclass', 'Sex', 'SibSp', "Parch"]
# X = pd.get_dummies(train_data[features])
# X_test = pd.get_dummies(test_data[features])

# model = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=1)
# model.fit(X,y)
# predictions = model.predict(X_test)

# output = pd.DataFrame({'PassengerID': test_data.PassengerId, 'Survived': predictions})
# output.to_csv('my_submission.csv', index=False)
# print("Your submission was susccessfully saved!")<jupyter_output><empty_output><jupyter_text>Setup<jupyter_code># For printing result inline at everywhere
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# For data frame
import pandas as pd 
# Reading the csv files 
train_data = pd.read_csv('train.csv')
test_data = pd.read_csv('test.csv')
# Let's see what is in the training sets 
train_data.head()
test_data.head()
train_data.tail()
test_data.tail()<jupyter_output><empty_output><jupyter_text>Pandas   Indexing, Slicing and Assigining  Naive Access  - Accessing object's property by accessing it as an attribute (dot notation) - Accessing object's property by indexing operator (dictionary style)- Accessing single specific value by using the list indexing<jupyter_code># What are the column indexes?
train_data.columns
train_data.index
# Let's access using dot notation 
train_data.PassengerId
train_data.Survived
# Let's access using indexing operator 
train_data['PassengerId']
train_data['Survived']
# Let's access the individual values
train_data.PassengerId[0]
train_data['PassengerId'][0]<jupyter_output><empty_output><jupyter_text>iloc vs. lociloc - for index-based selection<jupyter_code># Let's assign 0 to both iloc, which will give all the column of 0th row, hence [row,col]
test_data.iloc[0]
# The above is same as below
test_data.iloc[0,:]
# For access to all the row of a single column 
test_data.iloc[:,0]
# Let's slice some up
test_data.iloc[1:3,0]
# Let's get 0th to 2nd 
test_data.iloc[:3,0]
# Let's put some list as an indice 
test_data.iloc[[0,1,2],0]
# Let's access from the bottom 
test_data.iloc[-5:,0]<jupyter_output><empty_output><jupyter_text>loc - for label-based selection<jupyter_code># Remeber what was the column names?
test_data.columns
# Of course, we can just simply access the 0th row of all column
test_data.loc[0]
test_data.loc[0,:]
# We can index by label using loc, it is also [row, col]
test_data.loc[0,'Name']
# Let's grab multiple labels
test_data.loc[0,['Pclass','Name','Sex']]
# Heck, you can even slice the index using ":"! This will find all the index between Pclass and Sex!
test_data.loc[0,'Pclass':'Age']<jupyter_output><empty_output><jupyter_text>The problem with iloc and loc - The indexing scheme is slightly different<jupyter_code># What do we mean by that? Take a look at these
train_data.iloc[0:5,0]
train_data.loc[0:5,'PassengerId']
# Notice that iloc is 0 through 4, but loc is 0 through 5, 
# this is because loc access by string and if the ent string is exclusive it will cause confusion<jupyter_output><empty_output><jupyter_text>Manipulating Index<jupyter_code># Wanna change the index to something else?
train_data.set_index('Sex')
# Also, this won't change the original data set
train_data<jupyter_output><empty_output><jupyter_text>Conditional Selection<jupyter_code># Let's say I want to select all the female, this will return the boolean table 
train_data['Sex'] == 'female'
# We can combine this with loc, then it will return a table of only female
train_data.loc[train_data['Sex'] == 'female']
# Or a little more complex conditional statement, also you cannot use "and", you need to use "&"
# Choosing female and older than 30 
train_data.loc[(train_data['Sex'] == 'female') & (train_data['Age'] > 30)]
# train_data[(train_data['Sex'] == 'female') & (train_data['Age'] > 30)]

# Choosing female or  older than 30 
train_data.loc[(train_data['Sex'] == 'female') | (train_data['Age'] > 30)]
# You can also do the same thing like without loc
train_data.loc[train_data['Sex'] == 'female']
train_data[train_data['Sex'] == 'female']
# However, you can only specify col in loc 
train_data.loc[train_data['Sex'] == 'female','Sex']<jupyter_output><empty_output><jupyter_text>Builtin Conditional Function isin()<jupyter_code># Let's use some built in conditional statement 
# isin() to choose specific value from the list, which is quite similar to the condition above 
train_data.loc[train_data['Embarked'] == 'S']
train_data.loc[train_data['Embarked'].isin(['S'])]
# However, the benefit of the isin is to choose multiple 
train_data.loc[train_data['Embarked'].isin(['S','Q','C'])]<jupyter_output><empty_output><jupyter_text>isnull()<jupyter_code># isnull() to choose the NaN
train_data.loc[train_data['Cabin']=='NaN']
train_data.loc[train_data['Cabin'].isnull()]<jupyter_output><empty_output><jupyter_text>Assigning Data<jupyter_code># Simply using the assigning operator 
train_data['Test_column'] = "Hello"
train_data.head()<jupyter_output><empty_output><jupyter_text>Summary Function describe() - high-level summary(count, mean, min, max, 25-75%)  mean() - get the average   unique() - get the unique values   value_counts() - get the unique values and frequency of the unique value   info() - non-null number, data type and summary<jupyter_code>train_data.describe()
train_data['Age'].describe()
train_data.mean()
train_data['Age'].mean()
train_data['Age'].unique()
train_data['Age'].value_counts()
train_data.info()<jupyter_output><empty_output><jupyter_text>Maps & Apply map is a function that takes one set of values and "maps" them to another set of values. there are 2 functions   1. map()     - only on one series- applymap()    - can be applied to multiple series- apply()     -  can be applied to series and dataframe<jupyter_code># Let's format so there are only two trailing zeros
format = lambda x: '%.2f' % x
train_data['Age'].map(format)<jupyter_output><empty_output><jupyter_text>Grouping Groupby() - Group the columns and changed the index to grouped category<jupyter_code># This just return Groupby object
train_data.groupby('Survived')
# But we can see the distribution of people who survived in the each category
train_data.groupby('Survived').count()
# Other functions could be applied to 
train_data.groupby('Survived').min()
# We can even create a multi level groupby 
train_data.groupby(['Survived','Sex']).count()<jupyter_output><empty_output><jupyter_text>agg() method - useful for statistical summary<jupyter_code># I want the count, minimum, maximum, average, and total of the age in the survived and non-survived group 
train_data.groupby(['Survived'])['Age'].agg([len,'min','max','mean','sum'])<jupyter_output><empty_output><jupyter_text>Sorting I wanna sort!<jupyter_code># Let's look at unsorted list
train_data
# Let's sort this in descending order, which is the default
train_data.sort_values(by='Age')
# Let's sort this in ascending order 
train_data.sort_values(by='Age', ascending=False)
# Let's sort by more than one column 
train_data.sort_values(by=['Age','Survived'], ascending=False)
# We can sort by index too 
# Let's set the index
train_data.set_index('Age')
# Let's sort the index descending
train_data.set_index('Age').sort_index()
# Let's sort the index ascending 
train_data.set_index('Age').sort_index(ascending=False)<jupyter_output><empty_output><jupyter_text>Dtypes<jupyter_code># What is the data types, show me!
# dtypes for dataframe
train_data.dtypes
# dtype for series
train_data['Survived'].dtype
# Convert the datatype using astype 
train_data['PassengerId'].dtype
train_data['PassengerId'].astype('float64')<jupyter_output><empty_output><jupyter_text>Missing data<jupyter_code># If you look at the data, you notice there are some missing values
train_data.head()
# This will print out if the selected value is NaN in boolean format for every single cells
train_data.isnull()
# We can obtain better summary using sum()
train_data.isnull().sum()
# We can obtain the percentage of the missing value
train_data.isnull().sum()/len(train_data) * 100
# sum/len is actually mean! 
train_data.isnull().mean() * 100<jupyter_output><empty_output><jupyter_text>Extracting missing values<jupyter_code># This will extract all the data with age = Nan
train_data[pd.isnull(train_data['Age'])]
# This is the same code as above
train_data.loc[train_data['Age'].isnull()]<jupyter_output><empty_output><jupyter_text>Filling missing values<jupyter_code>train_data2 = train_data.copy()
train_data2['Age'].fillna('Hey')<jupyter_output><empty_output><jupyter_text>Replacing values<jupyter_code>train_data2.isnull().mean()*100
train_data2['Age'].replace("Unknown",'NaN')
train_data2.isnull().mean()*100
del train_data2
train_data.drop('Test_column', axis=1)<jupyter_output><empty_output><jupyter_text>I think we know enough pandas, lets move on to     Data Exploration Before we dive into the machine learning, let's see what we have in here using our pandas skill  column description is as follows![image.png](attachment:image.png) Let's take a look at the train and test data sets.  Head and tail is useful to peak the first and last n number data sets  describe() will prodivde a descriptive statistics, such as mean, count, std...  info() provide nice summary of column name, non-null value count, and data types  And lastly, we want to know what percentage of data in column is NaN<jupyter_code># First 5
train_data.head()
# Last 5
train_data.tail()
# descriptive statistics
train_data.describe()
# Concise summary 
train_data.info()
# NaN percentage
train_data.isnull().mean()*100<jupyter_output><empty_output><jupyter_text>Data typesThere are many strings in the data sets, which require a conversion to the numerical value because ML algorithm only takes numerical values Missing valuesBased on this data, we know that there are many NaN in cabin, so we might just drop this column.  Moreover, there are some NaN in Age and a little in Embarked.  We need to deal with them later<jupyter_code>test_data.head()
test_data.tail()
test_data.describe()
test_data.info()
test_data.isnull().mean()*100<jupyter_output><empty_output><jupyter_text>You will notice that this data do no have "survived" column, because this data set will be used to calculate the "survived" prediction.The rest is the same as the train data sets. Since this is data exploration, let's take a look at each column!  SexBased on the data below, we can see that the survival rate of female is significantly higher than the survival rate of male<jupyter_code>train_data[['Sex','Survived']].groupby('Sex',as_index=False).mean()<jupyter_output><empty_output><jupyter_text>Pclass<jupyter_code>train_data[['Pclass','Survived']].groupby('Pclass',as_index=False).mean()<jupyter_output><empty_output><jupyter_text>Embarked<jupyter_code># Only Look for embarked column and drop na row
embarked = train_data.dropna(subset=['Embarked'])
embarked[["Embarked",'Survived']].groupby(["Embarked"], as_index=False).mean()<jupyter_output><empty_output><jupyter_text>Age<jupyter_code>train_data['Age'].isnull().value_counts()
train_data['category_age'] = pd.cut(train_data['Age'],5)
train_data[["category_age",'Survived']].groupby(["category_age"], as_index=False).mean()<jupyter_output><empty_output><jupyter_text>Sibsp and Parch<jupyter_code>Fare<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11602638727202173, 'fraction_numerical': 0.00682964687621265, 'mean_word_length': 6.483159117305458, 'pattern_counts': {'":': 1, '<': 113, '<?xml version=': 0, '>': 116, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '485947', 'license': 'MIT', 'n_tokens_mistral': 4342, 'n_tokens_neox': 4151, 'n_words': 1266, 'path': 'titanic/titanic_learn.ipynb', 'repo_name': 'whashi44/data_analysis', 'size': '12885'}