<jupyter_start><jupyter_text>*This notebook was created by [<NAME>](https://twitter.com/jeande_d) for the love of machine learning community. For any feedback, errors or suggestion, he can be reached on email (johnjw7084 at gmail dot com), [Twitter](https://twitter.com/jeande_d), or [LinkedIn](https://linkedin.com/in/nyandwi).*  Exploratory Data Analysis (EDA) EDA is an important step in ML project. This is where we get to dig into the data, trying to understand more about it. By performing EDA, we can:* Know the summary statistics* Spot if there are missing values and the appropriate strategy for handling it* Spot if the data is skewed (or imbalanced)* Correlation between features* Understand the important features (or attributes) worth keeping and unhelpful features * Above all, answer some of the pressing questions about the data. These questions can be specific to data and its features but will around things like why this and this, or what could have caused this and this based on the analysis, etc...There are no right or wrong questions, the idea here is to use data to see if we can help answer some questions.  Real World Data Exploratory Analysis : TitanicIn this lab, we will use Titanic dataset which is available on Kaggle. We can also load it from Seaborn datasets but we will use the real data from [Kaggle](https://www.kaggle.com/c/titanic/data).Contents: * [1. Imports](1)* [2.  Loading the dataset](2)* [3. Quick look into the dataset](3)* [4. Summary statistics](4)* [5. Basic information](5)* [6. Missing data](6)* [7. More analysis](7)* [8. Checking feature correlations](8)  1.   Imports<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>2. Loading the dataset<jupyter_code>titanic = sns.load_dataset('titanic')<jupyter_output><empty_output><jupyter_text>If the data was coming from external sources, say `csv`, we could have used Pandas to read the file.```titanic = pd.read_csv('.....')```  3. Quick Look into the dataset<jupyter_code>titanic.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 15 columns):
 #   Column       Non-Null Count  Dtype   
---  ------       --------------  -----   
 0   survived     891 non-null    int64   
 1   pclass       891 non-null    int64   
 2   sex          891 non-null    object  
 3   age          714 non-null    float64 
 4   sibsp        891 non-null    int64   
 5   parch        891 non-null    int64   
 6   fare         891 non-null    float64 
 7   embarked     889 non-null    object  
 8   class        891 non-null    category
 9   who          891 non-null    object  
 10  adult_male   891 non-null    bool    
 11  deck         203 non-null    category
 12  embark_town  889 non-null    object  
 13  alive        891 non-null    object  
 14  alone        891 non-null    bool    
dtypes: bool(2), category(2), float64(2), int64(4), object(5)
memory usage: 80.7+ KB<jupyter_text>By default, titanic.head() will display the first five rows. But you can display the rows you want, just pass the integer. Example: `titanic.head(10)`. .`tail() will display the last rows<jupyter_code>titanic.head()
titanic.tail()<jupyter_output><empty_output><jupyter_text>4. Summary statistics<jupyter_code>titanic.describe()
titanic.describe().transpose()<jupyter_output><empty_output><jupyter_text>5. Basic Information1. How many people who survived and died from the titanic crash? Can you use the appropriate visualization to show these people?<jupyter_code>titanic['survived'].value_counts()
sns.countplot(data=titanic, x='survived')<jupyter_output><empty_output><jupyter_text>2. How many `pclass` are there in the dataset? I it the same as the the text column `class`?<jupyter_code>titanic['pclass'].value_counts()
titanic['class'].value_counts()
p_class = titanic[['pclass', 'class']]
p_class.head()<jupyter_output><empty_output><jupyter_text>It seems that both of these columns are the same except that one is numeric and other is text. They basically contain the same information of the class that the passengers paid for. 3. How many genders are there in the dataset?<jupyter_code>titanic['sex'].value_counts()
sns.countplot(data=titanic, x='sex')<jupyter_output><empty_output><jupyter_text>4. Show the passengers who were less than 20 years<jupyter_code>less_than_20 = titanic[titanic['age'] < 20 ]

less_than_20.head()
len(less_than_20)<jupyter_output><empty_output><jupyter_text>5. Show the categories of the passengers in the column `who`. Use Pie chart to illustrate their values<jupyter_code>titanic.who.value_counts().plot(kind='pie')<jupyter_output><empty_output><jupyter_text>6. How many unique cities are there in the column `embark_town`? Plot their occurences.<jupyter_code>titanic['embark_town'].nunique()
titanic['embark_town'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>There is no limit to how deep youu can go to understand the dataset.  6. Missing Data<jupyter_code>titanic.isnull().sum()<jupyter_output><empty_output><jupyter_text>There are missing data in age and deck columns. We can also use heatmaps to show the missing values<jupyter_code>sns.heatmap(titanic.isnull(), yticklabels=False, cbar=False)<jupyter_output><empty_output><jupyter_text>7. More Analysis<jupyter_code>sns.countplot(data=titanic, x='survived', palette='autumn', hue='sex')
sns.countplot(data=titanic, x='survived', palette='viridis', hue='class')
titanic.age.plot(kind='hist',bins=30,color='green')
sns.scatterplot(data=titanic, x='age', y='fare', hue='class')<jupyter_output><empty_output><jupyter_text>That is obvious. The passengers who were in the first class paid more than other classes. And many females were in the first class.<jupyter_code>sns.scatterplot(data=titanic, x='age', y='fare', hue='sex')<jupyter_output><empty_output><jupyter_text>The purpose of this lab was to learn about the exploratory analysis. There is no limit to what you can do. The more time you spend with the data, the good your analysis will be. I used questions to make it simple and lead the way but there is no proper format for this. The goal is to be one with data (CC: Andrej).   8. Checking Correlating FeaturesChecking correlation can help you to see the similarity between features. If two features correlate, that means they contain the same information and if one of them is removed, the analysis/model can be less affected.<jupyter_code>correlation = titanic.corr()
correlation
correlation['survived']<jupyter_output><empty_output><jupyter_text>If you want to check how the features correlate with the label (`survived`, just for example), here is how to do it:<jupyter_code>#### Visualizing correlation

plt.figure(figsize=(12,7))

sns.heatmap(correlation,annot=True,cmap='crest')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09969481180061038, 'fraction_numerical': 0.021653829385263768, 'mean_word_length': 4.822335025380711, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 70, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '280183', 'license': 'MIT', 'n_tokens_mistral': 2259, 'n_tokens_neox': 2128, 'n_words': 752, 'path': '4_real_world_data_analysis_and_preparation/1_exploratory_data_analysis.ipynb', 'repo_name': 'bhuvanakundumani/machine_learning_complete', 'size': '6895'}