<jupyter_start><jupyter_text>Python for Data Science & Analysis  Notes 4.2, Exploratory Data Analysis & Visualization  Objectives* describe a standard EDA workflow* EXTRA: describe a standard communication & reporting process* write a program which uses pandas to:    * obtain a subset of columns using list indexes    * obtain a subset of rows using locator syntax      * obtain descriptive statistics     * applies a test to a column to discretize it    * factor a continuous column with a discrete column* write a program which uses seaborn and matplotlib to:    * show univariate plots (eg., distplot)    * show multivariate plots (eg., scatterplot)    * EXTRA: customize plots  Part 1: What is EDA?  What is Exploratory Data Analysis? * goal:    * investigate    * explain    * describe    * understand * questions?    * is there enough data?    * is the data correct?    * what is the distribution of each column?    * how do the columns correlate? * method    * visual    * primarily descriptive  What are the tools for EDA in Python? Simple pair: Pandas for data exploration, seaborn for visualization....but lots of options.   What are the challenges around EDA?  Challenges of EDA:* Strategic * Organizational* Technical   Part 2: EDA with Pandas<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import plotly.express as px

ti = pd.read_csv('datasets/titanic.csv')
ti.head(3)

df = pd.read_csv('datasets/titanic.csv')<jupyter_output><empty_output><jupyter_text>How do we EDA with Pandas? * data quality, structure, etc. metrics; $\rightarrow$ `.info()`* descriptive stats for all single columns; $\rightarrow$ `.describe()`, `.mean()`, `.value_counts()`* descriptive stats for all pairs of columns; $\rightarrow$ `.corr()`* domain-specific row subsets; $\rightarrow$ `df.loc`* introduce factoring columns; $\rightarrow$ `.groupby`* introduce domain-specific novel columns; $\rightarrow$ `df['bmi'] = df['w'] / df['h'] ** 2`     The purpose of computing a correlation score (default: pearson) is to determine *possible* relationships, rather than accurately measure correlation.Note the *sign* of the score, and it's rough magnitude. You should plot all columns and pairs *regardless*.   Exercise (20 min) Perform an EDA of the titanic dataset with pandas, include:* data quality, structure, etc. metrics    * `.info()`, `.isna().sum()`, etc.* descriptive stats for all single columns    * `.describe()`, `.mean()`, `.median()`, `.std()`, `.value_counts()`* descriptive stats for all pairs of columns    * ie., `.corr()`<jupyter_code>#data quality, structure, etc. metrics;
df.info
new_df = df.dropna()

print(new_df.to_string())
new_df.info
#To discover duplicates, we can use the duplicated() method.
#The duplicated() method returns a Boolean values for each row:
print(new_df.duplicated())
#remove duplicates
#df.drop_duplicates(inplace = True)
#descriptive stats for all single columns
new_df.describe()
#descriptive stats for all single columns
new_df.value_counts()
#descriptive stats for all single columns
new_df.mean()
new_df.median()
new_df.std()
men = new_df.loc[new_df.sex == 'male']["survived"]
rate_men = sum(men)/len(men)

print("% of men who survived:", rate_men)
women = new_df.loc[new_df.sex == 'female']["survived"]
rate_women = sum(women)/len(women)

print("% of women who survived:", rate_women)
#descriptive stats for all pairs of columns
new_df.corr()

new_df.plot()

px.histogram(ti, x='age', title='Was There An Age Bias?')
#ti = pd.read_csv("datasets/titanic.csv")
px.scatter(ti, x='age', y='survived', color='class', size='fare', title='Was There A Class Bias?')
fig = px.pie(ti, values='fare', names='sex', title='Was There A Gender Bias?')
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12018201284796574, 'fraction_numerical': 0.0021413276231263384, 'mean_word_length': 4.88503937007874, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '546822', 'license': 'MIT', 'n_tokens_mistral': 1161, 'n_tokens_neox': 1093, 'n_words': 425, 'path': '04_2_EDA_Pandas_Titianic_Dataset.ipynb', 'repo_name': 'MavrellousG/DFE_py_Analytics', 'size': '3736'}