<jupyter_start><jupyter_text>Practicing Descriptive Statistics with `NumPy` and `pandas`Much like the theme of the `R` practice notebook, data exploration isn't complete without manipulating the data. Splitting the data across categories and discovering the underlying features of the categories could lead to better, more powerful insights. In this practice, we will be working with the data manipulation capabilities of `pandas` and the statistical functionality that `Numpy` provides to further investigate the *Game of Thrones* data.<jupyter_code>import pandas as pd
import numpy as np

with open('../../datasets/game-of-thrones/GoT_age_at_death.csv') as file:
    df = pd.read_csv(file)
    df.columns  = ['character', 'age', 'dead', 'gender', 'affiliation'] # rename the columns
    
    # turn age, gender, and affiliation into categorical data
    df['dead'] = df['dead'].astype('category') 
    df['gender'] = df['gender'].astype('category')
    df['affiliation'] = df['affiliation'].astype('category')
df.head(7)<jupyter_output><empty_output><jupyter_text>In module 1, we didn't introduce you to `pandas`' `groupby` method, but it operates on a data frame the same way that `dplyr`'s `group_by` method does. Below is how we would group the data frame by `gender` (so 0 and 1) and produce the `mean` age for both.<jupyter_code>df.groupby(['gender'])['age'].mean()<jupyter_output><empty_output><jupyter_text>We could go even more fine-grained than this. **Activity 1**: *Find the mean age per `gender` and per `affiliation`.*<jupyter_code># Activity 1 code goes here
# -------------------------


df.groupby(['gender'])['age'].mean()
df.groupby(['affiliation'])['age'].mean()<jupyter_output><empty_output><jupyter_text>In the `R` practice, we emphasized descriptive statistics of those characters who have died in the series. Just for the sake of comparison, let's look at those who are alive. **Activity 2**: *Create a subset from the data frame of only those who are living. Call this subset `alive_chars`.*<jupyter_code># Activity 2 code goes here
# -------------------------


alive_chars = df[df['dead']==0]<jupyter_output><empty_output><jupyter_text>Remember in the lab, we used `NumPy` to find the mean age of the entire dataset, the syntax for which is below. **Activity 3**: *Find the mean age of those who are alive.*<jupyter_code>np.mean(df.age)
# Activity 3 code goes here
# -------------------------


np.mean(alive_chars.age)<jupyter_output><empty_output><jupyter_text>Below is one of the ways that we found the standard deviation.<jupyter_code>np.std(df.age)<jupyter_output><empty_output><jupyter_text>But remember, without any extra arguments passed to the method, we get the population standard deviation. What we want is the sample standard deviation.**Activity 4**: *Find the sample standard deviation of the ages of the characters that are still alive.*<jupyter_code># Activity 4 code goes here
# -------------------------


np.std(alive_chars.age,ddof = 1)<jupyter_output><empty_output><jupyter_text>In the `R` practice we referred to the symmetry of a distribution. As a reminder, when the mean is greater than the median, it means that the data is skewed to the right, and when the median is greater than the mean, the data is skewed to the left. When the mean and median are the same, it means that the data are symmetrical. In other words, in a right skewed distribution, most of the data points are below the mean, but there are some high value points pulling the mean higher; the opposite is true for a left skew.Below is an example of how we found the median before.<jupyter_code>np.median(df.age)<jupyter_output><empty_output><jupyter_text>**Activity 5**: *Is the distribution of the age of those who are alive skewed to the right, the left, or symmetrical?*<jupyter_code># Activity 5 code goes here
# -------------------------

median = np.median(alive_chars.age)
mean = np.mean(alive_chars.age)


if mean > median:
    print('right')
elif median > mean:
    print('left')
else:
    print('symmetric')<jupyter_output>right<jupyter_text>`NumPy` also gives us the functionality to see what value a certain percentage of the rows fall under by calling the `percentile` method. We did this to find the 65th percentile of ages of the entire data fame by doing the following...<jupyter_code>np.percentile(df.age, 65)<jupyter_output><empty_output><jupyter_text>Now, remember we discussed the Inter Quartile Range in the `R` lab notebook.**Activity 6**: *Find the the Inter Quartile Range (IQR) of the `alive_chars` age variable.*<jupyter_code># Activity 6 code goes here
# -------------------------


q75, q25 = np.percentile(alive_chars.age, [75 ,25])
iqr = q75 - q25
iqr<jupyter_output><empty_output><jupyter_text>Bivariate AnalysisNow we are going to switch back over to the *Stature Hand and Foot* data frame, and practice some bivariate analysis on some variables.<jupyter_code>with open('../../datasets/stature-hand-foot/stature-hand-foot.csv') as file2:
    df2 = pd.read_csv(file2)
    df2['gender'] = df2['gender'].astype('category')
    df2.columns = ['gender', 'height', 'hand_length', 'foot_length']<jupyter_output><empty_output><jupyter_text>First, we should actually split the data based on male and female. **Activity 7**: *Create two subsets of the data, one for females and the other for males. Name the two subsets `female` and `male` respectively.*<jupyter_code># Activity 7 code goes here
# -------------------------
# male is 1, female is 2

female = df2[df2['gender']==2]
male = df2[df2['gender']==1]<jupyter_output><empty_output><jupyter_text>You will recall that we could create a correlation and covariance matrix in `R`. The same functionality is available in `pandas`. This is a good way to quickly spot some linear relationships between variables. Below is an example of how to create a correlation matrix on the entire data frame.<jupyter_code>df2.corr()<jupyter_output><empty_output><jupyter_text>Simple enough. And in `pandas` it knows to ignore the non-quantitative columns. **Activity 8**: *Create a covariance and correlation matrices for both the males and females of this dataset.*<jupyter_code># Activity 8 code goes here
# -------------------------


female.corr()
male.corr()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12892430278884462, 'fraction_numerical': 0.00844621513944223, 'mean_word_length': 4.490813648293964, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '332569', 'license': 'MIT', 'n_tokens_mistral': 1898, 'n_tokens_neox': 1789, 'n_words': 730, 'path': 'Assingments/module05/students/module2/practices/.ipynb_checkpoints/practice_descriptive_stats_python-checkpoint.ipynb', 'repo_name': 'tonysulfaro/MI-250', 'size': '6275'}