<jupyter_start><jupyter_text>Super Heroes DatasetOne of our exercise during bootcamp in Eskwelabs Notebook by DavidData came from:     Source: Kaggle     Title: Super Heroes Dataset, How's your average superhero?     by:ClaudioDavi     link: https://www.kaggle.com/claudiodavi/superhero-set        Description by ClaudioDaviSuper Heroes DatasetContextSuper Heroes have been in popular culture for a long time and now more than ever. Since its creation, super heroes have not been very diverse, but that is changing rapidly. This dataset aims to provide an overview about heroes and their physical as well as power characteristics, helping researchers and curious minds identify trends and patterns.ContentThis data was collected in June/2017 from superherodb and not updated since, so it may not be up to date. There are two datasets here. The first one lists the characteristics of every super hero found on my source, the second contains information about every superpower and lists if they are present in any given hero.AcknowledgementsThis data was scraped from SuperHeroDb https://www.superherodb.com/.InspirationWhat are the characteristics of your favorite super heroes?  Dataset:1. heroes_information.csv(48.04 KB) -This file holds the demographics of each hero as well as which comic/studio they originate from. 2. super_hero_powers.csv(656.55 KB) - The format is Boolean. Each hero is listed under hero names and then each remain field has a TRUE/FALSE entry regarding if that hero has that power.  My objectives for this project are following:  * practice with Pandas * data cleaning * visualization * providing insights * practice stats  Let's begin  import necessary libraries<jupyter_code>#data manipulation
import pandas as pd
import numpy as np

#visualization
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Read the two datasets<jupyter_code>#added parameter index_col to remove unnamed:0
df_heroes_info = pd.read_csv('data/heroes_information.csv', index_col = 0)
df_super_pow = pd.read_csv('data/super_hero_powers.csv')<jupyter_output><empty_output><jupyter_text>Explore the two raw datasets  heroes_information.csv == df_heroes_info   what it looks like<jupyter_code>df_heroes_info.head()
df_heroes_info.tail()
df_heroes_info.sample(n = 5, random_state =42)<jupyter_output><empty_output><jupyter_text>Observation: We have negative weight  Shape of Data set<jupyter_code>df_heroes_info.shape
#we have 734 rows and 10 columns<jupyter_output><empty_output><jupyter_text>type of Data set<jupyter_code>df_heroes_info.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 734 entries, 0 to 733
Data columns (total 10 columns):
name          734 non-null object
Gender        734 non-null object
Eye color     734 non-null object
Race          734 non-null object
Hair color    734 non-null object
Height        734 non-null float64
Publisher     719 non-null object
Skin color    734 non-null object
Alignment     734 non-null object
Weight        732 non-null float64
dtypes: float64(2), object(8)
memory usage: 63.1+ KB<jupyter_text>Observation df_heroes_info: * 734 rows* 10 columns* data type: compose of objects and floats64* missing values columns Publisher, Weight  Check missing values<jupyter_code>df_heroes_info.isna().sum()<jupyter_output><empty_output><jupyter_text>Missing values in terms of percentage<jupyter_code>df_heroes_info.isna().sum()/len(df_heroes_info)<jupyter_output><empty_output><jupyter_text>another way of checking missing values (for smaller datasets)credit to my teammate Stephene for this code<jupyter_code>stats = []
for col in df_heroes_info.columns:
    stats.append((col, df_heroes_info[col].nunique(), df_heroes_info[col].isnull().sum() * 100 / df_heroes_info.shape[0], df_heroes_info[col].value_counts(normalize=True, dropna=False).values[0] * 100, df_heroes_info[col].dtype))
    
stats_df = pd.DataFrame(stats, columns=['Feature', 'Unique_values', 'Percentage of missing values', 'Percentage of values in the biggest category', 'type'])
stats_df.sort_values('Percentage of missing values', ascending=False)
df_heroes_info.duplicated(subset='name').sum()
df_heroes_info[df_heroes_info.duplicated(subset='name')]
df_heroes_info[df_heroes_info['name'] == 'Angel']
df_heroes_info[df_heroes_info['name'] == 'Vindicator']
df_heroes_info[df_heroes_info.duplicated()]<jupyter_output><empty_output><jupyter_text>Observation: * 19 have the same name but different attributes* Only Goliath is the duplicated with all same attributes   removing one duplicate<jupyter_code>df_heroes_info = df_heroes_info[~df_heroes_info.duplicated()].reset_index(drop=True)
#(734, 10) to (733, 11)
df_heroes_info.shape<jupyter_output><empty_output><jupyter_text>number unique names<jupyter_code>df_heroes_info['name'].nunique()<jupyter_output><empty_output><jupyter_text>super_hero_powers.csv == df_super_pow   what it looks like<jupyter_code>df_super_pow.head()
df_super_pow.tail()
df_super_pow.sample(n = 5, random_state =42)
#to see all the columns thank you pdeguzman96
pd.options.display.max_rows = df_super_pow.shape[1]
df_super_pow.head(2).T<jupyter_output><empty_output><jupyter_text>Shape of Data set<jupyter_code>df_super_pow.shape<jupyter_output><empty_output><jupyter_text>type of data set<jupyter_code>df_super_pow.info(verbose = True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 667 entries, 0 to 666
Data columns (total 168 columns):
hero_names                      object
Agility                         bool
Accelerated Healing             bool
Lantern Power Ring              bool
Dimensional Awareness           bool
Cold Resistance                 bool
Durability                      bool
Stealth                         bool
Energy Absorption               bool
Flight                          bool
Danger Sense                    bool
Underwater breathing            bool
Marksmanship                    bool
Weapons Master                  bool
Power Augmentation              bool
Animal Attributes               bool
Longevity                       bool
Intelligence                    bool
Super Strength                  bool
Cryokinesis                     bool
Telepathy                       bool
Energy Armor                    bool
Energy Blasts                   bool
Duplication                     bool
Size[...]<jupyter_text>Check missing values<jupyter_code>df_super_pow.isna().sum()<jupyter_output><empty_output><jupyter_text>removing one duplicate<jupyter_code>df_super_pow.duplicated(subset='hero_names').sum()<jupyter_output><empty_output><jupyter_text>number unique names<jupyter_code>df_super_pow['hero_names'].nunique()<jupyter_output><empty_output><jupyter_text>Observation df_super_pow: * 667 rows* 168 columns* all unique hero_names* data type: compose of one object and rest bool* no missing data  Merge both dataset<jupyter_code>#before we can merge the dataset we need to change name to hero_names

df_heroes_info = df_heroes_info.rename(columns={'name': 'hero_names'})
df_heroes_merged = pd.merge(df_heroes_info, df_super_pow, how = 'left')
df_heroes_merged.head()
df_heroes_merged.shape
df_heroes_merged.isna().sum()
#let fix the name of columns 
df_heroes_merged.rename(columns={'hero_names': 'Heroes Name'}, inplace = True)<jupyter_output><empty_output><jupyter_text>Pit shop<jupyter_code>#save it first just in case
df_heroes_merged2 = df_heroes_merged.copy()
df_heroes_merged2.head()
df_heroes_merged2['Web Creation'].value_counts()<jupyter_output><empty_output><jupyter_text>Replace True or False with 1 or 0s<jupyter_code>df_heroes_merged = df_heroes_merged.replace(to_replace= (True, False), value = (1,0))
#checking if replaced were done correctly
df_heroes_merged['Web Creation'].value_counts()<jupyter_output><empty_output><jupyter_text>Value Counts<jupyter_code>df_heroes_merged.select_dtypes(['object']).columns.values
### let make a columns that we are interested
df_heroes_col = ['Gender', 'Eye color', 'Hair color','Publisher', 'Skin color', 'Alignment']
plt.figure(figsize=(13, 10))
i = 1
for col in df_heroes_col:
    plt.subplot(3, 2, i)
    
    sns.countplot(df_heroes_merged[col])
    plt.xticks(rotation=90)
    plt.tick_params(labelbottom=True)
    
    i += 1

plt.tight_layout()
df_heroes_merged.head()<jupyter_output><empty_output><jupyter_text><jupyter_code>df_heroes_merged['Phoenix Force'].value_counts()<jupyter_output><empty_output><jupyter_text>Observations + How to deal with themWe have missing values We have - in columns Gender, Eye Color, Hair Color, Skin Color and Alignment we have values -99 in the columns of Height and Weight upon checking website - safe to say - are unknownsReplace - with unknow, and -99 missing values<jupyter_code>df_heroes_merged = df_heroes_merged.replace(to_replace= '-', value = 'Unknown')
# we can still use mean function even with missing values question. Is the value correct. Let's make an experiment
df_heroes_merged2['Web Creation'].mean()
## experiment

x=  pd.Series([2, 3, np.nan, 4], dtype=pd.Int64Dtype())
x
x.mean()
np.mean([2,3,4])<jupyter_output><empty_output><jupyter_text>replace values of -99 with missing values<jupyter_code>df_heroes_merged = df_heroes_merged.replace(to_replace= -99.0, value = np.NaN)
df_heroes_merged
plt.figure(figsize=(13, 10))
i = 1
for col in df_heroes_col:
    plt.subplot(3, 2, i)
    
    sns.countplot(df_heroes_merged[col])
    plt.xticks(rotation=90)
    plt.tick_params(labelbottom=True)
    
    i += 1

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Insights from EDA* Majority of our superheroes are male* Blue eyes are has is the way to go* Majority of the skin and hair color are unknown* there are still more good superheroes than bad* Marvel and DC Domics are top producers of superheroes  Stats1. Who is the tallest superhero2. Top tallest superheroes3. Who is the heaviest superhero4. Who is lightest superhero5. What is the average weight6. What is the average height   Who is the heaviest superhero<jupyter_code>df_heroes_merged[df_heroes_merged['Height']== df_heroes_merged['Height'].max()]
df_heroes_merged[df_heroes_merged['Height']== df_heroes_merged['Height'].max()]['Heroes Name']<jupyter_output><empty_output><jupyter_text>Dont do this because you are getting the max of Heroes Name which is the last letter of the Alphabet Z<jupyter_code>#Zoom is not the tallest superhero. It was shown because of max name in the alphabet which start with Z
df_heroes_merged[['Heroes Name','Height']].max()<jupyter_output><empty_output><jupyter_text>Top 10 tallest superheroes<jupyter_code>df_height = df_heroes_merged[['Heroes Name', 'Height']].sort_values('Height', ascending = False)
df_height.head(10)
df_height.set_index('Heroes Name')[:10].plot.bar(legend = None)
plt.title("Tallest Superheroes", fontsize = 14)
plt.ylabel('Height cm')<jupyter_output><empty_output><jupyter_text>3. Who is the heaviest superhero<jupyter_code>df_heroes_merged[df_heroes_merged['Weight']== df_heroes_merged['Weight'].max()]
df_heroes_merged[df_heroes_merged['Weight']== df_heroes_merged['Weight'].max()][['Heroes Name','Weight']]<jupyter_output><empty_output><jupyter_text>Sasquatch https://www.superherodb.com/sasquatch/10-863/<jupyter_code>df_weight = df_heroes_merged[['Heroes Name', 'Weight']].sort_values('Weight', ascending = False)
df_weight.set_index('Heroes Name')[:10].plot.bar()
plt.title("Heaviest Superheroes")
plt.ylabel('Weight Kg')<jupyter_output><empty_output><jupyter_text>Thats weird Fin Fang Foom should be at least top on the list. Weight of Fin Fang Foom is based on tons that's why Fin Fang Foom https://www.superherodb.com/fin-fang-foom/10-1518/  Let's investigate and try to fix it :)<jupyter_code>#height of superheroes can be a good bases of weight
df_heroes_merged[['Heroes Name', 'Height', 'Weight']].sort_values('Height', ascending = False)[:50]<jupyter_output><empty_output><jupyter_text>Fin Fang Foom, Galactus, Groot original unit of weight is under tons. Need to convert it to kg. Before we can assume that Wolfsbane and rest will be in Kg. Let's check lightest<jupyter_code>df_weight_lt = df_heroes_merged[['Heroes Name', 'Weight']].sort_values('Weight', ascending = True)
df_weight_lt[:20]<jupyter_output><empty_output><jupyter_text>Need to convert it to kg original unit of weight. 1 ton = 907.185kg.Fin Fang Foom, Galactus, Groot, Iron Monger<jupyter_code>conv_weight_hero = ['Fin Fang Foom', 'Galactus', 'Groot', 'Iron Monger']
#fObj[dfObj['Product'].isin(['Mangos', 'Grapes']) ]

df_heroes_merged[df_heroes_merged['Heroes Name'].isin(conv_weight_hero)][['Heroes Name','Height','Weight']]

df_heroes_merged[df_heroes_merged['Weight'] == 18]<jupyter_output><empty_output><jupyter_text>It seems that we can't see replace because there are so same values so need to use other technique<jupyter_code>##let fix it later
#df.xs('C')['x']=10
df_heroes_merged.xs(255)['Weight'] = 18*907.185
df_heroes_merged[df_heroes_merged['Weight'] == 18]
df_heroes_merged.at[255,'Weight']= 16329 #255 index Fin Fang Foom 18tons to 16329kg
df_heroes_merged.at[272,'Weight']= 14515 #272index Galactus 16tons to 14515kg
df_heroes_merged.at[301,'Weight']= 3629 #301 index Groot 4tons to 3629kg
df_heroes_merged.at[345,'Weight']= 1814 #345 index Iron Monger 2tons to 1814.37kg

#check
df_heroes_merged[df_heroes_merged['Weight'] == 18]
df_weight_corrected = df_heroes_merged[['Heroes Name', 'Weight']].sort_values('Weight', ascending = False)
df_weight_corrected.set_index('Heroes Name')[:10].plot.bar()
plt.title("Heaviest Superheroes")
plt.ylabel('Weight Kg')<jupyter_output><empty_output><jupyter_text>4. Who is the lightest superhero<jupyter_code>df_heroes_merged[df_heroes_merged['Weight']== df_heroes_merged['Weight'].min()][['Heroes Name','Weight']]<jupyter_output><empty_output><jupyter_text>What is the average height<jupyter_code>df_heroes_merged['Height'].mean()<jupyter_output><empty_output><jupyter_text>What is the average weight<jupyter_code>df_heroes_merged['Weight'].mean()<jupyter_output><empty_output><jupyter_text>Answers to our Stats question1. Who is the tallest superhero * Fin Fang Foom	from Marvel Comics with Height of 975.0cm or 32ft. It's about as tall as a Telephone Pole.2. Top 10 tallest superheroes    * Fin Fang Foom	975.0, Galactus	876.0, Groot 701.0, Wolfsbane 366.0, MODOK 366.0, Sasquatch	305.0 Onslaught	305.0, Ymir	304.8, Rey 297.0, Juggernaut 287.0     3. Who is the heaviest superhero  * Sasquatch 900kg but based on the website Fin Fang Foom should be one of the top heaviest with 16329kg. Cause is because Weight is on kg and tons. During scraping but we already corrected it with the necessary adjustment 4. Who is lightest superhero    * <NAME> with 14kg5. What is the average weight    * 186.72cm6. What is the average height  * 185.47kg   BMI<jupyter_code>df_heroes_merged3 = df_heroes_merged.copy()
df_heroes_merged3['Height'] = df_heroes_merged3['Height'] * 0.01
df_heroes_merged3['bmi'] = df_heroes_merged3['Weight']/(df_heroes_merged3['Height'])**2
df_heroes_merged3<jupyter_output><empty_output><jupyter_text>Attributes  create total power columns<jupyter_code>#total powers
df_heroes_merged['total powers'] = df_heroes_merged.iloc[:, 10:].sum(axis = 1)
df_heroes_merged.head()
df_super_total_pow= df_heroes_merged[['Heroes Name','total powers']]
df_super_total_pow= df_heroes_merged.sort_values('total powers',ascending=False)
plt.figure(figsize= (12,12))
sns.barplot(x = 'Heroes Name', y = 'total powers', data = df_super_total_pow.head(21), palette='CMRmap')
plt.xticks(rotation = 90)
plt.xlabel("Name of Superhero",fontsize=18)
plt.ylabel("No. of Superpowers",fontsize=18)
plt.title('Top 20 Superheroes having highest no. powers')
plt.figure(figsize= (12,12))
sns.barplot(x = 'Heroes Name', y = 'total powers', data = df_super_total_pow.head(11), palette="CMRmap", hue = 'Publisher')
plt.xticks(rotation = 90)
plt.xlabel("Name of Superhero",fontsize=18)
plt.ylabel("No. of Superpowers",fontsize=18)
plt.title('Top 10 Superheroes having highest no. powers')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Does Batman have super powers beside being rich?<jupyter_code>df_bat = df_heroes_merged[df_heroes_merged['Heroes Name'] == 'Batman'] 
df_bat = df_bat.iloc[0, 10:] == 1
df_bat[df_bat].index.tolist<jupyter_output><empty_output><jupyter_text>Does Batman have super powers beside being rich?* 17 to be exact 'Agility', 'Durability', 'Stealth', 'Underwater breathing', 'Marksmanship', 'Weapons Master', 'Intelligence', 'Super Strength', 'Stamina', 'Super Speed', 'Weapon-based Powers', 'Peak Human Condition', 'Reflexes', 'Gliding', 'Power Suit', 'Vision - Night', 'Vision - Infrared'  Top count of attributes used<jupyter_code>df_heroes_merged['Agility'].sum()
df_heroes_merged['Agility'].value_counts()
df_heroes_merged.iloc[:, 10:].sum(axis = 0)
attributes = pd.DataFrame(df_heroes_merged.iloc[:, 10:].sum(axis = 0))
attributes.rename(columns = {0:'Count'}, inplace = True)
attributes = attributes.reset_index()
attributes.rename(columns = {'index':'Attribute'}, inplace = True)
attributes.drop(index = 167, inplace = True)
plt.figure(figsize= (9,9))
sns.barplot(x = 'Count', y = 'Attribute', data = attributes.sort_values('Count', ascending = False).head(10), palette='CMRmap')
plt.title('Number of Attributes')
plt.xticks(rotation = 90)

attributes[attributes['Count'] < 2]<jupyter_output><empty_output><jupyter_text>Average attribute by a publisher<jupyter_code>df_heroes_merged.groupby('Publisher')['total powers'].mean().sort_values(ascending= False)
#### <NAME> highest average of attributes<jupyter_output><empty_output><jupyter_text>Gender reference to Publisher<jupyter_code>df_gender = df_heroes_merged.groupby(['Publisher','Gender'])[['Heroes Name']].count().copy()
df_gender
df_gender = df_gender.unstack(level=-1)
df_gender['male_ratio'] =df_gender['Heroes Name']['Male']/(df_gender['Heroes Name']['Male'] +df_gender['Heroes Name']['Female'])
df_gender['male_ratio'].sort_values(ascending = False)
## <NAME>, Image Comics, Stra Trek has a high male ratio<jupyter_output><empty_output><jupyter_text>Ratio of Good in terms of Publisher<jupyter_code>df_alig = df_heroes_merged.groupby(['Publisher','Alignment'])['Heroes Name'].count().unstack(level=-1).copy()
df_alig['ratio good'] = df_alig['good'] / (df_alig['good'] + df_alig['bad'])
df_alig.sort_values(by='ratio good', ascending= False).head(10)
# DC has more good than Marvel but a small margin<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11000379136651682, 'fraction_numerical': 0.02404809619238477, 'mean_word_length': 6.063504208110176, 'pattern_counts': {'":': 0, '<': 173, '<?xml version=': 0, '>': 172, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '625270', 'license': 'MIT', 'n_tokens_mistral': 6641, 'n_tokens_neox': 6076, 'n_words': 1541, 'path': 'Day 2 SuperHeroes.ipynb', 'repo_name': 'Sbad24/Day-2-Superhero-Dataset', 'size': '18480'}