<jupyter_start><jupyter_text>Pandas  What is pandas? Pandas is an open-source python library that can be used to create and query small datasets. You can use it to load  .csv files to your jupyter notebook. To learn more about data structures in pandas see their webpage. In this lecture we will go through some of the useful functions of this library while looking at the heroes dataset from Kaggle.NOTE: Before continuing, please download the dataset and add it to the folder where this notebook resides.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading files The command read_csv() loads a ".csv" file to jupyter notebook as an object of type pandas.dataframe.<jupyter_code>df_heroes_info = pd.read_csv("./heroes/heroes_information.csv")
df_heroes_powers = pd.read_csv("./heroes/super_hero_powers.csv")<jupyter_output><empty_output><jupyter_text>To take a peek at the dataframes we can use the method head which displays the top 5 entries.<jupyter_code>df_heroes_info.head()
df_heroes_powers.head()<jupyter_output><empty_output><jupyter_text>Querying the dataset To access a particular column of the dataframe we can use the access the elements like a python dictionary. The returned object is of the type panda.Series.<jupyter_code>print(df_heroes_info['name'])
print(type(df_heroes_info['name']))<jupyter_output>0                 A-Bomb
1             <NAME>
2               <NAME>
3            Abomination
4                Abraxas
5          Absorbing Man
6            <NAME>
7           <NAME>
8               Agent 13
9              Agent Bob
10            Agent Zero
11            Air-Walker
12                  Ajax
13            <NAME>
14           <NAME>
15          <NAME>
16     <NAME>
17                 Alien
18      <NAME>
19                 Amazo
20                  Ammo
21        <NAME>
22                 Angel
23                 Angel
24            <NAME>
25       <NAME>
26                Angela
27            Animal Man
28             Annihilus
29               Ant-Man
             ...        
704               Walrus
705          War Machine
706              Warbird
707              Warlock
708                 Warp
709              Warpath
710                 Wasp
711              Watcher
712            Weapon XI
[...]<jupyter_text>To see all the available possible powers recorded we can use the method columns() on the df_heroes_powers dataframe.<jupyter_code>list_powers = [power for power in df_heroes_powers.columns][1:]
for powers in list_powers: 
    print(powers)<jupyter_output>Agility
Accelerated Healing
Lantern Power Ring
Dimensional Awareness
Cold Resistance
Durability
Stealth
Energy Absorption
Flight
Danger Sense
Underwater breathing
Marksmanship
Weapons Master
Power Augmentation
Animal Attributes
Longevity
Intelligence
Super Strength
Cryokinesis
Telepathy
Energy Armor
Energy Blasts
Duplication
Size Changing
Density Control
Stamina
Astral Travel
Audio Control
Dexterity
Omnitrix
Super Speed
Possession
Animal Oriented Powers
Weapon-based Powers
Electrokinesis
Darkforce Manipulation
Death Touch
Teleportation
Enhanced Senses
Telekinesis
Energy Beams
Magic
Hyperkinesis
Jump
Clairvoyance
Dimensional Travel
Power Sense
Shapeshifting
Peak Human Condition
Immortality
Camouflage
Element Control
Phasing
Astral Projection
Electrical Transport
Fire Control
Projection
Summoning
Enhanced Memory
Reflexes
Invulnerability
Energy Constructs
Force Fields
Self-Sustenance
Anti-Gravity
Empathy
Power Nullifier
Radiation Control
Psionic Powers
Elasticity
Substance Secretion
Eleme[...]<jupyter_text>To count the number of elements in each column we can use the function value_counts. Suppose we want to see the number of heroes that fight for the good side. We can query the dataset as follows:<jupyter_code>df_heroes_info['Alignment'].value_counts()<jupyter_output><empty_output><jupyter_text>Next we want to find the 5 most common powers and all unique powers in dataset. One way to do it is as follows:<jupyter_code>num_powers = df_heroes_powers.sum(axis=0)[1:] #+ np.zeros(len(list_powers)-1)<jupyter_output>Agility                   242
Accelerated Healing       178
Lantern Power Ring         11
Dimensional Awareness      25
Cold Resistance            47
Durability                257
Stealth                   126
Energy Absorption          77
Flight                    212
Danger Sense               30
Underwater breathing       21
Marksmanship              119
Weapons Master            105
Power Augmentation          8
Animal Attributes          25
Longevity                 113
Intelligence              158
Super Strength            360
Cryokinesis                19
Telepathy                  92
Energy Armor                8
Energy Blasts             147
Duplication                16
Size Changing              55
Density Control            15
Stamina                   289
Astral Travel               4
Audio Control               7
Dexterity                   6
Omnitrix                 True
                         ... 
Intuitive aptitude       True
Vision - Microscopic       19
Melting   [...]<jupyter_text>The 5 most common powers<jupyter_code>common_pow = num_powers.sort_values(ascending=False)[0:5].index
for power in common_pow:
    print(power)<jupyter_output>Super Strength
Stamina
Durability
Super Speed
Agility<jupyter_text>All the unique power<jupyter_code>unique_pow = num_powers.sort_values(ascending=True)[num_powers.sort_values(ascending=True)==1].index
for power in unique_pow:
    print(power)<jupyter_output>Thirstokinesis
Intuitive aptitude
Hyperkinesis
Hair Manipulation
Electrical Transport
Anti-Gravity
Spatial Awareness
Biokinesis
Omnitrix
Speed Force
Phoenix Force
Molecular Dissipation
Banish
Changing Armor<jupyter_text>We can also do more complex query such as look at the powers that each publisher uses.<jupyter_code>pub_power_data = pd.merge(df_heroes_info[['name','Publisher']],df_heroes_powers,left_on='name',right_on='hero_names',left_index=True)
grouped = pub_power_data.groupby('Publisher')
grouped.sum().sum(axis=1)<jupyter_output><empty_output><jupyter_text>Visualizing Data Visualizing data is fundamental in understanding more about the dataset. There are plenty of Python libraries for visualization. We will be using the matplotlib library for all out plots. If we want to look at the heroes created by each publisher, we can visualize the data using a bar graph.<jupyter_code>heroes_publisher = pd.value_counts(df_heroes_info['Publisher'])
heroes_publisher.plot(kind='bar')
plt.show()<jupyter_output><empty_output><jupyter_text>We can also have different formatting styles for our plots. The different types of formats available are:<jupyter_code>styles = plt.style.available
for style in styles:
    print(style)<jupyter_output>bmh
classic
dark_background
fivethirtyeight
ggplot
grayscale
seaborn-bright
seaborn-colorblind
seaborn-dark-palette
seaborn-dark
seaborn-darkgrid
seaborn-deep
seaborn-muted
seaborn-notebook
seaborn-paper
seaborn-pastel
seaborn-poster
seaborn-talk
seaborn-ticks
seaborn-white
seaborn-whitegrid
seaborn
_classic_test<jupyter_text>To learn more about the different styles available and creating your own style visit here. Let's use the seaborn-talk style for now<jupyter_code>plt.style.use('seaborn-talk')
heroes_publisher.plot.bar()
plt.show()<jupyter_output><empty_output><jupyter_text>Let us look at the height distribution of our heroes using a histogram.<jupyter_code>plt.figure(figsize=(12,9))
df_heroes_info['Height'].plot.hist(20)
plt.xlabel('Height')
plt.ylabel('Frequency')
plt.show()<jupyter_output><empty_output><jupyter_text>Did you notice something odd with this plot? Height cannot be negative! What's happened here is that for all the characters for whom the data is missing their height was initialized as -99. This brings us to our next topic which is missing data.  Missing Data Impossible values such as -99 for height or NaN are often used in datasets to fill unknown indexes. The easiest way to resolve this is to remove these indexes.Replotting the histogram without the outliers we have<jupyter_code>df_heroes_info[df_heroes_info['Height']!=-99]['Height'].plot.hist(20)
plt.xlabel('Height')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05521324151262535, 'fraction_numerical': 0.020538842575812493, 'mean_word_length': 3.3545502367175173, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '617039', 'license': 'MIT', 'n_tokens_mistral': 2618, 'n_tokens_neox': 2386, 'n_words': 820, 'path': 'Week 6 _ Condiitonal Probability/Lecture_pandas.ipynb', 'repo_name': 'ebishwaraj/ProbabilityStatisticsPython_DSE210x_UCSD_DataScienceMicroMasters', 'size': '8351'}