<jupyter_start><jupyter_text>An Introduction to `pandas`Pandas! They are adorable animals. You might think they are [the worst animal ever](https://www.reddit.com/r/todayilearned/comments/3azkqx/til_naturalist_chris_packham_said_he_would_eat/cshqy9y) but that is not true. You might sometimes think `pandas` is the worst library every, and that is only *kind of* true.The important thing is **use the right tool for the job**. `pandas` is good for some stuff, SQL is good for some stuff, writing raw Python is good for some stuff. You'll figure it out as you go along.Now let's start coding. Hopefully you did `pip install pandas` before you started up this notebook.<jupyter_code># import pandas, but call it pd. Why? Because that's What People Do.
import pandas as pd #so that you don't have to type pandas later -- most people use pd instead of pands<jupyter_output>/Users/Monica/.virtualenvs/dataanalysis/lib/python3.5/site-packages/matplotlib/__init__.py:1035: UserWarning: Duplicate key in file "/Users/Monica/.matplotlib/matplotlibrc", line #2
  (fname, cnt))<jupyter_text>When you import pandas, you use `import pandas as pd`. That means instead of typing `pandas` in your code you'll type `pd`.You don't *have* to, but every other person on the planet will be doing it, so you might as well. Now we're going to read in a file. Our file is called `NBA-Census-10.14.2013.csv` because we're **sports moguls**. `pandas` can `read_` different types of files, so try to figure it out by typing `pd.read_` and hitting tab for autocomplete.<jupyter_code># We're going to call this df, which means "data frame"
# It isn't in UTF-8 (I saved it from my mac!) so we need to set the encoding
#saved on mac, therefore the encoding needs to be mac_roman!
#it will not open if the encoding is not set :()
df = pd.read_csv('NBA-Census-10.14.2013.csv', encoding='mac_roman')

# encoding, the most common are: mac_roman if saved on a mac, latin-1 if saved on PC or UTF-8
 # 'pd.read_csv?' will give you more info about how read_csv works<jupyter_output><empty_output><jupyter_text>**A dataframe is basically a spreadsheet**, except it lives in the world of Python or the statistical programming language R. They can't call it a spreadsheet because then people would think those programmers used Excel, which would make them boring and normal and they'd have to wear a tie every day. Selecting rowsNow let's look at our data, since that's what data is for<jupyter_code># Let's look at all of it

print(df)<jupyter_output>Name  Age           Team  POS   #       2013 $  Ht (In.)  \
0           <NAME>   26      Cavaliers    F  33   $3,250,000        78   
1       <NAME>   31        Celtics    F  45  $10,105,855        79   
2          <NAME>   30  Trail Blazers    G  25   $2,652,000        73   
3     <NAME>   27          Magic    C  40     $762,195        83   
4    <NAME>   33           Jazz    F  44  $11,046,000        79   
5         <NAME>   22         Pacers    F   9   $1,246,680        79   
6       <NAME>   25   Timberwolves    F  10   $5,000,000        79   
7     <NAME>   22   Timberwolves    F   7   $5,016,960        80   
8          <NAME>   26         Lakers  F/C  27   $3,563,600        82   
9        <NAME>   30           Suns  F/C   8   $6,500,000        83   
10      <NAME>   25      Grizzlies    G   7   $3,135,000        75   
11         <NAME>   36           Nets    G  31[...]<jupyter_text>If we scroll we can see all of it. But maybe we don't want to see all of it. Maybe we hate scrolling?<jupyter_code># Look at the first few rows

df.head() # shows header + first 5 rows!<jupyter_output><empty_output><jupyter_text>...but maybe we want to see more than a measly five results?<jupyter_code># Let's look at MORE of the first few rows
df.head(10) # shows the first 10 lines of the program<jupyter_output><empty_output><jupyter_text>But maybe we want to make a basketball joke and see the **final four?**<jupyter_code># Let's look at the final few rows

df.tail(4) # shows the final four<jupyter_output><empty_output><jupyter_text>So yes, `head` and `tail` work kind of like the terminal commands. That's nice, I guess.But maybe we're incredibly demanding (which we are) and we want, say, **the 6th through the 8th row** (which we do). Don't worry (which I know you were), we can do that, too.<jupyter_code># Show the 6th through the 8th rows

df[6:9]<jupyter_output><empty_output><jupyter_text>It's kind of like an array, right? Except where in an array we'd say `df[0]` this time we need to give it two numbers, the start and the end. Selecting columnsBut jeez, my eyes don't want to go that far over the data. I only want to see, uh, name and age.<jupyter_code># Get the names of the columns, just because

df.columns #prints out the name of the columns - casing must match actually
# If we want to be "correct" we add .values on the end of it
df.columns.values
# Select only name and age

columns_we_want = ['Name', 'Age']

#passing the list of colums we want to data frame
df[columns_we_want]
# Combing that with .head() to see not-so-many rows
# We can also do this all in one line, even though it starts looking ugly
# (unlike the cute bears pandas looks ugly pretty often)

df[['Name', 'Age']] # brackets brackets<jupyter_output><empty_output><jupyter_text>**NOTE:** That was not `df['Name', 'Age']`, it was `df[['Name', 'Age']]`. You'll definitely type it wrong all of the time. When things break with pandas it's probably because you forgot to put in a million brackets.  Describing your dataA powerful tool of pandas is being able to select a portion of your data, *because who ordered all that data anyway*.<jupyter_code>df['POS'] # shows you each position<jupyter_output><empty_output><jupyter_text>I want to know how **many people are in each position**. Luckily, pandas can tell me!<jupyter_code># Grab the POS column, and count the different values in it.
df['POS'].value_counts() # counts the number of values that match each position
df['Race'].value_counts() # race of players<jupyter_output><empty_output><jupyter_text>**Now that was a little weird, yes** - we used `df['POS']` instead of `df[['POS']]` when viewing the data's details.But now I'm curious about numbers: **how old is everyone?** Maybe we could, I don't know, get some statistics about age? Some statistics to **describe** age?<jupyter_code># Summary statistics for Age
df['Age'].value_counts() # statistics about age
df['Age'].describe() #statistics about NBA players and their ages
# That's pretty good. Does it work for everything? How about the money?
df.describe() # shows info for all of the numerical data

# EEEK minum weight = 20 lbs -- seems incorrect
df['Ht (In.)'].describe()
#df.columns # look at column names again

df['2013 $'].describe() # this column is string as opposed to int -- therefore it didn't work :()<jupyter_output><empty_output><jupyter_text>Unfortunately because that has dollar signs and commas it's thought of as a string. **We'll fix it in a second,** but let's try describing one more thing.<jupyter_code># Doing more describing<jupyter_output><empty_output><jupyter_text>That's stupid, though, what's an inch even look like? What's 80 inches? I don't have a clue. If only there were some wa to manipulate our data. Manipulating dataOh wait there is, HA HA HA.<jupyter_code># Take another look at our inches, but only the first few

df['Ht (In.)'].head()
# Divide those inches by 12
df['Ht (In.)'].head()/12 #divides every single value by 12 
# Let's divide ALL of them by 12
df['Ht (In.)']/12
# Can we get statistics on those?
height_in_feet = df['Ht (In.)']/12
height_in_feet.describe()
# Let's look at our original data again
df.head()<jupyter_output><empty_output><jupyter_text>Okay that was nice but unfortunately we can't do anything with it. It's just sitting there, separate from our data. If this were normal code we could do `blahblah['feet'] = blahblah['Ht (In.)'] / 12`, but since this is pandas, we can't. Right? **Right?**<jupyter_code># Store a new column
df['Ht (Ft.)'] = df['Ht (In.)']/12 # adds a new column with the height as feet

df.head()
df.sort_values('Ht (Ft.)') # automatically sorts from lowest to highest - ascending value
#shows the tallest players by height in feet
df.sort_values('Ht (Ft.)', ascending =False).head() # automatically sorts from lowest to highest - ascending value
#shows you who is/isn't above 6"5 ft. 
above_or_below_six_five = df['Ht (Ft.)'] > 6
above_or_below_six_five.value_counts() # returns how many players are or are not above 6"5<jupyter_output><empty_output><jupyter_text>That's cool, maybe we could do the same thing with their salary? Take out the $ and the , and convert it to an integer?<jupyter_code># Can't just use .replace

# Need to use this weird .str thing

# Can't just immediately replace the , either

# Need to use the .str thing before EVERY string method

# Describe still doesn't work.

# Let's convert it to an integer using .astype(int) before we describe it

# Maybe we can just make them millions?

# Unfortunately one is "n/a" which is going to break our code, so we can make n/a be 0

# Remove the .head() piece and save it back into the dataframe<jupyter_output><empty_output><jupyter_text>The average basketball player makes 3.8 million dollars and is a little over six and a half feet tall.But who cares about those guys? I don't care about those guys. They're boring. I want the real rich guys! Sorting and sub-selecting<jupyter_code># This is just the first few guys in the dataset. Can we order it?

# Let's try to sort them<jupyter_output><empty_output><jupyter_text>Those guys are making nothing! If only there were a way to sort from high to low, a.k.a. descending instead of ascending.<jupyter_code># It isn't descending = True, unfortunately

# We can use this to find the oldest guys in the league
#shows the oldest players 
df.sort_values('Age', ascending =False).head() # automatically sorts from lowest to highest - ascending value
# Or the youngest, by taking out 'ascending=False'

#shows the youngest players 
df.sort_values('Age').head() # automatically sorts from lowest to highest<jupyter_output><empty_output><jupyter_text>But sometimes instead of just looking at them, I want to do stuff with them. Play some games with them! Dunk on them~ `describe` them! And we don't want to dunk on everyone, only the players above 7 feet tall.First, we need to check out **boolean things.**<jupyter_code># Get a big long list of True and False for every single row.
#shows you who is/isn't above 6"5 ft. 
above_or_below_six_five = df['Ht (Ft.)'] > 6
# print(above_or_below_six_five)
# We could use value counts if we wanted
above_or_below_six_five.value_counts() # returns how many players are or are not above 6"5
# But we can also apply this to every single row to say whether YES we want it or NO we don't
above_or_below_six_five = df['Ht (Ft.)'] > 7
df[df['Race'] == 'Asian']
# Instead of putting column names inside of the brackets, we instead
# put the True/False statements. It will only return the players above 
# seven feet tall

# Or only the guards
df[df['Ht (Ft.)'] > 7]
# Or only the guards who are under 6 feet tall

# are you a guard? AND are below 6 feet tall?
df[(df['POS'] == 'G') & (df['Ht (Ft.)'] < 6)]
# It might be easier to break down the booleans into separate variables
is_a_guard =  df['POS'] == 'G'
is_below_six_feet = df['Ht (Ft.)'] < 6
df[is_a_guard & is_below_six_feet]
centers = df[df['POS'] == 'C']
guards = df[df['POS'] == 'G']
# We can save this stuff

centers['Ht (Ft.)'].describe()
guards['Ht (Ft.)'].describe()
# Maybe we can compare them to taller players?<jupyter_output><empty_output><jupyter_text>Drawing picturesOkay okay enough code and enough stupid numbers. I'm visual. I want graphics. **Okay?????** Okay.<jupyter_code>!pip install matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
# This will scream we don't have matplotlib.
df['Ht (Ft.)'].hist()<jupyter_output><empty_output><jupyter_text>`matplotlib` is a graphing library. It's the Python way to make graphs!<jupyter_code>%matplotlib inline
# save things as .png and not .jpeg

plt.savefig('heights.png')
# this will open up a weird window that won't do anything

# So instead you run this code<jupyter_output><empty_output><jupyter_text>But that's ugly. There's a thing called ``ggplot`` for R that looks nice. We want to look nice. We want to look like ``ggplot``.<jupyter_code># Import matplotlib
# What's available?

# Use ggplot

# Make a histogram

# Try some other styles<jupyter_output><empty_output><jupyter_text>That might look better with a little more customization. So let's customize it.<jupyter_code># Pass in all sorts of stuff!
# Most from http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html
# .range() is a matplotlib thing<jupyter_output><empty_output><jupyter_text>I want more graphics! **Do tall people make more money?!?!**<jupyter_code># How does experience relate with the amount of money they're making?

# At least we can assume height and weight are related

# At least we can assume height and weight are related
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html

# We can also use plt separately
# It's SIMILAR but TOTALLY DIFFERENT<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09602225731257989, 'fraction_numerical': 0.01857282502443793, 'mean_word_length': 4.129193983802545, 'pattern_counts': {'":': 0, '<': 105, '<?xml version=': 0, '>': 107, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '243042', 'license': 'MIT', 'n_tokens_mistral': 4436, 'n_tokens_neox': 4128, 'n_words': 1814, 'path': '07 - Introduction to Pandas.ipynb', 'repo_name': 'M0nica/python-foundations-hw', 'size': '13398'}