<jupyter_start><jupyter_text>**Data Science in Python workshop** <NAME>, Applied Science @ Discovery Channel___ ** teaching materials in these notebooks are solely the property of <NAME>, and cannot be used for other workshops or courses other than by the author or by license. Students are open to review these notebooks at any time, but with attribution of credit back to the author.  What Is Pandas?Pandas is a Python library that primarily adds two new datatypes to Python: `DataFrame` and `Series`.- A `Series` is a sequence of items, where each item has a unique label (called an `index`).- A `DataFrame` is a table of data. Each row has a unique label (the `row index`), and each column has a unique label (the `column index`).- Note that each column in a `DataFrame` can be considered a `Series` (`Series` index).> Behind the scenes, these datatypes use the NumPy ("Numerical Python") library. NumPy primarily adds the `ndarray` (n-dimensional array) datatype to Pandas. An `ndarray` is similar to a Python list — it stores ordered data. However, it differs in three respects:> - Each element has the same datatype (typically fixed-size, e.g., a 32-bit integer).> - Elements are stored contiguously (immediately after each other) in memory for fast retrieval.> - The total size of an `ndarray` is fixed.> Storing `Series` and `DataFrame` data in `ndarray`s makes Pandas faster and uses less memory than standard Python datatypes. Many libraries (such as scikit-learn) accept `ndarray`s as input rather than Pandas datatypes, so we will frequently convert between them.  Using PandasPandas is frequently used in data science because it offers a large set of commonly used functions, is relatively fast, and has a large community. Because many data science libraries also use NumPy to manipulate data, you can easily transfer data between libraries (as we will often do in this class!).Pandas is a large library that typically takes a lot of practice to learn. It heavily overrides Python operators, resulting in odd-looking syntax. For example, given a `DataFrame` called `cars` which contains a column `mpg`, we might want to view all cars with mpg over 35. To do this, we might write: `cars[cars['mpg'] > 35]`.Do not be discouraged if Pandas feels overwhelming. Gradually, as you use it, you will become familiar with which methods to use and the "Pandas way" of thinking about and manipulating data.<jupyter_code>#Because it's not part of base python, we have to import pandas. Standard convention is to import as pd
import pandas as pd
#For google colab, we need to upload files - there are other methods worth exploring on your own. 
from google.colab import files
uploaded = files.upload()
#we can use pandas to read files in from our google drive data folder! Cool, eh?
users = pd.read_table('./user.tbl', sep = '|')
users.info()
#We can return the user table - this is how we use it to look its nicest
users
#if we print the table, it returns a collection of series. Why? 
print(users)<jupyter_output>user_id  age gender     occupation zip_code
0          1   24      M     technician    85711
1          2   53      F          other    94043
2          3   23      M         writer    32067
3          4   24      M     technician    43537
4          5   33      F          other    15213
..       ...  ...    ...            ...      ...
938      939   26      F        student    33319
939      940   32      M  administrator    02215
940      941   20      M        student    97229
941      942   48      F      librarian    78209
942      943   22      M        student    77841

[943 rows x 5 columns]<jupyter_text>Viewing a dataframeBelow are some methods for inspecting a dataframe<jupyter_code>#we can look at just a sample of the data frame by using the .head() method
users.head(n=5)<jupyter_output><empty_output><jupyter_text>We would describe this as `the head method from the users dataframe object`. The `5` we pass it is a parameter. Some parameters can be implicit.<jupyter_code>users.head(5)<jupyter_output><empty_output><jupyter_text>And sometimes parameters have default values that are used if you don't input anything.<jupyter_code>users.head()<jupyter_output><empty_output><jupyter_text>If we want to view the tail, we can do it by using the `.tail()` method. Try it on your own!<jupyter_code>#Use .tail to view the end of a dataframe
users.tail(12)<jupyter_output><empty_output><jupyter_text>We can also look at what the columns are.<jupyter_code>for i, namespace in enumerate(users.columns):
  print(i, namespace)
#Because it's a list, we can actually iterate over it. We'll talk more about this code in a second!
for col in users.columns:
  print(col)
  print(users[col].head(5))<jupyter_output>user_id
0    1
1    2
2    3
3    4
4    5
Name: user_id, dtype: int64
age
0    24
1    53
2    23
3    24
4    33
Name: age, dtype: int64
gender
0    M
1    F
2    M
3    M
4    F
Name: gender, dtype: object
occupation
0    technician
1         other
2        writer
3    technician
4         other
Name: occupation, dtype: object
zip_code
0    85711
1    94043
2    32067
3    43537
4    15213
Name: zip_code, dtype: object<jupyter_text>Methods use () to transform something about a dataframe. If you just want to report something about the dataframe, it does not use `()`. For example, we can look at the types of data in our dataframe<jupyter_code>users.dtypes<jupyter_output><empty_output><jupyter_text>Notice that `gender` etc. are `objects` - this is pandas' designation for string like objects. It is typically, but not always a string. Why do you think zipcode is stored as a string? Why not an integer?<jupyter_code>#we can look at the shape of our dataframe - which is the length by width. How do you interpret this? 
users.shape<jupyter_output><empty_output><jupyter_text>Pandas can also aggregate information about the dataframe for us, such as size and nulls and types. To do this, it needs to transform the data - so it will have a `()` at the end.<jupyter_code>users.info()
#if you don't have the (), it will return this:
users.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 943 entries, 0 to 942
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   user_id     943 non-null    int64 
 1   age         943 non-null    int64 
 2   gender      943 non-null    object
 3   occupation  943 non-null    object
 4   zip_code    943 non-null    object
dtypes: int64(2), object(3)
memory usage: 37.0+ KB<jupyter_text>We can select specific columns by calling the column index against the dataframe. If we want the gender column of users, we could type ```users['gender']```OR```users.gender```You will see the second approach from those of us who are old school. But the first approach is considered best, extensible programming. Extensible means it will be future-proof. Image we had a column called column... we'd never be able to use `users.column`, because it will return information about the dataframe.<jupyter_code>users['gender']
#You can select multiple columns by passing a list of columns to the index of the dataframe Users
users[['gender', 'occupation']].head(5)<jupyter_output><empty_output><jupyter_text>Filtering and Sorting a dataframeWe can filter dataframes by structuring comparitors with the dataframe column indexes. It sounds complicated, but it's pretty easy.<jupyter_code>#First, let's create a comparison to return true or false
users['gender'] == 'M'
#we can now use that as a 'mask' in the index of our dataframe to return only instances where this is true
users[users['gender'] == 'M']<jupyter_output><empty_output><jupyter_text>Here's one of those times Pandas breaks from python. We can use logical operators like `and` or `or`, but they do so using the 'bitwise' operator. and = `&`or = `|`<jupyter_code>#What is this logically combined mask doing?
users[(users['gender'] == 'M') | (users['occupation'] == 'student')]
#What's this doing? Why the double parentheses?
users[(users.age < 30) & ((users.gender == 'F') & (users.occupation == 'student'))].head()
#we can save the above as a new dataframe using `.copy()`
young_women_students = users[(users.age < 30) & ((users.gender == 'F') & (users.occupation == 'student'))].copy()
young_women_students.shape<jupyter_output><empty_output><jupyter_text>If we want to filter an object feature by multiple values, we can also use a list<jupyter_code>#This is preferred to multiple and conditions for one feature. 
users[users['occupation'].isin(['student', 'retired'])]<jupyter_output><empty_output><jupyter_text>We can sort dataframes by both numerical and alphabetical values.<jupyter_code>young_women_students.sort_values(by = 'age').head()<jupyter_output><empty_output><jupyter_text>How does pandas sort a dataframe? What's the logic here? How would we reverse it?<jupyter_code>#sort the dataframe by age in the opposite direction
young_women_students.sort_values(by = 'age', ascending = False)<jupyter_output><empty_output><jupyter_text>Note, some methods show a view of the dataframe without changing it in memory. To hold these changes 'in place', you'd want to pass the inplace parameter.<jupyter_code>young_women_students.sort_values(by = 'age', ascending = False, inplace = True)
young_women_students.head()<jupyter_output><empty_output><jupyter_text>Basic calculationsPandas gives you tools to do some quick and easy analysis of a dataframe. Here, we'll start answering business questions with pandas! Some import methods: `.describe()` returns the summary statistics about a dataframe. Some of these are pretty obvious - mean, median, min, max - and we'll talk about the others in the next notebook.<jupyter_code>#What is the average age of users? 
users['age'].mean()
#interesting - how would you describe that more fully? 
users.age.describe()<jupyter_output><empty_output><jupyter_text>You: "We have 943 users, from age 7 to 73. The average age is 34, and the mean age is 31. That is because the 'middle' age is pulled down by our young users."Your boss:"How many people below 18 do we have in here?!!?"<jupyter_code>#Answer your boss' question here, try using the .value_counts() method against age, and sort it in the parameters!
users[users['age'] < 18]['age'].value_counts(ascending = False)<jupyter_output><empty_output><jupyter_text>Your Boss: "Fascinating! Thanks for the quick insights! Hey, someone in engineering has a follow-up question for you to look into if you have a second: "<jupyter_code>#What is the average age of male technicians in zipcodes starting with 9? Are any of those < 18? 
eng_question = users[((users['gender'] == 'M') & (users['occupation'] == 'technician')) & (users['zip_code'].str.startswith('9'))].copy()
print('average age: {}'.format(eng_question['age'].mean()))
print('number under 18: {}'.format(eng_question[eng_question['age'] < 18].shape[0]))<jupyter_output>average age: 30.0
number under 18: 0<jupyter_text>Drilling in with split, apply, combineNow people have some advanced questions that require breaking things down a bit more. Advertising wants to partner with a beverage company, but they have a big question: By gender, how many students are under the legal US drinking age? To answer this, we'll create a new feature in our dataframe, and then apply split-apply-combine methodology with groupby!. To create a new column, we simply name the new column against our dataframe `users['newColumn']` and assign it a value that is dependent on other columns.Let's say advertising wants a flag for women over 40. We can create that target using a boolean, and pandas will do all of the work for us.<jupyter_code>users['adFlag_women40+'] = (users['age'] > 40) & (users['gender'] == 'F')
users.head()
#How many users are good targets for this ad? Use your value_counts() method again!
users['adFlag_women40+'].value_counts(normalize = True)<jupyter_output><empty_output><jupyter_text>Back to our question above: By gender, how many students are under the legal drinking age in the US? We start by creating an under21 flag, and then using a groupby to count the number of flags in the student, aggregated at the level of gender.<jupyter_code>#create our drinking flag
users['adFlag_under21'] = users['age'] < 21
users['adFlag_under21'].value_counts(normalize = True)
#Talk through this code with yourself, and then let's talk together!
users[users.occupation == 'student'].groupby(['gender', 'adFlag_under21']).agg({'adFlag_under21': 'count'})
#validate this with your pandas skills - are we sure this is correct? How did you do it? 
users[(users.occupation == 'student') & (users.gender == 'F') & (users.adFlag_under21 == True)].shape[0]<jupyter_output><empty_output><jupyter_text>Advertising: "Cool, thanks! What would you say the average age of 21+ men and women are by occupation?"<jupyter_code>#Work your magic! How would you pass more than one column to the groupby method? Any collections you know and love? 
users[users.adFlag_under21 == False].groupby(['gender', 'occupation']).agg({'age': 'mean'})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08922301938635323, 'fraction_numerical': 0.02251564646618837, 'mean_word_length': 4.356909239574816, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 95, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '540470', 'license': 'MIT', 'n_tokens_mistral': 4001, 'n_tokens_neox': 3747, 'n_words': 1708, 'path': '2_pandasExploration.ipynb', 'repo_name': 'burgamacha/DS-Masterclass', 'size': '13114'}