<jupyter_start><jupyter_text>Pandas Advanced: Learning notebookIn this notebook we will be covering the following:- Pandas    - quick operations: sum, mean, median, max, min    - Select columns    - select rows: loc and iloc- Python    - Strings        - accessing by index    - Lists        - append        - accessing by index    - Loops    - Functions        - Return        - Print- Python and Pandas    - Applying Python functions over Pandas Series and DataFrames    - lambda functions    - Creating new columns as a result from an operation----<jupyter_code>import pandas as pd

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"<jupyter_output><empty_output><jupyter_text>Python  Strings So a string is just plain text<jupyter_code>mystring = 'hello world!'<jupyter_output><empty_output><jupyter_text>We can print it<jupyter_code>print(mystring)<jupyter_output><empty_output><jupyter_text>We can also access a specific character by index<jupyter_code>mystring[0]<jupyter_output><empty_output><jupyter_text>Negative index means to start counting backwards<jupyter_code>mystring[-1]<jupyter_output><empty_output><jupyter_text>> **quiz**: how to get only the space character from the string above?<jupyter_code>space_char = mystring#[?]<jupyter_output><empty_output><jupyter_text>SubstringsWe can also create substrings from strings this way:<jupyter_code>mystring[0:5]
mystring[6:]<jupyter_output><empty_output><jupyter_text>Look at the first example `mystring[0:5]`Even though we said we want every character from the 0th position to the 5th, the last character 'o' is in the index position 4. This means that the second argument of the slice is exclusive, and the first is inclusive. > quiz: how to get "hello world" without the "!"?<jupyter_code>#answer =<jupyter_output><empty_output><jupyter_text>Lists A list can hold multiple elements inside<jupyter_code>a = [1,2,3,4,5]<jupyter_output><empty_output><jupyter_text>And we can access lists pretty much the same way we access strings: with slices!<jupyter_code>a[0]
a[-1]<jupyter_output><empty_output><jupyter_text>Lists can hold  other elements such as strings<jupyter_code>b = ['aneeb', 'workshop']<jupyter_output><empty_output><jupyter_text>And we can also store multiple types<jupyter_code>c = [1,'hello',2,'world']<jupyter_output><empty_output><jupyter_text>appendWe can actually add elements to a list in the following way:<jupyter_code>d = []
d
d.append('abc')
d
d.append('def')
d<jupyter_output><empty_output><jupyter_text>Notice that append adds the element to the last position  Loops In python loops are pretty simple. Here's how you make a loop from 1 to 10:<jupyter_code>for i in range(1,10):
    print(i)<jupyter_output><empty_output><jupyter_text>hum... it appears range is also exclusive on the second argumentThe argument after `for i in ...` just has to be an iterable (like a list or a string!)<jupyter_code>a = [1,10,100,1000]
for i in a:
    print(i)<jupyter_output><empty_output><jupyter_text>We can do more interesting things with loops of course<jupyter_code>numbers = [2,3,4,5,6,7]

for n in numbers:
    if n % 2 == 0:
        print(f'{n} is even')
    else:
        print(f'{n} is odd')<jupyter_output><empty_output><jupyter_text>> quiz: given a list of numbers, how can i calculate the total sum of those numbers?<jupyter_code>mylist = [100,200,300,400]

# ...<jupyter_output><empty_output><jupyter_text>Functions Typical example of a function in python.<jupyter_code>def add(a,b):
    return a + b
add(1,2)
def multiply(a,b):
    total = a
    for e in range(1,b):
        total += a
    return total
multiply(3,7)<jupyter_output><empty_output><jupyter_text>Functions receive arguments, then perform operations and at the end they can return a result with `return`. However, functions don't need to always return something. For example might want to print something:<jupyter_code>def say_hello(name):
    
    print(f'Hello, {name}!')
say_hello('everyone')<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>data_path = "https://raw.githubusercontent.com/vohcolab/ANEEB-2021-Python-Workshop/main/Pandas/Pandas%20Advanced/data/sales.csv"
df = pd.read_csv(data_path)
df.head(3)
df.shape<jupyter_output><empty_output><jupyter_text>Pandas allows us to perform simple math operations out of the box<jupyter_code>df['sales']<jupyter_output><empty_output><jupyter_text>What's the total value of sales worldwide?<jupyter_code>df['sales'].sum()<jupyter_output><empty_output><jupyter_text>What about on average?<jupyter_code>df['sales'].mean()<jupyter_output><empty_output><jupyter_text>And the standard deviation?<jupyter_code>df['sales'].std()<jupyter_output><empty_output><jupyter_text>hum.... median?<jupyter_code>df['sales'].median()<jupyter_output><empty_output><jupyter_text>Selecting columns We select the columns we want<jupyter_code>df['date']<jupyter_output><empty_output><jupyter_text>this returned a series. But we can also select multiple columns<jupyter_code>df[['date','country']]<jupyter_output><empty_output><jupyter_text>Selecting rows `.iloc` is the most used method to select in pandas<jupyter_code>df.loc[5:10,:]
df.loc[5:10,['sales','n_items']]<jupyter_output><empty_output><jupyter_text>Python and Pandas Let's take leverage of what we know of Python and **apply** that to handling data in Pandas.<jupyter_code>df.head(3)<jupyter_output><empty_output><jupyter_text>Looking at the **sales** and **n_items** columns, it would be nice to know, on average, how much each item costed.We will apply a function over all the rows and for each row let's divide **sales** by **n_items** and check the result<jupyter_code>def sales_per_item(row):
    """
    Gets the average sales per item
    
    Parameters
    ----------
    row : pd.Series
        A pandas series corresponding to a row of the dataset of sales.
    """
    return row.sales / row.n_items

df.apply(sales_per_item, axis=1)<jupyter_output><empty_output><jupyter_text>----  Let's go by steps:- Our goal is to go over the rows and divide the values of the two columns: `sales` and `n_items`- Pandas's `apply` function allows us to... surprise, surprise... apply a function over the dataset- The first argument of `apply` receives a function. `apply` will pass the contents of each iteration to this function and the function should return a result.- The second argument of the apply funcion `axis=1` tells Pandas to apply a function over the rows. This means that for each iteration, the function receives a row, which is just a Pandas Series with the index being the columns of the dataframe, and the values being the values of that row for those columns There are multiple ways to get the same answer to this question. Each with their own quirks. For example, we don't need to define explicitly the `sales_per_item` function:<jupyter_code>df.apply(lambda row: row.sales / row.n_items, axis=1)<jupyter_output><empty_output><jupyter_text>See? same result. Let's understand what is happening:- `lambda row`: **lambda** is just there to tell python this will be a lambda function. **row** means that our function receives a single argument and will refer to it as **row**. It could be anything else really, like `lambda x`.- `: row.sales / row.n_items`: the `:` tells python that now comes the body of the function, which, in lambda functions, is only one line of code. It acts like a `return row.sales / row.n_items`.Lambda functions are pretty useful if we just want to do a simple operation for example, like dividing two numbers in this case.Creating a whole function for doing the same thing takes a bit of time, and `lambda` functions save us that time. ----  Creating new columns as a result of operations over Pandas objects Now, notice that the result is itself a pandas series. The index is the same as the original dataframe, that means we can create a new column for this new information!<jupyter_code>df['cost_per_item'] = df.apply(lambda row: row.sales / row.n_items, axis=1)

df.head(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09952841896252172, 'fraction_numerical': 0.00980392156862745, 'mean_word_length': 5.552032520325203, 'pattern_counts': {'":': 0, '<': 129, '<?xml version=': 0, '>': 132, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '221740', 'license': 'MIT', 'n_tokens_mistral': 2633, 'n_tokens_neox': 2527, 'n_words': 858, 'path': 'Pandas/Pandas Advanced/Learning Notebook.ipynb', 'repo_name': 'vohcolab/ANEEB-2021-Python-Workshop', 'size': '8058'}