<jupyter_start><jupyter_text>Videos and Exercises for Session 2: Data Structuring in Pandas IIn this combined teaching module and exercise set, you will be working with structuring data. We will start out with a recap of some basic function and methods that become available in pandas. Then there will be a short intermezzo, where you will be required to think a bit about how to write readible code and make use of the method chaining opportunities that come with python. Then finally, you will get to work with some more advanced data types that are handled natively with pandas. The structure is as follows:1. Welcome (Back to) Pandas    - DataFrames and Series    - Operations with Elementary Data Types in the Context of Pandas        - Boolean Series        - String Operations        - Numeric Operations and Methods2. Readible Code and Method Chaining3. More Advanced Data Types    - Categorical Data    - Time Series Data    **NOTE:** I might be speaking quite slowly in some of the videos. A good advice is to turn up the speed to x1.25 or x1.5 if you want to get through without spending too much time:)  PackagesLoad in the required modules and set up the plotting library:<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import YouTubeVideo<jupyter_output><empty_output><jupyter_text>Wonder what `%matplotlib inline` does? It's not too important for now. However, if you are interested, you can read more about it [here](https://stackoverflow.com/questions/43027980/purpose-of-matplotlib-inline).  Part 1: Welcome (Back to) PandasAs mentioned during the lecture, data structuring skills are necessary to become a great data scientist. There is no way around it.Let's start with the basics. In the first video, we start out by talking about the fundamentals of Pandas: - The Pandas DataFrame and the Pandas Series. - We then proceed and discuss how indices and colums can be named and selected. - Finally, we make some simple operations.**NOTE:** This is mainly repitition of what you already worked with in assignment 0. If you feel comfortable with this, you should proceed to the associated exercises that will get you warmed up for the rest of this notebook.<jupyter_code>YouTubeVideo('uSiqryfDYo0', width=640, height=360)<jupyter_output><empty_output><jupyter_text>In this set of introductory exercises, we are going to briefly touch upon some of the tools from the video. Again, be aware that there is going to be some repition of the content from assignment 0 here - but these things are all very useful to get under your skin!> **Ex. 2.1.1**: Run the following two lines below. Explain to yourself what the second line did, and how each entry affected it. In relation to this, what did the first line ensure? When can this be useful? Try and play around with the seed.<jupyter_code>np.random.seed(seed=161193)
np_arr=np.round(np.random.exponential(10, size=(10**5,3)),0)<jupyter_output><empty_output><jupyter_text>Now, we want you to investigate how easy it is to go from a numpy `array` to a pandas `DataFrame`. You are going to do something similar to this repeatedly in this course!> **Ex. 2.1.2**: Create a Pandas DataFrame called `my_df` from `np_arr`:<jupyter_code># YOUR CODE HERE
raise NotImplementedError()

my_df<jupyter_output><empty_output><jupyter_text>Before we continue working with this DataFrame, we want to make sure that you are capable of transforming it back to different types of containers. This is another very fundamental tool in your data science toolbox.> **Ex. 2.1.3**: In the following cell, generate an `array`, a `list` and a `dict` from `my_df` (you should be able to do each of these things in one line). Verify that you did this properly by printing your output. Inspect the dictionary: How do you access the value associated with the column index 2 and row index 8543?<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>A noticeable distiction between pandas dataframes and, say, numpy arrays is the option to label indices and (in particular) columns. This is helpful when working with large sets of data.> **Ex. 2.1.4**: Going back to the DataFrame `my_df`, we now want you to try the following: > 1. Label the columns 'apples', 'bananas' and 'oranges'> 2. Change the indices to 'cust**x**' where **x** is a sequential numbering of the lines starting from 1 (*hint:* There are many ways to do this, e.g. with a list comprehension!)> 3. Print the first 10 observations using the `head()` method.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>As we saw in the video, we can basically extract a series from a pandas dataframe by simply selecting one column from the dataframe and storing it as a new object.> **Ex. 2.1.5**: Now, we want you to select only a single column and inspect the output a bit:> 1. Select 'apples' and store it as 'apples_ser'. Try using the key-based method and the object-based method, respectively. > 2. Check the data type. Thinking about our first lines of code, why is it not an `integer`? Convert the data in both 'apples_ser' and 'my_df' to intergers.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Boolean Series<jupyter_code>YouTubeVideo('CfiV8QmfGVk', width=640, height=360)<jupyter_output><empty_output><jupyter_text>Boolean 'data' is the simplest data type that we work with. These variables are binary, and basically represent a True or False value. As seen in the video, they become extraordinarily useful when working with pandas series. In particular, boolean series can be applied when we only want to work with certain rows in a dataframe.> **Ex. 2.1.6**: Drop all rows for which 'Apples' is strictly smaller than 5.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Numeric Operications and Methods<jupyter_code>YouTubeVideo('ClB-n1f9sBM', width=640, height=360)<jupyter_output><empty_output><jupyter_text>As we saw in the video, we can easily make vectorized operations with multiple numpy arrays as well as pandas series and dataframes. This feature can become useful in many cases - in particular, when you have a large set of data and you do not want to make a (computationally exhaustive) loop through all you observation.> **Ex. 2.1.7**: In this exercise, we want you to make operations on multiple vectors and matrices using a combination of numpy arrays and pandas dataframes.> 1. Create a numpy array (with 3 elements) of costs associated with different types of fruits. Call this 'price'. Entries should be 1, 4 and 3, respectively.> 2. Create a new dataframe with information on each costumer's expenses associated with each type of fruit. This should involve multiplying your cost array with the 'my_df'.> 3. Create a new column called 'Total expenditure' by summing over fruit-specific prices for each customer (*hint:* Learn more about such sum [here](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sum.html)).> 4. Sort the dataframe by total expenditure, starting with the customer with highest expenditure.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>When we have a large data set, it is difficult to get an idea of it's characteristics by simply sorting and printing. In such cases, we much rather want to look at some kinds of descriptive statistics for our data. > **Ex. 2.1.8**: In this exercise, we will ask you to use two methods from the video in order to shed some light on the distribution of total expenditure:> 1. Use `describe()` to get an idea about the distribution of total expenditure in the data set. What does the distance between `mean` and the `50%` quantile tell you about the skewness?> 2. Use the `cut` method to split the sample into 6 equal sized bins of expenditures from 0 to 600. Find the *share* of observations within each of these bins (we will work more with *categorical data* later in this session).<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>String Operations<jupyter_code>YouTubeVideo('Yjdd8rKj_WI', width=640, height=360)<jupyter_output><empty_output><jupyter_text>Finally, we saw that pandas handles string variable natively. In particular, there is a set of string-specific operations (`.str`) that can be applied to a whole pandas series without having to loop through each row.> **Ex. 2.1.9**: In this final introductory exercise, we want you to > 1. Create a new variable (column) based on the index called 'Customer'. This should capitalize the customer id and write out 'Cust' to 'Customer' (i.e. you should get something similar to 'Customer **x**'). > 2. Reset the index.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Part 2: Readible CodeBefore proceeding to part 3 on advanced data types that are handled natively in pandas, we want you to think a bit about writing readable code. In particular, if we have lots of code, it may be very difficult for others or ourselves to read. Therefore, providing some structure and meta text can help reading the code.In the following, you will learn about readible code. Then we ask you to use what you learned in order to process the NOAA data from assignment 0. Don't spend too long on this exercise - we are going to use the data for the following exercises.  CommentingWhen making code it's good practice to document different parts of the code. In particular describing functions and complex code. The example below shows how to make multi-line comments (as a string, which is not assigned) and in-line comments using the `` character.```pythondef my_fct(x,y):    '''     Computes the sum of input values (multi-line comment as string)    '''    z = x+y  Here we perform the summation (in-line comment)    return z``` Method chainingWe can write multiple operations together by putting them one after the other, which is known as `method chaining`. Using this, we only need to assign them once to a new object and therefore we save a lot of code. We change the example below into one using a method chain:Example without method chain```pythondf_temp1 = df.loc[selection]df_temp2 = df_temp1.sort_values(by=X)df_out = df_temp2.reset_index()```Example with method chain - one line```pythondf_out = df.loc[selection].sort_values(by=X).reset_index()```As seen in the example, although using less code, our method chain will get more and more difficult to read if we include two or more operations. We can overcome this problem of long chains by splitting into multiple lines with line breaks:Example with method chain - line break```pythondf_out = df\        .loc[selection]\        .sort_values(by=X)\        .reset_index()```Note that the backslash allows us to make a visual line break, but the code is read as one line.  Exercises for Part 2We continue with the exercise of analyzing NOAA data. We start out reviewing what we did in Assignment 0.  > **Ex. 2.2.1:** The code below runs through all the steps we completed in Assignment 0.4 step by step. Your task is to document this code in your own words. You should also make your own annotation of parts. In addition, you should make the code more readable by applying method chaining.> >*Hint:* After loading the data and selecting the first four columns, everything can be stated in one method chain! It's OK if you use more, though.```pythonimport pandas as pddef load_weather(year):        url = f"ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/{year}.csv.gz"    df_weather = pd.read_csv(url,                              header=None)            df_weather = df_weather.iloc[:,:4]         column_names = ['station', 'datetime', 'obs_type', 'obs_value']    df_weather.columns = column_names         df_weather['obs_value'] = df_weather['obs_value'] / 10         selection_tmax = df_weather.obs_type == 'TMAX'    df_select = df_weather.loc[selection_tmax]        df_sorted = df_select.sort_values(by=['station', 'datetime'])    df_reset = df_sorted.reset_index(drop=True)    df_out = df_reset.copy()                return df_out```<jupyter_code># YOUR CODE HERE
raise NotImplementedError()

load_weather(1863)<jupyter_output><empty_output><jupyter_text>Part 3: Advanced Data TypesWhen you start working with data, you quickly realize that data comes in many different types. Some distinctions between data types are obvious to make. For instance, numeric variables should consists of meaningful values that represent something that can be measured (a length, weight, economic value, etc.) with a number. String variables, on the other hand, are variables that *may* also consist of letters and other characters in addition to numbers. And boolean variables are simply binary with only two possible values.However, we also encounter lots of other data types in applied work within social sciences. Categorical variables can only take a (often quite strictly) limited number of values. And time series data are characterized by being indexed in some time order. In this part, we will explore different these two data types and how you work with them in Pandas. > **Ex. 2.3.1:** Load the data for year 1863. Extract the area code (country and state) from the station name into a separate column.>> _Hint:_ The station column contains a GHCND ID, given to each weather station by NOAA. The format of these ID's is a 2 letter country/state code, followed by possible additional information on the third character and then an integer identifying the specific station. A simple approach is to assume a fixed length of the country ID. A more complex way would be to use the [`re`](https://docs.python.org/3.8/library/re.html) module.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Categorical Data Pandas has built-in features for working specifically with categorical data. In the video below, we will discuss how you can think about categorical data, and how you can efficiently work with it in pandas.<jupyter_code>YouTubeVideo('x2TCnT46cl4', width=640, height=360)<jupyter_output><empty_output><jupyter_text>Having watched the video introducing categorical data, we now want you to try and work with it.> **Ex. 2.3.2:** Do the following:> 1. Convert the `area` column to a categorical variable. > 2. Transform the `obs_value` column from a continuous to a categorical variable by partitioning it into `3` intervals. The first interval should contain observations with values of `obs_value` up to the 10% quantile. The second interval should contain observations with values of `obs_value` up to the 90% quantile. The third interval should contain the rest of the observations. Call this new column for `obs_value_cat`.  This can be done using the `pd.qcut()` method.> 3. Make another column with  `obs_value` as a categorical variable but this time label the 3 intervals as `["cold", "medium", "hot"]`. This can be done by specifying the `labels` parameter in the `pd.qcut()` method of pandas. Call this new column for `obs_value_cat_labeled`.<jupyter_code># INCLUDED IN ASSIGNMENT 1
# INCLUDED IN ASSIGNMENT 1<jupyter_output><empty_output><jupyter_text>Time Series DataOur coverage of basic Python did not include time. This is another elementary datatypes, that has its own native structure or maybe converted to an integer using a smart method. See more below.<jupyter_code>YouTubeVideo('DmkT7PdqgAA', width=640, height=360)<jupyter_output><empty_output><jupyter_text>> **Ex. 2.3.3:** Convert the date formated as string to datetime. Call this column `datetime_dt`. Make a new column named `month` with the month for each observation. Set the datetime variable as temporal index. >> *Hint:* Look up `.set_index()` setting the index.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>> **Ex. 2.3.4:** Update your annotated function above with processing of area and temporal data.<jupyter_code># INCLUDED IN ASSIGNMENT 1
# INCLUDED IN ASSIGNMENT 1# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>> **Ex. 2.3.5:** Make a timeseries plot for the station called `AGE00135039`.>> _Hint:_ for this you need to know a few methods of the pandas Series objects, for instance `.plot()`.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0702496826068557, 'fraction_numerical': 0.012332990750256937, 'mean_word_length': 4.888928444286223, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 128, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '19464', 'license': 'MIT', 'n_tokens_mistral': 4608, 'n_tokens_neox': 4329, 'n_words': 2281, 'path': 'teaching_material/module_2/module_2_exercises.ipynb', 'repo_name': 'MathildeWB/isds2021-1', 'size': '16541'}