<jupyter_start><jupyter_text>Exercise Set 4: Data Structuring 1*Afternoon, August 14, 2018*In this Exercise Set we will apply some of the basic things we have learned with pandas.  Load modulesWe begin by loading relevant packages.<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Exercise Section 4.1: Weather, part 1Some data sources are open and easy to collect data from. They can be 'scraped' as is and they are already in a table format. This Exercise part of exercises is the first part of three that work with weather data, the follow ups are Exercise Sections 6.1 and 7.1. Our source will be National Oceanic and Atmospheric Administration (NOAA) which have a global data collection going back a couple of centuries. This collection is called Global Historical Climatology Network (GHCN). A description of GHCN can be found [here](https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/readme.txt). > **Ex. 4.1.1:** Use Pandas' CSV reader to fetch  daily data weather from 1864 for various stations - available [here](https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/). > *Hint 1*: for compressed files you may need to specify the keyword `compression`.> *Hint 2*: keyword `header` can be specified as the CSV has no column names.> *Hint 3*: Specify the path, as the URL linking directly to the 1864 file.<jupyter_code># [Answer to Ex. 4.1.1]

# Will be in assignment 1<jupyter_output><empty_output><jupyter_text>> **Ex. 4.1.2:** Structure your weather DataFrame by using only the relevant columns (station identifier, data, observation type, observation value), rename them. Make sure observations are correctly formated (how many decimals should we add? one?).> *Hint:* rename can be done with `df.columns=COLS` where `COLS` is a list of column names.<jupyter_code># [Answer to Ex. 4.1.2]

# Will be in assignment 1<jupyter_output><empty_output><jupyter_text>> **Ex. 4.1.3:**  Select data for the station `ITE00100550` and only observations for maximal temperature. Make a copy of the DataFrame. Explain in a one or two sentences how copying works.> *Hint 1*: the `&` operator works elementwise on boolean series (like `and` in core python).> *Hint 2*: copying of the dataframe is done with the `copy` method for DataFrames.<jupyter_code># [Answer to Ex. 4.1.3]

# Will be in assignment 1<jupyter_output><empty_output><jupyter_text>> **Ex. 4.1.4:** Make a new column called `TMAX_F` where you have converted the temperature variables to Fahrenheit. > *Hint*: Conversion is $F = 32 + 1.8*C$ where $F$ is Fahrenheit and $C$ is Celsius.<jupyter_code># [Answer to Ex. 4.1.4]

# Will be in assignment 1<jupyter_output><empty_output><jupyter_text>> **Ex 4.1.5:**  Inspect the indices, are they following the sequence of natural numbers, 0,1,2,...? If not, reset the index and make sure to drop the old.<jupyter_code># [Answer to Ex. 4.1.5]

df_select = df_select.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>> **Ex 4.1.6:** Make a new DataFrame where you have sorted by the maximum temperature. What is the date for the first and last observations?<jupyter_code># [Answer to Ex. 4.1.6]

df_sorted = df_select.sort_values(by=['obs_value'])
print("Min temp: ", df_sorted['obs_value'].iloc[0] , "Max temp: ", df_sorted['obs_value'].iloc[-1])<jupyter_output>Min temp:  -6.3 Max temp:  34.8<jupyter_text>> **Ex 4.1.7:** CVS-files: save your DataFrame as a CSV file. what does index argument do?> Try to save the file using a relative path and an absolut path. With a relative you only specify the file name. This will save the file in the folder you are currently working in. With an absolute path, you specify the whole path, which allows you to save the file in a folder of your choice<jupyter_code># [Answer to Ex. 4.1.7]

# Using relativ path
df_sorted.to_csv('df_sorted.csv', index=False)
# Using absolute path
df_sorted.to_csv('C:/Users/rch690/Downloads/df_sorted.csv', index=False)
# The index argument regulates whether or not the index is transfered to the CSV-file. Default is that it is transfered.<jupyter_output><empty_output><jupyter_text>> (Bonus) **Ex. 4.1.8**: A very compact way of writing code and making list in Python, is called list comprehensions. Depending on what you are doing, list can be more or less efficient that for example vectorized operations using NumPy. >Read about list comprehenseions online, and use it to make a list with the numbers from 0 to a million (10\*\*6), and add 3 to each element. Do the same doing NumPy, and time both methods. Which method is faster? > *Hint 1*: Use the `timeit` package for timing each method<jupyter_code>%%timeit
[i+3 for i in range(10**6)]

%%timeit
np.arange(10**6)+3<jupyter_output>4.14 ms ± 265 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
{'fraction_non_alphanumeric': 0.09912536443148688, 'fraction_numerical': 0.026655560183256977, 'mean_word_length': 4.843065693430657, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '600693', 'license': 'MIT', 'n_tokens_mistral': 1581, 'n_tokens_neox': 1471, 'n_words': 656, 'path': 'material/session_4/exercise_4_sol.ipynb', 'repo_name': 'karlbindslev/sds_group29', 'size': '4802'}