<jupyter_start><jupyter_text>Fetching and storing dataIn this chapter we will consider 2 sources of data: the network and the filesystem. We will also look at storing your data to the filesystem. More often than not the data you need for your program will come from somewhere outside of your program.Often, that "somewhere" will be a file or some location on the internet. Especially for more complex data, it becomes essential to be able to fetch data from the outside, or persist data to somewhere outside of your program. 1. Climate dataFor the plotting project you've been commissioned to complete, you've been provided with a file containing monthly global average observations and access to a website for data from individual countries. You will need to read the global data from the file andload the country data from the website. Let us begin... 2. How to deal with the outside worldThe outside world is a dangerous and scary place. Here, in your Python programming environment you are relatively safe. But out there lie horrific things like missing files, URLs that point to nowhereand even... INVALID DATA!!! Luckily Python gives us some mechanisms to help us deal with this frightening environment and return to our comfortable Python interpreter as quickly as possible.There are, in general (this is true of _all_ operating environments), 3 steps to dealing with the outside world:1. "Open" a resource (could be a file or a URL, or...)2. "Read" from or "write" to the resource3. "Close" the resourcePython provides us with the concept of a _context manager_ which handles the "close" step for us. So we only need worry about opening the correct resource and performing the I/O we need.The `with` keyword tells Python to _manage a context_ ([Documentation](https://docs.python.org/3/reference/datamodel.htmlcontext-managers)). Within this context (within an indentedcode block) you're able to read from and write to the resource. As soon as you're outside of the code block, the resource will not exist any more. Let's see what a context manager looks like with an example reading from file and web resources...<jupyter_code># Open a file resource...
with open("data/readfile.txt") as file_resource: # file_resource is a context manager
    # Read from the file resource
    contents = file_resource.read()

# Outside the with block, the context manager has done it's job and closed the file resource.
# We cannot access file_resource now... but we still have the contents!
print(contents)
from urllib import request

# Open a web resource...
with request.urlopen("https://httpbin.org/get") as web_resource: # web_resource is a context manager
    # Read from the web resource
    contents = web_resource.read()

# Outside the with block, the context manager has done it's job and closed the web resource.
# We cannot access web_resource now... but we still have the contents!
print(contents)<jupyter_output><empty_output><jupyter_text>--- 3. Explore the global dataIn order to understand how to use the data in a file, we need to understand what's in it. As you saw above, the `read()` function reads all the data from the file. The global data file is probably quite large though so let's try to read less. We have 2 options: The `readline()` function will read up to a new line character, or we could pass an argument to the `read()` function that is the number of characters to read.<jupyter_code>with open("data/HadCRUT.5.0.1.0.analysis.summary_series.global.monthly.csv") as global_data_manager:
    first_line = global_data_manager.readline()
    second_line = global_data_manager.readline()
    
print(first_line + second_line)<jupyter_output><empty_output><jupyter_text>You may've guessed based on the extension that this is tabular data in "comma-separated values (CSV)" format. If not, then you can tell by the structure of the data we've read. Let us look at the data in more detail using a Jupyter Lab feature. Open the datafile with _CSVTable_.The [HadCRUT](https://www.metoffice.gov.uk/hadobs/hadcrut5/) website describes what we're looking at. Column,1. Year-Month labels2. 'Best-estimate' computed monthly means from 200 ensemble measurements as difference from 1961-1990 reference period.3. 2.5% confidence from 200 ensemble measurements.4. 97.5% confidence from 200 ensemble measurements.Although CSV is a conceptually simple data format it can be difficult to read and write correctly. The Python standard library provides a module (called `csv`) to help us easily read and write this format. Let's use this library to read the global data...<jupyter_code>import csv

global_data = []
with open("data/HadCRUT.5.0.1.0.analysis.summary_series.global.monthly.csv") as global_data_manager:
    reader = csv.DictReader(global_data_manager)
    
    for reading in reader:
        global_data += [reading]

global_data[:5]<jupyter_output><empty_output><jupyter_text>This is a great start. The next step is to clean these data up a little. Here are a few things we can do to make them easier to work with:1. Convert the temperature values from strings to numbers1. Use a `datetime` object to record the time rather than a string1. The confidence interval is not interesting for us at the moment so we can discard those.First, what is a `datetime` object? Python provides a module for dealing with dates, times, timezones, and differences between times in the standard library. See the documentation [here](https://docs.python.org/3/library/datetime.html). Or especially the section on format codes [here](https://docs.python.org/3/library/datetime.htmlstrftime-and-strptime-format-codes).<jupyter_code>from datetime import datetime

# An example of a date from the data is the string "1901-5"
date1 = datetime.strptime("1901-05", "%Y-%m")
print(f"Year: {date1.year}, Month: {date1.month}, Day: {date1.day}, day of the week: {date1.weekday()}") # Monday is 0
# An example of a date from the data is the string "1901-May"
date1 = datetime.strptime("1901-May", "%Y-%b")
print(f"Year: {date1.year}, Month: {date1.month}, Day: {date1.day}, day of the week: {date1.weekday()}") # Monday is 0
# An example of a parsing error:
datetime.strptime("May 1901", "%Y-%b")<jupyter_output><empty_output><jupyter_text>--- Exercise 8-1: Clean the data!Write a function to _clean_ the data we read from the file. The input to your function will be a dictionary like this:```python{  'Time': '1901-05',  'Anomaly (deg C)': '-0.24934465',  'Lower confidence limit (2.5%)': '-0.44425672',  'Upper confidence limit (97.5%)': '-0.05443258'}```The output of your function should look like this:```python{  'Time': datetime.datetime(1901, 5, 1, 0, 0),  'Temperature': -0.24934465}```<jupyter_code>def clean_global_data(reading):
    ""
    return _

assert clean_global_data({'Time': '1901-05','Anomaly (deg C)': '-0.24934465','Lower confidence limit (2.5%)': '-0.44425672','Upper confidence limit (97.5%)': '-0.05443258'}) == {'Time': datetime(1901, 5, 1, 0, 0),'Temperature': -0.24934465}<jupyter_output><empty_output><jupyter_text>--- Exercise 8-2: Load the dataWrite a function that reads the global temperature data file and returns a list of cleaned dictionaries.<jupyter_code>def global_data(filename):
    ""
    return _

assert global_data("data/HadCRUT.5.0.1.0.analysis.summary_series.global.monthly.csv")[4] == {'Time': datetime(1901, 5, 1, 0, 0),'Temperature': -0.24934465}<jupyter_output><empty_output><jupyter_text>--- 4. Saving data for laterAt this point you may wish to save your cleaned data for later inspection or use. The resource workflow of: `open`, `write`, `close` is the same as before. Let's see what this looks like in general:<jupyter_code># Open a file resource...
with open("test.txt", mode="w") as file_resource: # file_resource is a context manager. Notice the extra mode="w"
    # Write to the file resource
    file_resource.write("Hello outside world!")

# Outside the with block, the context manager has done it's job and closed the file resource.
# We cannot access file_resource now... but we still have the contents!<jupyter_output><empty_output><jupyter_text>Now you should be able to find a file called "test.txt" in the file browser on the left of this notebook.Naturally, the CSV library allows us to write correct CSV formatted data without the hastle...<jupyter_code>with open("test.csv", mode="w") as file_resource:
    writer = csv.DictWriter(file_resource, fieldnames=["Time", "Temperature"])
    writer.writeheader()
    writer.writerows(global_data("data/HadCRUT.5.0.1.0.analysis.summary_series.global.monthly.csv"))<jupyter_output><empty_output><jupyter_text>--- Exercise 8-3: Save the data!Write a function that takes a list of dictionaries as an argument and saves them to a CSV formatted file.<jupyter_code>def save_data(data, filename):
    ""
    _

glo = global_data("data/HadCRUT.5.0.1.0.analysis.summary_series.global.monthly.csv")
save_data(glo, "data/global_data.csv")<jupyter_output><empty_output><jupyter_text>--- 5. Exploring the per-Country dataDrawing on our exploration of the global data (which we had in a local file), let's explore the pre-country data. The only difference this time is that the data is coming from a web resource rather than a file resource.<jupyter_code># https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/BEL/Belgium
import csv

bel_data = []
with request.urlopen("https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/BEL/Belgium") as bel_data_manager:
    reader = csv.DictReader(bel_data_manager)
    
    for reading in reader:
        bel_data += [reading]

bel_data[:5]<jupyter_output><empty_output><jupyter_text>**An Error!** This one is a bit difficult to decipher without introducing you to another data type: `bytes`.Very often the files on your computer (or on remote computers) aren't just plain text (like CSV is), instead their data is serialised (stored) as binary numbers that have to be interpreted differently than text.* Here is a number as we would write it in base 10 (decimal): `32`.* Here is that number as it is written in a string: `"32"`.* Here is that number as a computer would write it in base 2 (binary): `00100000`* If I interpret this pattern of bytes as a character I get, `" "`* Here is that number as a computer would see the string representation without interpreting the data as text: `0011001100110010`* If I interpret this pattern of bits as a decimal number I get: `13106` At some level, all you're doing as a programmer is interpreting patterns of bytes. The data you read has no interpretation other than what you impose on in, you must tell the computer how we want to interpret it:<jupyter_code>bin(32)
int(32).to_bytes(1, byteorder='big').decode()
int.from_bytes(b"32", byteorder='big')
b'\x48\x65\x6c\x6c\x6f\x20\x57\x6f\x72\x6c\x64\x21'.decode()<jupyter_output><empty_output><jupyter_text>Hopefully this gives you some sense of what the `Error: iterator should return strings, not bytes (did you open the file in text mode?)` means. The Python `CSV` library expects text but we're trying to give it uninterpreted bytes. So the solution now seems rather trivial: `.decode()`!<jupyter_code># https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/BEL/Belgium
import csv 
bel_data = []
with request.urlopen("https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/BEL/Belgium") as bel_data_manager:
    text_data = bel_data_manager.read().decode().splitlines()
    reader = csv.DictReader(text_data, skipinitialspace=True)
    
    for reading in reader:
        bel_data += [reading]

bel_data[:5]<jupyter_output><empty_output><jupyter_text>--- Exercise 8-4: Clean the data!Write a function to clean the country data we read from the web. The input to your function will be a dictionary like this:```python{  'Temperature - (Celsius)': '5.76',  'Year': '1901',  'Statistics': 'May Average',  'Country': 'Belgium',  'ISO3': 'BEL'}```The output of your function should look like this:```python{  'Time': datetime.datetime(1901, 5, 1, 0, 0),  'Temperature': 5.76}```<jupyter_code>def clean_country_data(data):
    ""
    return _

assert clean_country_data(bel_data[0]) == {'Time': datetime(1901, 1, 1, 0, 0), 'Temperature': 4.04}<jupyter_output><empty_output><jupyter_text>--- Exercise 8-5: Load country dataWrite a function that accepts a URL and returns a list of dictionaries containing 'Time' and 'Temperature' keys. Use the second code cell to save the cleaned data using the `save_data()` function you wrote earlier.<jupyter_code>def country_data(url):
    ""
    return _

assert country_data("https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/BEL/Belgium")[4] == {'Time': datetime(1901, 5, 1, 0, 0),'Temperature': 9.15}
assert country_data("https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/RUS/Russia")[4] == {'Time': datetime(1901, 5, 1, 0, 0),'Temperature': 2.01}
assert country_data("https://climateknowledgeportal.worldbank.org/api/data/get-download-data/historical/tas/1901-2020/AUS/Australia")[4] == {'Time': datetime(1901, 5, 1, 0, 0),'Temperature': 18.02}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10284167794316644, 'fraction_numerical': 0.032025259359494816, 'mean_word_length': 4.91770462633452, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 5, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '286286', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4207, 'n_tokens_neox': 3858, 'n_words': 1573, 'path': '08_Files.ipynb', 'repo_name': 'vibbits/gentle-hands-on-python', 'size': '13302'}