<jupyter_start><jupyter_text>Weather Data CollectionThe following code was used to collect the weather data from the [NCDC API](https://www.ncdc.noaa.gov/cdo-web/webservices/v2). The API requires a token, which should be provided in place of `PASTE_YOUR_TOKEN_HERE` in the `make_request()` function below:<jupyter_code>import datetime as dt
import requests


STATIONS = {
    'GHCND:USW00023174': 'LA',
    'GHCND:USW00014732': 'NYC',
    'GHCND:USW00023234': 'SF',
    'GHCND:USW00012960': 'Houston',
    'GHCND:USW00013874': 'Atlanta',
    'GHCND:USW00094846': 'Chicago',
    'GHCND:USW00014739': 'Boston',
    'GHCND:USW00012839': 'Miami',
    'GHCND:USW00024233': 'Seattle',
    'GHCND:USW00023183': 'Phoenix',
    'GHCND:USW00022521': 'Honolulu'
}

def make_request(endpoint, payload=None):
    """
    Make a request to a specific endpoint on the weather API
    passing headers and optional payload.
    
    Parameters:
        - endpoint: The endpoint of the API you want to 
                    make a GET request to.
        - payload: A dictionary of data to pass along 
                   with the request.
    
    Returns:
        A response object.
    """
    return requests.get(
        f'https://www.ncdc.noaa.gov/cdo-web/api/v2/{endpoint}',
        headers={
            'token': 'PASTE_YOUR_TOKEN_HERE'
        },
        params=payload
    )

start = current = dt.date(2020, 1, 1)
end = dt.date(2021, 1, 1)

results = []

while current < end:
    # update the cell with status information
    print(f'\rGathering data for {str(current)}', end='')
    response = make_request(
        'data', 
        {
            'datasetid': 'GHCND',
            'stationid': STATIONS.keys(),
            'startdate': current,
            'enddate': current,
            'datatypeid': [
                'TAVG', 'TMAX', 'TMIN', 
                'SNOW', 'PRCP', 
                'AWND', 'TSUN',
                'ACSC', 'SCSH', 'PSUN'
            ],
            'units': 'standard',
            'limit': 1000
        }
    )

    if response.ok:
        # we extend the list instead of appending to avoid getting a nested list
        results.extend(response.json()['results'])

    # update the current date to avoid an infinite loop
    current += dt.timedelta(days=1)
print('\nDone')<jupyter_output>Gathering data for 2020-12-31
Done<jupyter_text>Generating the `weather.csv` fileThe data collected from the API looks like this:<jupyter_code>import pandas as pd
weather = pd.DataFrame(results)
weather.head()<jupyter_output><empty_output><jupyter_text>We have several different weather observations:<jupyter_code>weather.datatype.unique()<jupyter_output><empty_output><jupyter_text>We will pivot this information and convert station IDs to city names, so the data is easier to work with:<jupyter_code>weather = weather.assign(
    date=lambda x: pd.to_datetime(x.date), 
    city=lambda x: pd.Series(STATIONS).loc[x.station,].values
).drop(columns=['attributes', 'station']).pivot(
    index=['date', 'city'], columns='datatype', values='value'
).reset_index().set_index('date')
weather.columns = weather.columns.rename('')

weather<jupyter_output><empty_output><jupyter_text>Save to a CSV file:<jupyter_code>weather.to_csv('../weather.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12804878048780488, 'fraction_numerical': 0.0350609756097561, 'mean_word_length': 3.10125, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '338657', 'license': 'MIT', 'n_tokens_mistral': 1187, 'n_tokens_neox': 1076, 'n_words': 262, 'path': 'data/collection/weather.ipynb', 'repo_name': 'Heidi-/python-data-viz-workshop', 'size': '3280'}