<jupyter_start><jupyter_text>Create Repository Structure to Download the Data from NOAA API *This notebook creates a dictionary of project and weather stations, directories in which weather data will be downloaded*  Tasks1. All folders in data directory.2. A project dictionary where keys are project names and values are a list of corresponding stations. The stations are in ascending order by distance to project center.3. A station dictionary where keys are stations names and values are dataframe with corresponding data.<jupyter_code>import pandas as pd
import numpy as np
import pickle
import os
all_project_stations = pd.read_csv('./data/all_project_stations.csv')
usgs_data = pd.read_csv('./uswtdbCSV/uswtdb_v1_1_20180710.csv')
usgs_ge_data = usgs_data[usgs_data["t_manu"] == "GE Wind"]
usgs_ge_data = usgs_ge_data.replace(to_replace=-9999, value=np.nan)
usgs_ge_data["p_name"] = usgs_ge_data["p_name"].str.replace("/","&", regex=False)

# A dictionary with project names as keys and a list of corresponding stations.
# The stations are in ascending order by distance to project center.
project_dict = dict()

# A dictionary with station names as keys and dataframe as values.
station_dict = dict()<jupyter_output><empty_output><jupyter_text>Create directories for each project<jupyter_code>projects = usgs_ge_data["p_name"].unique()

for project in projects:
    try:
        os.mkdir("./data/weather/projects/" + project_name)
    except:
        print("Project folder already exists.")<jupyter_output>Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project folder already exists.
Project [...]<jupyter_text>Creating folders for each station in each project.<jupyter_code>for index, row in all_project_stations.iterrows():
    stationid = row['id']
    stationid_dir_name = stationid.replace(":", "_")
    projectid = row["p_name"]
    try:
        os.mkdir("./data/weather/projects/" + projectid + "/" + stationid_dir_name)
    except:
        print("Station folder already exists.")<jupyter_output>Station folder already exists.
Station folder already exists.
Station folder already exists.
Station folder already exists.
Station folder already exists.
Station folder already exists.<jupyter_text>Creating project dictionary:<jupyter_code>for project in projects:
    project_dict[project] = []
    project_stations = all_project_stations[all_project_stations["p_name"] == project]
    project_stations_sorted = project_stations.sort_values(by="dist_from_center")
    for index, row in project_stations_sorted.iterrows():
        project_dict[project].append(row["id"])<jupyter_output><empty_output><jupyter_text>Saving project dictionary to pickle file.<jupyter_code>pickle_out = open("./data/project_dict.pickle", "wb")
pickle.dump(project_dict, pickle_out)
pickle_out.close()
project_dict["Thunder Ranch"]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08025511559925591, 'fraction_numerical': 0.00451767207015679, 'mean_word_length': 5.546086956521739, 'pattern_counts': {'":': 1, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '91335', 'license': 'MIT', 'n_tokens_mistral': 1044, 'n_tokens_neox': 1019, 'n_words': 364, 'path': '02b - Preprocessing and Create Directory Structure.ipynb', 'repo_name': 'aaj2146/WindLSTM', 'size': '3763'}