<jupyter_start><jupyter_text>This notebook defines the criteria for all NOAA weather stations  Imports<jupyter_code>import os
from os import listdir
import pandas as pd
import seaborn as sns

from google.colab import drive
drive.mount('/content/drive')
%cd drive/My\ Drive/
os. getcwd()<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
[Errno 2] No such file or directory: 'drive/My Drive/'
/content/drive/My Drive<jupyter_text>Load Data  Validation of old data source to new.  Old data source had null values filled in as 0<jupyter_code>df_NOAA = pd.read_csv('UMBC_Energy/Use/All_NOAA.csv',sep='|')
df_NOAA
df_NOAA['state_station_dt'] = df_NOAA['state'] + df_NOAA['station'] + df_NOAA['date']
df_NOAA
df_NOAA.nunique()<jupyter_output><empty_output><jupyter_text>New Data Source for Weather Stations<jupyter_code>df_NOAA_Raw_2020 = pd.read_csv('UMBC_Energy/Raw/Raw_2020_NOAA.csv',sep='|')
df_NOAA_Raw_2019 = pd.read_csv('UMBC_Energy/Raw/Raw_2019_NOAA.csv',sep='|')
df_NOAA_Raw_2018 = pd.read_csv('UMBC_Energy/Raw/Raw_2018_NOAA.csv',sep='|')
df_NOAA_Raw_2017 = pd.read_csv('UMBC_Energy/Raw/Raw_2017_NOAA.csv',sep='|')
df_NOAA_Raw_2016 = pd.read_csv('UMBC_Energy/Raw/Raw_2016_NOAA.csv',sep='|')
df_NOAA_Raw_2015 = pd.read_csv('UMBC_Energy/Raw/Raw_2015_NOAA.csv',sep='|')
df_NOAA_Raw = pd.concat([df_NOAA_Raw_2020,df_NOAA_Raw_2019,df_NOAA_Raw_2018,df_NOAA_Raw_2017,df_NOAA_Raw_2016,df_NOAA_Raw_2015])
df_NOAA_Raw = df_NOAA_Raw.drop_duplicates('station_dt_key')
df_NOAA_Raw.nunique()<jupyter_output><empty_output><jupyter_text>Load dataset that has the state of each weather station<jupyter_code>df_station_details = pd.read_csv('UMBC_Energy/Raw/station_details.csv',sep='|')
df_station_details
df_station_details.nunique()<jupyter_output><empty_output><jupyter_text>Some ID's appeared in multiple states so had to keep one of them.  There wasn't a value gain or loss by dropping duplicates after the first id was found<jupyter_code>ids = df_station_details["id"]
df_station_details[ids.isin(ids[ids.duplicated()])].sort_values("id").reset_index()
df_station_details = df_station_details.drop_duplicates('id')
df_station_details<jupyter_output><empty_output><jupyter_text>Merge to pair station with State<jupyter_code>df_NOAA_Raw.merge(df_station_details,left_on='station',right_on='id',how='left').sort_values('date')
df_NOAA_Raw.merge(df_station_details,left_on='station',right_on='id',how='left').sort_values('date').reset_index()
df_NOAA_Raw_merge = df_NOAA_Raw.merge(df_station_details,left_on='station',right_on='id',how='left').sort_values('date').reset_index().drop(columns=['index'])<jupyter_output><empty_output><jupyter_text>**Load All State and Abbreviations**<jupyter_code>df_state_abbv = pd.read_csv('UMBC_Energy/Universal_Data/state_abb.csv',sep=',')
df_state_abbv = df_state_abbv.drop(columns='Abbrev')
df_state_abbv.head()<jupyter_output><empty_output><jupyter_text>Merge two datasets to get state abbreviations<jupyter_code>df_NOAA_merge_state = df_NOAA_Raw_merge.merge(df_state_abbv,left_on = 'state', right_on = 'State',how='left')
df_NOAA_merge_state
#df_NOAA_merge_state.to_csv('UMBC_Energy/Raw/Raw_All_NOAA.csv',sep='|',index=False)
df_NOAA_merge_state.groupby('Code').count()<jupyter_output><empty_output><jupyter_text>** Below: Selection Criteria = Sort on amount of occurences.  Search for stations with all features filled with values similar to top occurence stations.  More features with a value that is similar to the station who appear most frequent is preferred and selected for analysis**  DC<jupyter_code>#Selected Top 2 selected
df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "DC"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>DE<jupyter_code>#Selected First 2 and Tail id
df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "DE"').groupby('id').count().sort_values('station',ascending=False).head(9)<jupyter_output><empty_output><jupyter_text>MD<jupyter_code>#Selected Top 5
df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "MD"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>NJ<jupyter_code>#Selected Top 5

df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "NJ"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>OH<jupyter_code>#Selected Top 5
df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "OH"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>PA<jupyter_code># Selected 
df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "PA"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>VA<jupyter_code>df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "VA"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output><jupyter_text>WV<jupyter_code>df_NOAA_state_group = df_NOAA_merge_state[df_NOAA_merge_state.datacoverage >= .99]

df_NOAA_state_group.query('Code == "WV"').groupby('id').count().sort_values('station',ascending=False).head(6)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13674321503131523, 'fraction_numerical': 0.01791927627000696, 'mean_word_length': 10.927385892116183, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '63078', 'license': 'MIT', 'n_tokens_mistral': 2295, 'n_tokens_neox': 2184, 'n_words': 275, 'path': 'WeatherStation_Criteria_RawAllNOAA.ipynb', 'repo_name': 'VintageGold/Energy-Grid', 'size': '5748'}