<jupyter_start><jupyter_text>Course 601: Introduction to Data Science Course Project Topic: Weather Forecast Members:    - Professor: <NAME>    - Students:    <NAME>    <NAME>    <NAME>    <NAME>    <NAME>    <NAME>_________________________________________________________________________________  Project Task List- [x] Find Weather Dataset (KBWI airport, Maryland): https://www.ncdc.noaa.gov/cdo-web/datatools/lcd - **Levan**- [x] Merge Datasets - **Karteek**- [x] Fix Dataset column's data types using Excel - **Levan** - [x] Remove unnecessary columns and clean data type mixes in columns (two data types per column -> fix to single data type for consistency) - **Levan**- [ ] NaN values interpolation - - [ ] Visualize weather dataset ( feature insights, etc.) - (2 graphs from each member)    - [ ] **Sarang**    - [ ] **Aishwarya**    - [ ] **Spandana**    - [ ] **Karteek**    - [ ] **Suvidha**    - [ ] **Levan**- [ ] Visualization of correlation between features - **Sarang**  - [ ] Data Distribution - **Spandana, Suvidha, Aishwarya**- [ ] Correlate current data with POWER.LARC NASA's dataset (https://power.larc.nasa.gov/data-access-viewer/) - **Levan**- [ ] Split data for train, test, validate (7 days to predict next 7 days - needs group discussion)- [ ] Train Data using Logistic Regression- [ ] Evaluate LogReg model (precision, recall, etc)- [ ] Train Data using Random Forest- [ ] Evaluate RFR model (precision, recall, etc)- [ ] Train Data using RNN- [ ] Evaluate RFR model (precision, recall, etc)- [ ] **Paper**    - [ ] Abstract - **Aishwarya**    - [ ] Introduction -     - [ ] Previous Approaches    - [ ] Our changes/approach (description of each approach)    - [ ] Results    - [ ] Plots  Project implementation:  Module Imports<jupyter_code>import os
import pandas as pd
import numpy as np
import statistics<jupyter_output><empty_output><jupyter_text>Steps to merge the data and set column data types (dtypes)<jupyter_code># We provide merged file at the end, so this section is commented out

# Import initial data and check imported data

# # get current directory
# curr_dir = os.getcwd()

# data_dir = os.path.join(curr_dir, "main_data")

# # dataset name
# data1 = "1991_2000.csv"
# data2 = "2001_2010.csv"
# data3 = "2011_2020.csv"

# df1_dir = os.path.join(data_dir, data1)
# df2_dir = os.path.join(data_dir, data2)
# df3_dir = os.path.join(data_dir, data3)

# df1 = pd.read_csv(df1_dir) # dtype={0:str, 1:str, 2:str, 3:str, 4:int, })
# df2 = pd.read_csv(df2_dir)
# df3 = pd.read_csv(df3_dir)

# df_parts = [df1, df2, df3]
# merged_data = pd.concat(df_parts)
# merged_data.head(5)<jupyter_output><empty_output><jupyter_text>We see that a lot of values are labeled as NaN We have found that this happens due to unknown data type per column, and this can be fixed using manual set of dtype per column In the Excel, each column was investigated and assigned appropriate data type: integer, float (2 decimal), text<jupyter_code># Following is the list of column names from merged csv
'''
['STATION':str,
 'DATE':str,
 'REPORT_TYPE':str,
 'SOURCE':str,
 'AWND',
 'BackupDirection',
 'BackupDistance',
 'BackupDistanceUnit',
 'BackupElements',
 'BackupElevation',
 'BackupElevationUnit',
 'BackupEquipment',
 'BackupLatitude',
 'BackupLongitude',
 'BackupName',
 'CDSD',
 'CLDD',
 'DSNW',
 'DailyAverageDewPointTemperature',
 'DailyAverageDryBulbTemperature',
 'DailyAverageRelativeHumidity',
 'DailyAverageSeaLevelPressure',
 'DailyAverageStationPressure',
 'DailyAverageWetBulbTemperature',
 'DailyAverageWindSpeed',
 'DailyCoolingDegreeDays',
 'DailyDepartureFromNormalAverageTemperature',
 'DailyHeatingDegreeDays',
 'DailyMaximumDryBulbTemperature',
 'DailyMinimumDryBulbTemperature',
 'DailyPeakWindDirection',
 'DailyPeakWindSpeed',
 'DailyPrecipitation',
 'DailySnowDepth',
 'DailySnowfall',
 'DailySustainedWindDirection',
 'DailySustainedWindSpeed',
 'DailyWeather',
 'HDSD',
 'HTDD',
 'HeavyFog',
 'HourlyAltimeterSetting',
 'HourlyDewPointTemperature',
 'HourlyDryBulbTemperature',
 'HourlyPrecipitation',
 'HourlyPresentWeatherType',
 'HourlyPressureChange',
 'HourlyPressureTendency',
 'HourlyRelativeHumidity',
 'HourlySeaLevelPressure',
 'HourlySkyConditions',
 'HourlyStationPressure',
 'HourlyVisibility',
 'HourlyWetBulbTemperature',
 'HourlyWindDirection',
 'HourlyWindGustSpeed',
 'HourlyWindSpeed',
 'MonthlyAverageRH',
 'MonthlyDaysWithGT001Precip',
 'MonthlyDaysWithGT010Precip',
 'MonthlyDaysWithGT32Temp',
 'MonthlyDaysWithGT90Temp',
 'MonthlyDaysWithLT0Temp',
 'MonthlyDaysWithLT32Temp',
 'MonthlyDepartureFromNormalAverageTemperature',
 'MonthlyDepartureFromNormalCoolingDegreeDays',
 'MonthlyDepartureFromNormalHeatingDegreeDays',
 'MonthlyDepartureFromNormalMaximumTemperature',
 'MonthlyDepartureFromNormalMinimumTemperature',
 'MonthlyDepartureFromNormalPrecipitation',
 'MonthlyDewpointTemperature',
 'MonthlyGreatestPrecip',
 'MonthlyGreatestPrecipDate',
 'MonthlyGreatestSnowDepth',
 'MonthlyGreatestSnowDepthDate',
 'MonthlyGreatestSnowfall',
 'MonthlyGreatestSnowfallDate',
 'MonthlyMaxSeaLevelPressureValue',
 'MonthlyMaxSeaLevelPressureValueDate',
 'MonthlyMaxSeaLevelPressureValueTime',
 'MonthlyMaximumTemperature',
 'MonthlyMeanTemperature',
 'MonthlyMinSeaLevelPressureValue',
 'MonthlyMinSeaLevelPressureValueDate',
 'MonthlyMinSeaLevelPressureValueTime',
 'MonthlyMinimumTemperature',
 'MonthlySeaLevelPressure',
 'MonthlyStationPressure',
 'MonthlyTotalLiquidPrecipitation',
 'MonthlyTotalSnowfall',
 'MonthlyWetBulb',
 'NormalsCoolingDegreeDay',
 'NormalsHeatingDegreeDay',
 'REM',
 'REPORT_TYPE.1',
 'SOURCE.1',
 'ShortDurationEndDate005',
 'ShortDurationEndDate010',
 'ShortDurationEndDate015',
 'ShortDurationEndDate020',
 'ShortDurationEndDate030',
 'ShortDurationEndDate045',
 'ShortDurationEndDate060',
 'ShortDurationEndDate080',
 'ShortDurationEndDate100',
 'ShortDurationEndDate120',
 'ShortDurationEndDate150',
 'ShortDurationEndDate180',
 'ShortDurationPrecipitationValue005',
 'ShortDurationPrecipitationValue010',
 'ShortDurationPrecipitationValue015',
 'ShortDurationPrecipitationValue020',
 'ShortDurationPrecipitationValue030',
 'ShortDurationPrecipitationValue045',
 'ShortDurationPrecipitationValue060',
 'ShortDurationPrecipitationValue080',
 'ShortDurationPrecipitationValue100',
 'ShortDurationPrecipitationValue120',
 'ShortDurationPrecipitationValue150',
 'ShortDurationPrecipitationValue180',
 'Sunrise',
 'Sunset',
 'TStorms',
 'WindEquipmentChangeDate']
 '''
None
# Let's save merged data into CSV
#merged_data.to_csv("./main_data/merged_data.csv", index=False)
# merged_data.shape is (408747, 125)
# A lot of columns are not needed for our research, such as SOURCE.1 and Backup columns, together with ShortDuration data
# So we are going to discard it manually via Excel
# The average (daily and monthly) also can be discarded, since we have hourly data, thus we can calculate it on the go if needed

# Reason why we change data type manually, is that we need to look into actual values, whereas here, values are still shown as NaN
# But in the excel sheet, they are clearly visible, please see excel image attached.<jupyter_output><empty_output><jupyter_text>Sample images from excel analysis:![alt text](excel_analysis_sample/column_data_type_adjustment.PNG "Fixing Data Type")![alt text](excel_analysis_sample/cleaning_extra_strings_in_numbers.PNG "Removing unnecessary strings from numbers")<jupyter_code># Next we going to work on the NaN values conversion to actual values (set column's data types to their actual type)
# Removed columns:
'''
* Unnamed
* All "Backup..." columns
* All Daily Average
* All Monthly Average
* AWND
* CDSD
* CLDD
* DSNW
* HDSD
* HTDD
* HeavyFog
* TStorms
* WindEquipmentChangeDate
* SOURCE.1
* All "ShortDuration..."
* TStorms
* WindEquipmentChangeDate
* NormalsCoolingDegreeDay
* NormalsHeatingDegreeDay
'''
# list(merged_data.columns.values)
# print(len(list(weather_df.columns.values)))
# New number of columns is 87
# Columns that we get after removing unnecessary columns
# Let's label expected data type for each
'''
columns_dict = {'STATION':str,
 'DATE':str,
 'REPORT_TYPE':str,
 'SOURCE':str,
 'HourlyAltimeterSetting':float,
 'HourlyDewPointTemperature':int,
 'HourlyDryBulbTemperature':int,
 'HourlyPrecipitation':float,
 'HourlyPresentWeatherType':str,
 'HourlyPressureChange':float,
 'HourlyPressureTendency':float,
 'HourlyRelativeHumidity':float,
 'HourlySeaLevelPressure':float,
 'HourlySkyConditions':str,
 'HourlyStationPressure':float,
 'HourlyVisibility':float,
 'HourlyWetBulbTemperature':int,
 'HourlyWindDirection':str,
 'HourlyWindGustSpeed':float,
 'HourlyWindSpeed':float,
 'REM':str,
 'REPORT_TYPE.1':str,
 'Sunrise':str,
 'Sunset':str}
'''
None
# Finally, after setting all data types for all columns using excel, we can now import it into our notebook
# get current directory
curr_dir = os.getcwd()
data_folder = "main_data"
dataset_name = "merged_data_types_defined.csv"
dataset_location = os.path.join(curr_dir, data_folder, dataset_name)

pd.options.display.max_columns = None
weather_df = pd.read_csv(dataset_location, low_memory=False)
weather_df.head(5)
weather_df.shape
'''
Columns: 
    - Hourly Present Weather Type (HourlyPresentWeatherType)
    - Hourly Sky Conditions (HourlySkyConditions)
    - Hourly Wind Gust Speed (HourlyWindGustSpeed)
    - Sunrise (Sunrise)
    - Sunset (Sunset)
    - REM (REM)
    
They all seem to have a lot of NaN values, so let's check if that's true and if it is, we are going to discard them
'''
None
number_of_rows = weather_df.shape[0]

print("NaN occurence in HourlyPresentWeatherType: {} / {} => {}%".format(weather_df['HourlyPresentWeatherType'].isna().sum(), number_of_rows, round(weather_df['HourlyPresentWeatherType'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in HourlySkyConditions: {} / {} => {}%".format(weather_df['HourlySkyConditions'].isna().sum(), number_of_rows, round(weather_df['HourlySkyConditions'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in HourlyWindGustSpeed: {} / {} => {}%".format(weather_df['HourlyWindGustSpeed'].isna().sum(), number_of_rows, round(weather_df['HourlyWindGustSpeed'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in Sunrise: {} / {} => {}%".format(weather_df['Sunrise'].isna().sum(), number_of_rows, round(weather_df['Sunrise'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in Sunset: {} / {} => {}%".format(weather_df['Sunset'].isna().sum(), number_of_rows, round(weather_df['Sunset'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in REM: {} / {} => {}%".format(weather_df['REM'].isna().sum(), number_of_rows, round(weather_df['REM'].isna().sum()/number_of_rows*100, 2)))

# HourlyPressureChange HourlyPressureTendency
print("NaN occurence in HourlyPressureChange: {} / {} => {}%".format(weather_df['HourlyPressureChange'].isna().sum(), number_of_rows, round(weather_df['HourlyPressureChange'].isna().sum()/number_of_rows*100, 2)))
print("NaN occurence in HourlyPressureTendency: {} / {} => {}%".format(weather_df['HourlyPressureTendency'].isna().sum(), number_of_rows, round(weather_df['HourlyPressureTendency'].isna().sum()/number_of_rows*100, 2)))
# Most of these columns miss significant number of data and do not play most important role in the analysis.
# Hourly Present Weater Type could be important, 
# but since we have other types of data (Temp, Humidity, Precip, etc.) we can discard it
del weather_df['HourlyPresentWeatherType']
del weather_df['HourlySkyConditions']
del weather_df['HourlyWindGustSpeed']
del weather_df['Sunrise']
del weather_df['Sunset']
del weather_df['REM']

del weather_df['HourlyPressureChange']
del weather_df['HourlyPressureTendency']
weather_df.head(10)
# Now let us see NaN distribution among important columns that we have kept
# And if NaN exists, proceed with interpolation
col_list = list(weather_df.columns.values)
number_of_rows = weather_df.shape[0]

print("NaN occurrences in following columns:")
for c in col_list:
    nan_sum = weather_df[c].isna().sum()
    nan_percentage = round(nan_sum/number_of_rows*100, 2)
    print("    *{}: => {}%".format(c, nan_percentage))
# Alright, ~35% max of missing values, interpolation may help us in current pre-process 
column_name = "HourlyAltimeterSetting"

weather_df['DATE'].head(5)
# sample date
weather_df['DATE'].iloc[1][5:13]
# columns with float dtype
float_columns = ["HourlyAltimeterSetting", "HourlyDewPointTemperature", 
                 "HourlyDryBulbTemperature", "HourlyPrecipitation", 
                 "HourlyRelativeHumidity", "HourlySeaLevelPressure", 
                 "HourlyStationPressure", "HourlyVisibility", 
                 "HourlyWetBulbTemperature", "HourlyWindDirection"]
for col in float_columns:
    weather_df[col].astype(float)
# Columns with consecutive NaNs
'''
{'HourlyAltimeterSetting',
 'HourlyDewPointTemperature',
 'HourlyDryBulbTemperature',
 'HourlyPrecipitation',
 'HourlyRelativeHumidity',
 'HourlyVisibility',
 'HourlyWetBulbTemperature',
 'HourlyWindDirection'}
'''
None
# find NaN
# get its row number
# get its date
# gather all similar dates, ignoring year
# print it
# average it
# put that average value into that NaN

weather_df_original = weather_df.copy(deep=True)
weather_df_original.to_csv("with_nan.csv")
weather_df_original.head(3)
weather_df_original.keys()
weather_df_w_Nan = pd.read_csv("with_nan.csv", low_memory=False)

col_list = list(weather_df_w_Nan.columns.values)
number_of_rows = weather_df_w_Nan.shape[0]

col_set_with_consecutive_nan = set()

for c in col_list:
    print("Going over column", c)
    if c in float_columns:
        for r in range(0, number_of_rows):
            if r+1 < number_of_rows:
                first_val = weather_df_w_Nan[c][r]
                second_val = weather_df_w_Nan[c][r+1]
                #print(first_val, second_val)
                #print(type(first_val), type(second_val))
                # consecutive case of NaN values
                # collect similar days from other years and replace it with average:
                if((np.isnan(first_val)) and (np.isnan(second_val))):
                    print("Consecutive case")
                    # r - row number
                    # date
                    correlated_date = weather_df_w_Nan["DATE"][r][5:13]
                    # get all similar rows with date
                    lst_of_correlated_days = []
                    for i in range(0, number_of_rows):
                        if weather_df_w_Nan["DATE"][i][5:13] == correlated_date:
                            if not np.isnan(weather_df_w_Nan[c][i]):
                                lst_of_correlated_days.append(weather_df_w_Nan[c][i])
#                             elif np.isnan(weather_df_w_Nan[c][i]):
#                                 try:
#                                     avg = (weather_df_w_Nan[c][i-1]+weather_df_w_Nan[c][i+1])/2
#                                     lst_of_correlated_days.append(avg)
#                                 except:
#                                     print("errored out")
#                                     pass
                                
                    if lst_of_correlated_days == []:
                        print("WARNING, ASSOCIATED DATA IS MISSING:")
                    else:
                        # SET
                        #weather_df_w_Nan[c][r] = statistics.mean(lst_of_correlated_days)
                        #print("Was: ", weather_df_w_Nan[c][r])
                        #print(statistics.mean(lst_of_correlated_days))
                        #print(lst_of_correlated_days)
                        weather_df_w_Nan.loc[ r, c ] = round(float(statistics.mean(lst_of_correlated_days)), 2)
                        #print("Became: ", weather_df_w_Nan[c][r])
                # replace NaN values with average (non-consecutive case)
                elif((np.isnan(first_val)) and (not np.isnan(second_val))):
                    #print("Non-consecutive case")
                    if r == 0:
                        # SET
                        indx = str(r)
                        #print("Was: ", weather_df_w_Nan[c][r])
                        weather_df_w_Nan.loc[ r, c ] = weather_df_w_Nan[c][r+1]
                        #weather_df_w_Nan[c][r] = weather_df_w_Nan[c][r+1]
                        #print("Became: ", weather_df_w_Nan[c][r])
                    else:
                        indx = str(r)
                        # SET
                        #print("Was: ", weather_df_w_Nan[c][r])
                        #weather_df_w_Nan[c][r] = (weather_df_w_Nan[c][r-1]+weather_df_w_Nan[c][r+1])/2
                        weather_df_w_Nan.loc[ r, c ] = round(float((weather_df_w_Nan[c][r-1]+weather_df_w_Nan[c][r+1])/2), 2)
                        #print("Became: ", weather_df_w_Nan[c][r])
                else:
                    pass
col_list = list(weather_df.columns.values)
number_of_rows = weather_df.shape[0]

print("NaN occurrences in following columns:")
for c in col_list:
    nan_sum = weather_df[c].isna().sum()
    nan_percentage = round(nan_sum/number_of_rows*100, 2)
    print("    *{}: {}/{} => {}%".format(c, nan_sum, number_of_rows, nan_percentage))<jupyter_output>NaN occurrences in following columns:
    *STATION: 0/408747 => 0.0%
       * int64
    *DATE: 0/408747 => 0.0%
       * object
    *REPORT_TYPE: 0/408747 => 0.0%
       * object
    *SOURCE: 0/408747 => 0.0%
       * object
    *HourlyAltimeterSetting: 94241/408747 => 23.06%
       * float64
    *HourlyDewPointTemperature: 74217/408747 => 18.16%
       * float64
    *HourlyDryBulbTemperature: 74179/408747 => 18.15%
       * float64
    *HourlyPrecipitation: 126307/408747 => 30.9%
       * float64
    *HourlyRelativeHumidity: 74222/408747 => 18.16%
       * float64
    *HourlySeaLevelPressure: 111310/408747 => 27.23%
       * float64
    *HourlyStationPressure: 142701/408747 => 34.91%
       * float64
    *HourlyVisibility: 70211/408747 => 17.18%
       * float64
    *HourlyWetBulbTemperature: 142718/408747 => 34.92%
       * float64
    *HourlyWindDirection: 85277/408747 => 20.86%
       * float64
    *HourlyWindSpeed: 70837/408747 => 17.33%
       * float64
    *REPORT_TYPE.1: 0/4087[...]
{'fraction_non_alphanumeric': 0.12077957727147955, 'fraction_numerical': 0.025748009881965413, 'mean_word_length': 3.9702592087312416, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 49, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '433234', 'license': 'MIT', 'n_tokens_mistral': 6497, 'n_tokens_neox': 5866, 'n_words': 1251, 'path': '.ipynb_checkpoints/Weather_Forecast-checkpoint.ipynb', 'repo_name': 'LevanSulimanov/Weather_Forecasting', 'size': '18293'}