<jupyter_start><jupyter_text>Data cleaning and feature engineering___________________________________________________________________________________________  ProblemSolar energy is a renewable energy source that depends on the irradiation data parameters to be efficient. Therefore, before investing in a new solar plant, it is necessary to gather the most solar data possible.  Acquisition of data 1) Process a "raw" satellite image of the location.2) Use a collection of past satellite images at that location to create a "background" 3) Determine the cloud opacity - Decomposition of cloudy and cloud-free regions. Then, apply a 3D decomposition model to cloudy regions to characterize the thickness of each layer to sunlight.4) Apply a modified version of the REST2 clear-sky radiation model, that allows us to use the latest global aerosol (dust, salt, smoke, etc.) and water vapor content to generate precise estimates of the solar radiation available to cloud-free regions. Purpose:Using the data collected, we want to predict solar irradiance. For more information on the data acquisition process, please head over to https://solcast.com/solar-radiation-data/inputs-and-algorithms/!  Import Libraries<jupyter_code># Import Libraries
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in Dataframe<jupyter_code># Read in dataframe using pandas
df = pd.read_csv('../datasets/raw/solar_ir_abj_60M_ny.csv')<jupyter_output><empty_output><jupyter_text>Dataframe column names<jupyter_code># Print the column names
print(df.columns)

# Print the number of columns
print(len(df.columns))<jupyter_output>Index(['PeriodEnd', 'PeriodStart', 'Period', 'AirTemp', 'Azimuth',
       'CloudOpacity', 'DewpointTemp', 'Dhi', 'Dni', 'Ebh', 'Ghi',
       'GtiFixedTilt', 'GtiTracking', 'PrecipitableWater', 'RelativeHumidity',
       'SnowWater', 'SurfacePressure', 'WindDirection10m', 'WindSpeed10m',
       'Zenith', 'AlbedoDaily'],
      dtype='object')
21<jupyter_text>We have 21 columns.  Feature selection<jupyter_code># Feature selection
df = df[['PeriodEnd', 'PeriodStart', 'AirTemp', 'Dhi', 'Dni', 'Ghi',
       'PrecipitableWater', 'RelativeHumidity', 'SurfacePressure',
       'WindDirection10m', 'WindSpeed10m']]<jupyter_output><empty_output><jupyter_text>Label each sample with their respective periods and important weather data (selected by a subject matter expert)  Randomly display 10 rows<jupyter_code># Randomly display 10 rows
df.sample(10)<jupyter_output><empty_output><jupyter_text>Relabel column names<jupyter_code># Rename column names
old_names = ['PeriodEnd', 'PeriodStart', 'AirTemp', 'Dhi', 'Dni', 'Ghi','PrecipitableWater', 'RelativeHumidity',
             'SurfacePressure', 'WindDirection10m', 'WindSpeed10m'] 
new_names = ['PeriodEnd','PeriodStart','Temperature', 'DHI', 'DNI', 'Radiation', 'Precipitation', 'Humidity', 
             'Pressure', 'WindDirection', 'WindSpeed']
df.rename(columns = dict(zip(old_names, new_names)), inplace = True)<jupyter_output><empty_output><jupyter_text>Display dataframe<jupyter_code># Display dataframe
df.head()<jupyter_output><empty_output><jupyter_text>Display a summary of the dataframe (show the datatype of each feature)<jupyter_code># Summary of dataframe
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 131421 entries, 0 to 131420
Data columns (total 21 columns):
 #   Column         Non-Null Count   Dtype  
---  ------         --------------   -----  
 0   PeriodEnd      131421 non-null  object 
 1   PeriodStart    131421 non-null  object 
 2   Period         131421 non-null  object 
 3   Temperature    131421 non-null  float64
 4   Azimuth        131421 non-null  int64  
 5   CloudOpacity   131421 non-null  float64
 6   DewpointTemp   131421 non-null  float64
 7   DHI            131421 non-null  int64  
 8   DNI            131421 non-null  int64  
 9   Ebh            131421 non-null  int64  
 10  Radiation      131421 non-null  int64  
 11  GtiFixedTilt   131421 non-null  int64  
 12  GtiTracking    131421 non-null  int64  
 13  Precipitation  131421 non-null  float64
 14  Humidity       131421 non-null  float64
 15  SnowWater      131421 non-null  float64
 16  Pressure       131421 non-null  float64
 17  WindDirection  131421 non-nul[...]<jupyter_text>Convert data types for prediction<jupyter_code># Convert features to appropriate datatypes for predictions
df['Radiation']     = df['Radiation'].astype(float)
df['Temperature']   = df['Temperature'].astype(float) # or int
df['Pressure']      = df['Pressure'].astype(float)
df['Humidity']      = df['Humidity'].astype(int) # or int
df['WindDirection'] = df['WindDirection'].astype(float)
df['WindSpeed']     = df['WindSpeed'].astype(float)
df['PeriodStart'] = pd.to_datetime(df['PeriodStart']).dt.to_period('T').dt.to_timestamp()
df['PeriodEnd'] = pd.to_datetime(df['PeriodEnd']).dt.to_period('T').dt.to_timestamp()<jupyter_output><empty_output><jupyter_text>Display dataframe<jupyter_code># Summary of dataframe
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 131421 entries, 0 to 131420
Data columns (total 21 columns):
 #   Column         Non-Null Count   Dtype         
---  ------         --------------   -----         
 0   PeriodEnd      131421 non-null  datetime64[ns]
 1   PeriodStart    131421 non-null  datetime64[ns]
 2   Period         131421 non-null  object        
 3   Temperature    131421 non-null  float64       
 4   Azimuth        131421 non-null  int64         
 5   CloudOpacity   131421 non-null  float64       
 6   DewpointTemp   131421 non-null  float64       
 7   DHI            131421 non-null  int64         
 8   DNI            131421 non-null  int64         
 9   Ebh            131421 non-null  int64         
 10  Radiation      131421 non-null  float64       
 11  GtiFixedTilt   131421 non-null  int64         
 12  GtiTracking    131421 non-null  int64         
 13  Precipitation  131421 non-null  float64       
 14  Humidity       131421 non-null  int32         
 15  [...]<jupyter_text>Display top 6 rows<jupyter_code># Display top 6 rows
df.head(6)<jupyter_output><empty_output><jupyter_text>Create new features - year, month, and day<jupyter_code># Create new features(year, month, day) from start and end time of data collection  
df['YearPS'] = pd.to_datetime(df['PeriodStart']).dt.year
df['YearPE'] = pd.to_datetime(df['PeriodEnd']).dt.year

df['MonthPS'] = pd.to_datetime(df['PeriodStart']).dt.month
df['MonthPE'] = pd.to_datetime(df['PeriodEnd']).dt.month

df['HourPS'] = pd.to_datetime(df['PeriodStart']).dt.hour
df['HourPE'] = pd.to_datetime(df['PeriodEnd']).dt.hour<jupyter_output><empty_output><jupyter_text>Display top 5 samples<jupyter_code># Display the top 5 samples
df.head(5)<jupyter_output><empty_output><jupyter_text>Create a feature to identify time of sunrise<jupyter_code># Create a new feature for sunrise using the function below; fill time for sunrise
def time_to_sunrise(row):
    """
    A function for identifying time of sunrise.
    """
    value= 0
    if row['HourPS'] == 4 and row['HourPE'] == 5 and row['Radiation'] > 0:
        value= '05:00:00'
    elif row['HourPS'] == 5 and row['HourPE'] == 6 and row['Radiation'] > 0:
        value= '06:00:00'
    return value<jupyter_output><empty_output><jupyter_text>Add the feature into the dataframe<jupyter_code># Create a new column using the function above
df['Sunrise'] = df.apply(time_to_sunrise, axis=1)
df.head(5)<jupyter_output><empty_output><jupyter_text>Create a feature to identify time of sunset<jupyter_code># Create a new feature for sunset using the function below; fill time for sunset
def time_to_sunset(row):
    """
    A function for identifying time of sunset.
    """
    value= 0
    if row['HourPS'] == 17 and row['HourPE'] == 18 and row['Radiation']> 0:
        value= '18:00:00'
    elif row['HourPS'] == 18 and row['HourPE']== 19 and row['Radiation']> 0:
        value= '19:00:00'
    return value<jupyter_output><empty_output><jupyter_text>Add the feature into dataframe<jupyter_code># Create a new column using the function above
df['Sunset'] = df.apply(time_to_sunset, axis=1)<jupyter_output><empty_output><jupyter_text>Show unique values of sunset feature<jupyter_code># Unique values of sunset feature
df.Sunset.unique()<jupyter_output><empty_output><jupyter_text>Summary of dataframe<jupyter_code># Show summary of our dataframe
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 131421 entries, 0 to 131420
Data columns (total 29 columns):
 #   Column         Non-Null Count   Dtype         
---  ------         --------------   -----         
 0   PeriodEnd      131421 non-null  datetime64[ns]
 1   PeriodStart    131421 non-null  datetime64[ns]
 2   Period         131421 non-null  object        
 3   Temperature    131421 non-null  float64       
 4   Azimuth        131421 non-null  int64         
 5   CloudOpacity   131421 non-null  float64       
 6   DewpointTemp   131421 non-null  float64       
 7   DHI            131421 non-null  int64         
 8   DNI            131421 non-null  int64         
 9   Ebh            131421 non-null  int64         
 10  Radiation      131421 non-null  float64       
 11  GtiFixedTilt   131421 non-null  int64         
 12  GtiTracking    131421 non-null  int64         
 13  Precipitation  131421 non-null  float64       
 14  Humidity       131421 non-null  int32         
 15  [...]<jupyter_text>Change interger values to datetime object<jupyter_code># Changing integer values to datetime format
df_new = df.loc[~((df['Radiation'] == 0) & (df['DNI'] == 0) & (df['DHI'] == 0)),:]<jupyter_output><empty_output><jupyter_text>Display summary of dataframe<jupyter_code># Show summary of our dataframe
df_new.info()<jupyter_output><empty_output><jupyter_text>Display 10 random samples<jupyter_code># display 10 random samples of the dataframe
df_new.sample(10)<jupyter_output><empty_output><jupyter_text>Convert datetime column<jupyter_code># Convert datetime column of endperiod to just date
df_new['Date'] = pd.to_datetime(df_new['PeriodEnd']).dt.normalize()

# Convert the column (it's a string) to datetime type
dt_series = pd.to_datetime(df_new['Date'])

# Create datetime index passing the datetime series
dt_index = pd.DatetimeIndex(dt_series.values)

# Set new index
df_new1 = df_new.set_index(dt_index)<jupyter_output>C:\Users\sulma\AppData\Local\Programs\Python\Python37\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>Display first 5 rows<jupyter_code># Display first 5 rows
df_new1.head()<jupyter_output><empty_output><jupyter_text>Summary of dataframe<jupyter_code># Display summary of dataframe
df_new1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 71885 entries, 2007-01-01 to 2021-12-28
Data columns (total 30 columns):
 #   Column         Non-Null Count  Dtype         
---  ------         --------------  -----         
 0   PeriodEnd      71885 non-null  datetime64[ns]
 1   PeriodStart    71885 non-null  datetime64[ns]
 2   Period         71885 non-null  object        
 3   Temperature    71885 non-null  float64       
 4   Azimuth        71885 non-null  int64         
 5   CloudOpacity   71885 non-null  float64       
 6   DewpointTemp   71885 non-null  float64       
 7   DHI            71885 non-null  int64         
 8   DNI            71885 non-null  int64         
 9   Ebh            71885 non-null  int64         
 10  Radiation      71885 non-null  float64       
 11  GtiFixedTilt   71885 non-null  int64         
 12  GtiTracking    71885 non-null  int64         
 13  Precipitation  71885 non-null  float64       
 14  Humidity       71885 non-null  int32         
 15  Sn[...]<jupyter_text>Tabulate the average for the solar and weather features<jupyter_code># Sample daily average for the solar and weather features
df_new1['Daily_radiation'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['Radiation'].mean()
df_new1['Daily_DNI'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['DNI'].mean()
df_new1['Daily_DHI'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['DHI'].mean()
df_new1['Daily_Temp'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['Temperature'].mean()
df_new1['Daily_Precip'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['Precipitation'].mean()
df_new1['Daily_Humidity'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['Humidity'].mean()
df_new1['Daily_Pressure'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['Pressure'].mean()
df_new1['Daily_WindDir'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['WindDirection'].mean()
df_new1['Daily_WindSpeed'] = df_new1.reset_index().groupby(pd.Grouper(key='Date', freq='1D'))['WindSpeed'].mean()<jupyter_output><empty_output><jupyter_text>Summary of dataframe<jupyter_code># display a summary of our dataframe
df_new1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 71885 entries, 2007-01-01 to 2021-12-28
Data columns (total 39 columns):
 #   Column           Non-Null Count  Dtype         
---  ------           --------------  -----         
 0   PeriodEnd        71885 non-null  datetime64[ns]
 1   PeriodStart      71885 non-null  datetime64[ns]
 2   Period           71885 non-null  object        
 3   Temperature      71885 non-null  float64       
 4   Azimuth          71885 non-null  int64         
 5   CloudOpacity     71885 non-null  float64       
 6   DewpointTemp     71885 non-null  float64       
 7   DHI              71885 non-null  int64         
 8   DNI              71885 non-null  int64         
 9   Ebh              71885 non-null  int64         
 10  Radiation        71885 non-null  float64       
 11  GtiFixedTilt     71885 non-null  int64         
 12  GtiTracking      71885 non-null  int64         
 13  Precipitation    71885 non-null  float64       
 14  Humidity         7188[...]<jupyter_text>Drop irrelevant features<jupyter_code># Drop irrelevant features
new_df= df_new1.drop(['PeriodStart', 'YearPS',
                      'HourPS', 'HourPE',
                      'Sunrise', 'Sunset',
                      'MonthPS', 'YearPE'], axis= 1)<jupyter_output><empty_output><jupyter_text>Display top 10 rows<jupyter_code># Display first 10 rows
new_df.head(10)<jupyter_output><empty_output><jupyter_text>Summary of dataframe<jupyter_code># Show a summary of the dataframe
new_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 71885 entries, 2007-01-01 to 2021-12-28
Data columns (total 31 columns):
 #   Column           Non-Null Count  Dtype         
---  ------           --------------  -----         
 0   PeriodEnd        71885 non-null  datetime64[ns]
 1   Period           71885 non-null  object        
 2   Temperature      71885 non-null  float64       
 3   Azimuth          71885 non-null  int64         
 4   CloudOpacity     71885 non-null  float64       
 5   DewpointTemp     71885 non-null  float64       
 6   DHI              71885 non-null  int64         
 7   DNI              71885 non-null  int64         
 8   Ebh              71885 non-null  int64         
 9   Radiation        71885 non-null  float64       
 10  GtiFixedTilt     71885 non-null  int64         
 11  GtiTracking      71885 non-null  int64         
 12  Precipitation    71885 non-null  float64       
 13  Humidity         71885 non-null  int32         
 14  SnowWater        7188[...]<jupyter_text>Drop all duplicate values except for the last value<jupyter_code># Dropping ALL duplicate values exceept the last value
new_df = new_df[~new_df.Date.duplicated(keep = 'last')]<jupyter_output><empty_output><jupyter_text>Display the number of rows and columns<jupyter_code># show the number of rows and columns available
new_df.shape<jupyter_output><empty_output><jupyter_text>Top 10 samples<jupyter_code># Show top 10 samples
new_df.head(10)<jupyter_output><empty_output><jupyter_text>Display column names<jupyter_code># List out the column names
new_df.columns<jupyter_output><empty_output><jupyter_text>Select pertinent features<jupyter_code># Select column features
final_df= new_df[['MonthPE', 'Date',
                  'Daily_Temp','Daily_Precip',
                  'Daily_Humidity', 'Daily_Pressure',
                  'Daily_WindDir','Daily_WindSpeed',
                  'Daily_DNI', 'Daily_DHI',
                  'Daily_radiation']]<jupyter_output><empty_output><jupyter_text>Top 5 rows<jupyter_code># Print the first 5 rows
final_df.head()<jupyter_output><empty_output><jupyter_text>Display the shape<jupyter_code># Show the number of rows and column
final_df.shape<jupyter_output><empty_output><jupyter_text>Export cleaned dataset<jupyter_code># Save cleaned dataset
final_df.to_csv('../datasets/cleaned/cleaned_solar_irradiation.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10747310264611806, 'fraction_numerical': 0.06182029659784821, 'mean_word_length': 3.274422073079791, 'pattern_counts': {'":': 0, '<': 151, '<?xml version=': 0, '>': 155, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '650227', 'license': 'MIT', 'n_tokens_mistral': 6357, 'n_tokens_neox': 5586, 'n_words': 1409, 'path': 'Assignments/Assignment #4/notebooks/data_cleaning.ipynb', 'repo_name': 'SulmanK/Omdena-MLOPS', 'size': '17195'}