<jupyter_start><jupyter_text>2.1 Explanatory data analysis and Data cleaning Data Cleaning<jupyter_code># Import libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Importing the dataset<jupyter_code>df = pd.read_csv('../data/GEFCom2014Data/Wind/raw_data_incl_features.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Inspecting tha data<jupyter_code>df.info()
df.isna().sum()<jupyter_output><empty_output><jupyter_text>The dataset has 175 NaNs, we linearly interpolate them.<jupyter_code>df.interpolate(method='linear', inplace=True)<jupyter_output><empty_output><jupyter_text>Creating dummy variables for ```WD100CARD ``` and  ```WD10CARD ```<jupyter_code>df = pd.get_dummies(df, columns = ['WD100CARD','WD10CARD'], drop_first=True)<jupyter_output><empty_output><jupyter_text>Saving the cleaned dataset in ```clean_data.csv```<jupyter_code># Save cleaned dataframe including to csv
df.to_csv('../data/GEFCom2014Data/Wind/clean_data.csv', index = False)<jupyter_output><empty_output><jupyter_text>EDA View statistical properties of the target and the windspeeds / winddirections<jupyter_code>df[['TARGETVAR', 'U10', 'V10', 'U100', 'V100', 'WS10', 'WS100']].describe().T<jupyter_output><empty_output><jupyter_text>The dateframe has ~175 K obersavtions in total, and ~17,5 K observations per windfarm. 46 features and the target variable.<jupyter_code>df.shape
for zone in df.ZONEID.unique():
    print("Windfarm {}: ".format(zone),df[df['ZONEID']==zone].shape)<jupyter_output>Windfarm 1:  (17544, 47)
Windfarm 2:  (17544, 47)
Windfarm 3:  (17544, 47)
Windfarm 4:  (17544, 47)
Windfarm 5:  (17544, 47)
Windfarm 6:  (17544, 47)
Windfarm 7:  (17544, 47)
Windfarm 8:  (17544, 47)
Windfarm 9:  (17544, 47)
Windfarm 10:  (17544, 47)<jupyter_text>Distributions of Target and features Distribution of the target ``` TARGETVAR```<jupyter_code>sns.set_theme()
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10,8))
sns.histplot(data = df, x = df.TARGETVAR)
plt.title('Distribution of target');
print('min:', round(df.TARGETVAR.min(),2))
print('max:', round(df.TARGETVAR.max(),2))
print('mean:', round(df.TARGETVAR.mean(),2))<jupyter_output>min: 0.0
max: 1.0
mean: 0.36<jupyter_text>Distribution of the target on each windfarm<jupyter_code>fig, ax = plt.subplots(nrows=2, ncols=5, sharey=True, figsize=(15,8))
fig.suptitle('Distribution of target in every windfarm')
for zone in df.ZONEID.unique():
    plt.subplot(2,5,zone)   
    plt.title("Windfarm {}".format(zone))
    sns.histplot(data = df, x = df[df['ZONEID']==zone].TARGETVAR)
    
fig.tight_layout();<jupyter_output><empty_output><jupyter_text>Distribution of the Windspeed  ``` WS100``` on 100m. The windspeed has values between 0 and 22 m/s.The mean is 6,6 m/s.<jupyter_code>fig, ax = plt.subplots(nrows=2, ncols=5, sharey=True,figsize=(15,8))
fig.suptitle('Distribution of WS100 in every windfarm')
for zone in df.ZONEID.unique():
    plt.subplot(2,5,zone)   
    plt.title("Windfarm {}".format(zone))
    sns.histplot(data = df, x = df[df['ZONEID']==zone].WS100, bins=30)
fig.tight_layout();   
print('min:', round(df.WS100.min(),2))
print('max:', round(df.WS100.max(),2))
print('mean:',round(df.WS100.mean(),2))
# distribution of zonal and meridional wind at the different sites
fig,ax = plt.subplots(nrows=1, ncols=3, figsize=(20,7))
ax = np.reshape(ax,3)

for i,var in enumerate(['U100','V100','WS100']): 
    sns.boxplot(data=df, x='ZONEID', y=var, color='gray', showmeans=True, ax=ax[i])
    ax[i].grid()
    if i < 2:
        ax[i].set_ylim([-23.,23.])
    else:
        ax[i].set_ylim([-5.,23.])
    ax[i].set_ylabel('{} {}'.format(var,"$\mathregular{[m\ s^{-1}]}$"))<jupyter_output><empty_output><jupyter_text>Distribution of zonal wind at height 10 m ```U10```<jupyter_code>zones = df.ZONEID.unique()

fig, axs = plt.subplots(2,5,figsize = (30, 10))

axs = axs.ravel()

for zone in zones:
    sns.histplot(data = df[df['ZONEID'] == zone], x = 'U10', bins = 20, ax = axs[zone - 1])
    axs[zone - 1].set_title('ZONEID {}'.format(zone))

fig.tight_layout()<jupyter_output><empty_output><jupyter_text>Distribution of zonal wind at height 100 m ```U100```<jupyter_code>zones = df.ZONEID.unique()

fig, axs = plt.subplots(2,5,figsize = (30, 10))

axs = axs.ravel()

for zone in zones:
    sns.histplot(data = df[df['ZONEID'] == zone], x = 'U100', bins = 20, ax = axs[zone - 1])
    axs[zone - 1].set_title('ZONEID {}'.format(zone))

fig.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17275528169014084, 'fraction_numerical': 0.05413732394366197, 'mean_word_length': 6.123824451410658, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '576009', 'license': 'MIT', 'n_tokens_mistral': 2000, 'n_tokens_neox': 1801, 'n_words': 360, 'path': 'notebooks/2_1_EDA_Data_Cleaning.ipynb', 'repo_name': 'christine-berlin/Capstone_WindPowerPredicting', 'size': '4544'}