<jupyter_start><jupyter_text>WIND TURBINES PREDICTIVE MAINTENANCE THROUGH ML: DATA PREPARATION - SOLUTION (3)  Load libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
%matplotlib inline
sns.set_style('darkgrid')<jupyter_output><empty_output><jupyter_text>1) Load data<jupyter_code>DATA_ROOT = "Training/"
DATA_CLEAN_ROOT = 'data_clean'
DATA = 'df.csv'
DATA_MAST_CLEAN = 'mast.p'
DATA_FAILURE = 'Training_failures.csv'
df=pd.read_csv(os.path.join(DATA_CLEAN_ROOT, DATA))
df<jupyter_output><empty_output><jupyter_text>Set up the index<jupyter_code>df.Timestamp = pd.to_datetime(df.Timestamp) 
df.set_index(['Turbine_ID','Timestamp'], inplace=True)
df<jupyter_output><empty_output><jupyter_text>2) Remove low variance features  It is interesting to remove all low variance features (i.e. features that are approximately constant) since they bring small information that will not improve the model performance.**What are the features that have zero variance? Remove them from the dataset.** *hint*: use `describe` or `std` function.  To display all the columns run `pd.set_option('display.max_columns', None)` instruction.  Describe function<jupyter_code>df.describe()
pd.set_option('display.max_columns', None)
df.describe()<jupyter_output><empty_output><jupyter_text>STD function<jupyter_code>for col in df.columns:
    if df[col].std()==0:
        print(col)<jupyter_output>Prod_LatestAvg_ActPwrGen2
Prod_LatestAvg_ReactPwrGen2<jupyter_text>Removing the features that have zero variance<jupyter_code>df=df.drop(['Prod_LatestAvg_ActPwrGen2','Prod_LatestAvg_ReactPwrGen2'],axis=1)<jupyter_output><empty_output><jupyter_text>3) Split the dataset   Data from 2016 will be used for training the ML algorithms while data from 2017 will be used for testing purposes.<jupyter_code>df.reset_index(inplace=True, drop=False)
df.set_index('Timestamp',inplace=True)
Data_tr=df.loc['2016']
Data_tr.reset_index(inplace=True, drop=False)
Data_tr.set_index(['Turbine_ID','Timestamp'], inplace=True)
Data_ts=df.loc['2017']
Data_ts.reset_index(inplace=True, drop=False)
Data_ts.set_index(['Turbine_ID','Timestamp'], inplace=True)
# Training data
Data_tr
# Testing data
Data_ts<jupyter_output><empty_output><jupyter_text>4) Create new features for training dataset  In order to properly train a classifier, it is important that the data is correctly labelled. In this work, the objective is to detect the failures 60 days before it occurs and isolate the component of the turbine that is going to fail. Therefore, a “Failure component” column was added, based on Failure dataset. The column takes the following labels/values:- `Gearbox: 1`- `Generator: 2`- `Generator_Bearing: 3`- `Transformer: 4`    - `Hydraulic: 5`- `The incoming failure is after 60 days: -1`Note: Sometimes samples are missing when the failure occurs, so you have to specify the timestamp immediately preceding it.<jupyter_code># load the failures in the training set and set up the index
Failures_tr=pd.read_csv('failures_tr.csv')
Failures_tr.set_index('Turbine_ID', inplace=True)

# reset the index of training data
Data_tr.reset_index(inplace=True, drop=False)<jupyter_output><empty_output><jupyter_text>Define a function to add the component column to each turbine data<jupyter_code>def add_com_variable(Tur_data, time, com):
    
    dys='60 days'     # Change it to timedelta in order to subtract from another timedelta. 
    dys= pd.to_timedelta(dys)    #Timedeltas are absolute differences in times, expressed in difference units
    
    # index in time and component list
    k=0 
    
    # list for component column
    lc=np.empty
    lc = np.empty(len(Tur_data.index))
    lc[:] = -1
    
    for index, row in Tur_data.iterrows():   #Iterate over DataFrame rows
        if k==len(time):
             break
        a=row.Timestamp   # take the timestamp for each row 
        b=pd.to_datetime(time[k])    # take the time from the time list and change it to datetime
        c=b-a  # subtract the two timestamp, the result is timedelta.
        if c<=dys:   # compare the two timedelta, if c<60 days 
            lc[index]=com[k]  # change the content from -1 to the failure label
            
        # increase k to take the next failure    
        for i in time:   
            if row.Timestamp==pd.to_datetime(i):
                k+=1
                
    # Add the created list to the Turbine T01 data and print the moment when the failure occurred.            
    Tur_data.insert(81,"Component",lc, True)
    return Tur_data<jupyter_output><empty_output><jupyter_text>*Remark*: The function parameters (time and com) must be defined from the oldest to the newest.  Turbine T01<jupyter_code>Failures_tr.loc['T01']
# Get the training data for turbine T01 and reset the index.
TUR01_tr= Data_tr[Data_tr.Turbine_ID=='T01']    
TUR01_tr.reset_index(inplace=True, drop= True)
# use the function above and the failures to add the component variable to training data for turbine T01
TUR01_tr= add_com_variable(TUR01_tr, ['2016-07-18T02:10:00+00:00'], [1])
TUR01_tr
sum(TUR01_tr.Component == 1)<jupyter_output><empty_output><jupyter_text>Turbine T06<jupyter_code>Failures_tr.loc['T06']
# Get the training data for turbine T06 and reset the index.
TUR06_tr= Data_tr[Data_tr.Turbine_ID=='T06']    
TUR06_tr.reset_index(inplace=True, drop= True)

# use the function above to add the component variable to training data for turbine T06
time, com =['2016-04-04T18:50:00+00:00','2016-07-10T20:40:00+00:00','2016-07-24T17:00:00+00:00', \
            '2016-09-04T08:00:00+00:00','2016-10-02T17:00:00+00:00','2016-10-27T16:20:00+00:00'], [5 ,2 ,2 ,2 ,2 ,2]

TUR06_tr= add_com_variable(TUR06_tr, time, com)
sum(TUR06_tr.Component == 2)<jupyter_output><empty_output><jupyter_text>Turbine T07<jupyter_code>Failures_tr.loc['T07']
# Get the training data for turbine T07 and reset the index.
TUR07_tr= Data_tr[Data_tr.Turbine_ID=='T07']    
TUR07_tr.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to training data for turbine T07


time, com=['2016-04-30T12:40:00+00:00','2016-07-10T03:40:00+00:00','2016-08-23T02:20:00+00:00'], [3 ,4 ,4]

TUR07_tr= add_com_variable(TUR07_tr, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T09<jupyter_code>Failures_tr.loc['T09']
# Get the training data for turbine T09 and reset the index.
TUR09_tr= Data_tr[Data_tr.Turbine_ID=='T09']    
TUR09_tr.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to training data for turbine T09


time, com=['2016-06-07T16:50:00+00:00','2016-08-22T18:20:00+00:00',\
           '2016-10-11T08:00:00+00:00','2016-10-17T09:10:00+00:00'], [3 ,3 ,1 ,3]

TUR09_tr= add_com_variable(TUR09_tr, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T11<jupyter_code>Failures_tr.loc['T11']
# Get the training data for turbine T11 and reset the index.
TUR11_tr= Data_tr[Data_tr.Turbine_ID=='T11']    
TUR11_tr.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to training data for turbine T11


time, com =['2016-03-03T18:50:00+00:00','2016-10-17T17:40:00+00:00'], [2 ,5]

TUR11_tr= add_com_variable(TUR11_tr, time, com)
# combine the data
Data_tr=pd.concat([TUR01_tr, TUR06_tr, TUR07_tr, TUR09_tr, TUR11_tr],ignore_index=True) 
Data_tr<jupyter_output><empty_output><jupyter_text>5) Create same new features for testing dataset --> Ground truth<jupyter_code># load the failures in the testing set and set up the index
Failures_ts=pd.read_csv('failures_ts.csv')
Failures_ts.set_index('Turbine_ID', inplace=True)

# reset the index of testing data
Data_ts.reset_index(inplace=True, drop=False)<jupyter_output><empty_output><jupyter_text>Turbine T01<jupyter_code>Failures_ts.loc['T01']
# Get the testing data for turbine T01 and reset the index.
TUR01_ts= Data_ts[Data_ts.Turbine_ID=='T01']    
TUR01_ts.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to testing data for turbine T01

time, com=['2017-08-11T13:10:00+00:00'], [4]

TUR01_ts= add_com_variable(TUR01_ts, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T06<jupyter_code>Failures_ts.loc['T06']
# Get the testing data for turbine T06 and reset the index.
TUR06_ts= Data_ts[Data_ts.Turbine_ID=='T06']    
TUR06_ts.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to testing data for turbine T06

time, com=['2017-08-19T09:40:00+00:00','2017-10-17T08:30:00+00:00'], [5 ,1]

TUR06_ts= add_com_variable(TUR06_ts, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T07<jupyter_code>Failures_ts.loc['T07']
# Get the testing data for turbine T07 and reset the index.
TUR07_ts= Data_ts[Data_ts.Turbine_ID=='T07']    
TUR07_ts.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to testing data for turbine T07

time, com=['2017-06-17T11:30:00+00:00','2017-08-20T06:00:00+00:00',\
           '2017-08-21T14:40:00+00:00','2017-10-19T10:10:00+00:00'], [5 ,3 ,2 ,5]

TUR07_ts= add_com_variable(TUR07_ts, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T09<jupyter_code>Failures_ts.loc['T09']
# Get the testing data for turbine T09 and reset the index.
TUR09_ts= Data_ts[Data_ts.Turbine_ID=='T09']    
TUR09_ts.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to testing data for turbine T09

time, com=['2017-01-25T12:50:00+00:00','2017-09-16T15:40:00+00:00','2017-10-18T08:30:00+00:00'], [3 ,5 ,1]

TUR09_ts= add_com_variable(TUR09_ts, time, com)<jupyter_output><empty_output><jupyter_text>Turbine T11<jupyter_code>Failures_ts.loc['T11']
# Get the testing data for turbine T11 and reset the index.
TUR11_ts= Data_ts[Data_ts.Turbine_ID=='T11']    
TUR11_ts.reset_index(inplace=True, drop= True)
# use the function above to add the component variable to testing data for turbine T11

time, com=['2017-04-26T18:00:00+00:00','2017-09-12T15:30:00+00:00'], [5 ,5]

TUR11_ts= add_com_variable(TUR11_ts, time, com)
# combine the data
Data_ts=pd.concat([TUR01_ts, TUR06_ts, TUR07_ts, TUR09_ts, TUR11_ts],ignore_index=True)
Data_ts
# Save clean data
Data_tr.to_csv('data_clean/Data_tr.csv')
Data_ts.to_csv('data_clean/Data_ts.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11489032756788095, 'fraction_numerical': 0.07633587786259542, 'mean_word_length': 5.707712248865846, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '562005', 'license': 'MIT', 'n_tokens_mistral': 4291, 'n_tokens_neox': 3801, 'n_words': 936, 'path': 'JupyterNotebooks/3_Prepare the data_Solution.ipynb', 'repo_name': 'Sulles/WindFarmProject', 'size': '10349'}