<jupyter_start><jupyter_text>Table of Contents<jupyter_code>%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')

// needed to generate the Table of contents 
// taken from github.com/kmahelona/ipython_notebook_goodies<jupyter_output><empty_output><jupyter_text>(To work with Atom and Jupyter at the same time)<jupyter_code>%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Data Wrangling  Data Collection Locating the dataTo train this model we will use the Turbofan Engine Degradation Simulation Data Set from NASA ([Link to dataset](https://ti.arc.nasa.gov/c/6/)).><NAME> and <NAME> (2008). "Turbofan Engine Degradation Simulation Data Set", NASA Ames Prognostics Data Repository (http://ti.arc.nasa.gov/project/prognostic-data-repository), NASA Ames Research Center, Moffett Field, CA<jupyter_code>#load python packages
import os
import glob
import pandas as pd
#import datetime
#import seaborn as sns
#import matplotlib.pyplot as plt
#import numpy as np
#%matplotlib inline
#navigate to the data folder
os.chdir('..')
print(os.getcwd())
# Download data in data/raw folder
! mkdir data/raw
! cd data/raw && wget -O turbofan.zip https://ti.arc.nasa.gov/c/6/ && unzip -o *.zip
! cd data/raw && rm *.zip

#NOTE the cd executed with the magic command do not change the working directory of the python script
#We are still working off the project root folder<jupyter_output>mkdir: cannot create directory ‘data/raw’: File exists
--2020-08-12 18:24:44--  https://ti.arc.nasa.gov/c/6/
Resolving ti.arc.nasa.gov (ti.arc.nasa.gov)... 192.168.127.12, 2001:4d0:6311:2227:14b6:372b:2078:2a94
Connecting to ti.arc.nasa.gov (ti.arc.nasa.gov)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://ti.arc.nasa.gov/m/project/prognostic-repository/CMAPSSData.zip [following]
--2020-08-12 18:24:45--  https://ti.arc.nasa.gov/m/project/prognostic-repository/CMAPSSData.zip
Reusing existing connection to ti.arc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 12425978 (12M) [application/zip]
Saving to: ‘turbofan.zip’

turbofan.zip        100%[===================>]  11.85M  9.25MB/s    in 1.3s    

2020-08-12 18:24:46 (9.25 MB/s) - ‘turbofan.zip’ saved [12425978/12425978]

Archive:  turbofan.zip
  inflating: Damage Propagation Modeling.pdf  
  inflating: readme.txt              
  inflating: RUL_FD001.txt          [...]<jupyter_text>Data LoadingAfter downloading and unzipping the files, the data folder has the following structure. ```data/raw├── Damage Propagation Modeling.pdf├── readme.txt├── RUL_FD001.txt├── RUL_FD002.txt├── RUL_FD003.txt├── RUL_FD004.txt├── test_FD001.txt├── test_FD002.txt├── test_FD003.txt├── test_FD004.txt├── train_FD001.txt├── train_FD002.txt├── train_FD003.txt└── train_FD004.txt```From the readme file we can extract the file structure. There are 4 sets of files:1. FD0012. FD0023. FD0034. FD004Each containing 3 types of files:1. Training data2. Test data3. Remaining Usable Life (RUL) dataTest and training data files have the same column structure:1. unit number2. cycle time3. operating setting 14. operating setting 25. operating setting 36. sensor reading 17. sensor reading 2...8. sensor reading 21The RUL data file has a single column corresponding to the RUL value.We will import the txt files and consolidate the data in 3 labelled csv files which will reside in the data folder:```data├── RUL.csv├── test.csv└── train.csv```<jupyter_code># define column names
sensors_list = ["s{}".format(s) for s in range(1,22)]
train_cols = ['unit_number','cycle_time','op_setting_1', 'op_setting_2', 'op_setting_3'] + sensors_list
test_cols = train_cols
RUL_cols = ['RUL']

# load Training data
file_paths = glob.glob("data/raw/train_*.txt")

df_list = []
for file_path in file_paths : 
    # read txt file
    individual_df = pd.read_csv(file_path, sep=' ', header=None, usecols = [i for i in range(26)])
    individual_df.columns = train_cols
    #extract dataset Id from filename
    data_set=file_path[-9:-4]
    individual_df['dataset']=file_path[-9:-4] 
    #append temporary dataframe to list
    df_list.append(individual_df)
#merge into single dataframe
df_train=pd.concat(df_list)
df_train.head()

# write to csv
df_train.to_csv('data/train.csv', index=False)
# load Test data
file_paths = glob.glob("data/raw/test_*.txt")

df_list = []
for file_path in file_paths : 
    # read txt file
    individual_df = pd.read_csv(file_path, sep=' ', header=None, usecols = [i for i in range(26)])
    individual_df.columns = test_cols
    #extract dataset Id from filename
    data_set=file_path[-9:-4]
    individual_df['dataset']=file_path[-9:-4] 
    #append temporary dataframe to list
    df_list.append(individual_df)
#merge into single dataframe
df_test=pd.concat(df_list)
df_test.head()

# write to csv
df_test.to_csv('data/test.csv', index=False)
# load RUL data
file_paths = glob.glob("data/raw/RUL_*.txt")

df_list = []
for file_path in file_paths : 
    # read txt file
    individual_df = pd.read_csv(file_path, sep=' ', header=None, usecols=[0])
    individual_df.columns = RUL_cols
    individual_df['unit_number'] = individual_df.index + 1 #converting index to unit_number
    #extract dataset Id from filename
    data_set=file_path[-9:-4]
    individual_df['dataset']=file_path[-9:-4] 
    #append temporary dataframe to list
    df_list.append(individual_df)
#merge into single dataframe
df_RUL=pd.concat(df_list)
df_RUL.head()

# write to csv
df_RUL.to_csv('data/RUL.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12529002320185614, 'fraction_numerical': 0.04497590576476888, 'mean_word_length': 4.677811550151976, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '228522', 'license': 'MIT', 'n_tokens_mistral': 2126, 'n_tokens_neox': 1950, 'n_words': 525, 'path': 'notebooks/1 - Importing Data.ipynb', 'repo_name': 'andrea-fantini/Predictive_Maintenance_Fanjet', 'size': '5609'}