<jupyter_start><jupyter_text>Root Cause Analysis of IT incedentsAutomatic identification of the root cause for a system issue empowers level two technicians while reducing escalations, reduces resolution times, improving user satisfaction.<jupyter_code>#Install necessary packages in conda. Use pip in not using conda.
import sys
!conda install --yes --prefix {sys.prefix} pandas tensorflow scikit-learn<jupyter_output>Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda



# All requested packages already installed.<jupyter_text>Preprocessing Incident Data - Loading the Dataset1. Load the CSV into a Pandas data frame using the read_csv method. 2. Print the data types and contents of the data frame to verify load.<jupyter_code>import pandas as pd
import os
import tensorflow as tf

#Load the data file into a Pandas Dataframe
symptom_data = pd.read_csv("root_cause_analysis.csv")

#Take a look at the data loaded
print(symptom_data.dtypes)
symptom_data.head()<jupyter_output>ID              int64
CPU_LOAD        int64
MEMORY_LOAD     int64
DELAY           int64
ERROR_1000      int64
ERROR_1001      int64
ERROR_1002      int64
ERROR_1003      int64
ROOT_CAUSE     object
dtype: object<jupyter_text>Convert dataKeras works with NumPy arrays. We need to convert the data accordingly. 1. Convert root cause into a numeric value.2. Convert the data frame into a NumPy array using the to_numpy function.3. Separate the training attributes features (F_train) into the target (T_train) array. 4. Extract target var into T_train.5. Use the one-hot encoding on this categorical target attribute for it to be consumed by Keras - see utils.to_categorical function.6. Print the shape of these variables to check if all the processing was successful.<jupyter_code>from sklearn import preprocessing

#convert from text to numeric values
label_encoder = preprocessing.LabelEncoder()
symptom_data['ROOT_CAUSE'] = label_encoder.fit_transform(
                                symptom_data['ROOT_CAUSE'])

#Convert Pandas DataFrame to a numpy vector
np_symptom = symptom_data.to_numpy().astype(float)

#Extract the feature variables (F)
F_train = np_symptom[:,1:8]

#Extract the target variable (T), convert to one-hot-encoding
T_train=np_symptom[:,8]
T_train = tf.keras.utils.to_categorical(T_train,3)

print("Number of samples and shape of feature variables :", F_train.shape)
print("Number of samples and shape of target variable :",T_train.shape)<jupyter_output>Number of samples and shape of feature variables : (1000, 7)
Number of samples and shape of target variable : (1000, 3)<jupyter_text>Building the Model with KerasCreate a softmax model with three dense layers in Keras to predict the root cause based on the symptoms. 1. Set up the hyperparameters for the neural network. 2. Use Keras sequential model which is appropriate for a plain stack of layers where each layer has exactly one input tensor and one output tensor.3. The number of output classes will be set to match the unique number of labels in the target variable, namely root cause. 4. Add a hidden layer size of 128. 5. Validate with 20% of the input records. 6. Create the sequential model. 7. Add the first hidden layer as a dense layer with activation as RLU. 8. Add a second similar layer. 9. Add a softmax layer to provide categorical labels. 10. Compile the model (configure its learning process) using Adam optimizer and set the loss function as categorical, cross, and trophy. 11. Fit the model with F_train and T_train in order to create the model. 12. Print the model summary to see the model structure.<jupyter_code>from tensorflow import keras
from tensorflow.keras import optimizers
from tensorflow.keras.regularizers import l2

#Training Parameters
EPOCHS=20
BATCH_SIZE=100
VERBOSE=1
OUTPUT_CLASSES=len(label_encoder.classes_)
N_HIDDEN=128
VALIDATION_SPLIT=0.2

#Create a Keras sequential model
model = tf.keras.models.Sequential()
#Add a Dense Layer
model.add(keras.layers.Dense(N_HIDDEN,
                             input_shape=(7,),
                              name='Dense-Layer-1',
                              activation='relu'))

#Add a second dense layer
model.add(keras.layers.Dense(N_HIDDEN,
                              name='Dense-Layer-2',
                              activation='relu'))

#Add a softmax layer for categorial prediction
model.add(keras.layers.Dense(OUTPUT_CLASSES,
                             name='Final',
                             activation='softmax'))

#Compile the model, using Adam optimizer
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

#Build the model
model.fit(F_train,
          T_train,
          batch_size=BATCH_SIZE,
          epochs=EPOCHS,
          verbose=VERBOSE,
          validation_split=VALIDATION_SPLIT)

model.summary()<jupyter_output>Train on 800 samples, validate on 200 samples
Epoch 1/20
800/800 [==============================] - 1s 902us/sample - loss: 1.0196 - accuracy: 0.6075 - val_loss: 0.9169 - val_accuracy: 0.7700
Epoch 2/20
800/800 [==============================] - 0s 82us/sample - loss: 0.8389 - accuracy: 0.7950 - val_loss: 0.7675 - val_accuracy: 0.7600
Epoch 3/20
800/800 [==============================] - 0s 84us/sample - loss: 0.6842 - accuracy: 0.8112 - val_loss: 0.6537 - val_accuracy: 0.7800
Epoch 4/20
800/800 [==============================] - 0s 85us/sample - loss: 0.5661 - accuracy: 0.8188 - val_loss: 0.5800 - val_accuracy: 0.7900
Epoch 5/20
800/800 [==============================] - 0s 87us/sample - loss: 0.4955 - accuracy: 0.8225 - val_loss: 0.5408 - val_accuracy: 0.7700
Epoch 6/20
800/800 [==============================] - 0s 107us/sample - loss: 0.4582 - accuracy: 0.8313 - val_loss: 0.5276 - val_accuracy: 0.8000
Epoch 7/20
800/800 [==============================] - 0s 99us/sample - loss: 0.441[...]<jupyter_text>Predicting Root Causes<jupyter_code>#Pass individual flags to Predict the root cause
CPU_LOAD=1
MEMORY_LOAD=0
DELAY=0
ERROR_1000=0
ERROR_1001=1
ERROR_1002=1
ERROR_1003=0

prediction=model.predict_classes(
    [[CPU_LOAD,MEMORY_LOAD,DELAY,
      ERROR_1000,ERROR_1001,ERROR_1002,ERROR_1003]])

print(label_encoder.inverse_transform(prediction))
#Predicting as a Batch
print(label_encoder.inverse_transform(
        model.predict_classes([[1,0,0,0,1,1,0],
                                [0,1,1,1,0,0,0],
                                [1,1,0,1,1,0,1],
                                [0,0,0,0,0,1,0],
                                [1,0,1,0,1,1,1]])))<jupyter_output>['DATABASE_ISSUE' 'NETWORK_DELAY' 'MEMORY' 'DATABASE_ISSUE'
 'DATABASE_ISSUE']
{'fraction_non_alphanumeric': 0.11093200468933177, 'fraction_numerical': 0.05641852286049238, 'mean_word_length': 3.5774647887323945, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '166881', 'license': 'Apache-2.0', 'n_tokens_mistral': 2347, 'n_tokens_neox': 2065, 'n_words': 694, 'path': 'RootCauseAnalysisKerasTF.ipynb', 'repo_name': 'TimSim/RootCauseAnalysis', 'size': '6824'}