<jupyter_start><jupyter_text>Observation1. The most of the feature mean value is greater than the 75 percentile so it is extreamly postive skewed2. Most of the features max value is greater than the 75% so they have outliers3. The features perf_6_month_avg and perf_12_month_avg has max value 1 and min value -99 so most of the missing value is replace with -99.<jupyter_code># Checking the Null Value Columns

def NullValsCols(df):
  return df.columns[df.isnull().any()].to_list()

NullValsCols(backorddf)<jupyter_output><empty_output><jupyter_text>Observations  1. Out of 23 Features 22 feature having Misssing values<jupyter_code># Finding Missing Value tota with Percentage

def MissingValsPercent(df):
  missing_col = NullValsCols(df)
  total_mising_vals = df[missing_col].isnull().sum().sort_values(ascending=False)
  tota_missing_perc = ((df[missing_col].isnull().sum() / df[missing_col].isnull().count()) * 100 ).sort_values(ascending=False)
  return pd.concat([total_mising_vals, tota_missing_perc], axis=1, keys=['Total', 'Percentage(%)'])

MissingValsPercent(backorddf)<jupyter_output><empty_output><jupyter_text>Observation1. Out of 22 Missing features there is lead_time feaure only having 6% of missing value from overall dataset remaining fetures having only 2 missing record.<jupyter_code># Filter the numerical feature columns

numerical_feat = backorddf.columns[backorddf.dtypes != 'object'].to_list()
categorical_feat = backorddf.columns[backorddf.dtypes == 'object'].to_list()
print("Numerical Features:===========>", numerical_feat)
print("Categorical Features:===========>", categorical_feat)
# Finding the Nul value Index for drop 

for x in NullValsCols(backorddf):
  if x != 'lead_time':
    print(x,"============>",backorddf[backorddf[x].isnull()].index)<jupyter_output>national_inv ============> Int64Index([1687860, 242075], dtype='int64')
in_transit_qty ============> Int64Index([1687860, 242075], dtype='int64')
forecast_3_month ============> Int64Index([1687860, 242075], dtype='int64')
forecast_6_month ============> Int64Index([1687860, 242075], dtype='int64')
forecast_9_month ============> Int64Index([1687860, 242075], dtype='int64')
sales_1_month ============> Int64Index([1687860, 242075], dtype='int64')
sales_3_month ============> Int64Index([1687860, 242075], dtype='int64')
sales_6_month ============> Int64Index([1687860, 242075], dtype='int64')
sales_9_month ============> Int64Index([1687860, 242075], dtype='int64')
min_bank ============> Int64Index([1687860, 242075], dtype='int64')
potential_issue ============> Int64Index([1687860, 242075], dtype='int64')
pieces_past_due ============> Int64Index([1687860, 242075], dtype='int64')
perf_6_month_avg ============> Int64Index([1687860, 242075], dtype='int64')
perf_12_month_avg ============> Int64Ind[...]<jupyter_text>Observation1. So that 1 Missing value over 22 feature index is last row. so we can drop that last row [1687860]<jupyter_code>backorddf.drop(backorddf.index[[1687860, 242075]], inplace=True)
backorddf.shape
MissingValsPercent(backorddf)
backorddf['lead_time'] = backorddf.lead_time.fillna(backorddf.lead_time.median())<jupyter_output><empty_output><jupyter_text>Here the lead_time feature having high outlier skewed distribution. so we can use the Midian imputation for missing value<jupyter_code>MissingValsPercent(backorddf)<jupyter_output><empty_output><jupyter_text>Now there is no Missing Data in Dataset<jupyter_code># Drop sku because we dont need unique identifier

backorddf.drop('sku', axis=1, inplace=True)
# Handling the categorical Features

for x in [x for x in categorical_feat if x != 'sku']:
  print(x,"================>",backorddf[x].unique())<jupyter_output>potential_issue ================> ['No' 'Yes']
deck_risk ================> ['No' 'Yes']
oe_constraint ================> ['No' 'Yes']
ppap_risk ================> ['No' 'Yes']
stop_auto_buy ================> ['Yes' 'No']
rev_stop ================> ['No' 'Yes']
went_on_backorder ================> ['No' 'Yes']<jupyter_text>Obeservations1. So the categorical features having only 2 category ['Yes', 'No']<jupyter_code># Feature Engineering

for x in categorical_feat:
  if x != 'sku':
    backorddf[x].replace({'Yes': 1, 'No': 0}, inplace=True)
    backorddf[x].astype(int)<jupyter_output><empty_output><jupyter_text>So now we encoded the categories like 'Yes' is 1 and 'No' is 0<jupyter_code># Checking the Imabalanced Data

backorddf.went_on_backorder.value_counts()<jupyter_output><empty_output><jupyter_text>Observation 1. So above data this dataset is highly imbalanced<jupyter_code># Imbalanced Data Handling 
# we have to undersample the data


from imblearn.under_sampling import NearMiss

# Implementing Undersampling for Handling Imbalanced 
nm = NearMiss()#NearMiss do not take random_state arguement 
X,y = nm.fit_sample(backorddf[[x for x in backorddf.columns.to_list() if x != 'went_on_backorder']],backorddf['went_on_backorder'])<jupyter_output><empty_output><jupyter_text>We are undersampled the dataset for balancing the Dataset<jupyter_code># Combine X,y to Dataframe for further processing

import numpy as np

def ConvertToDf(X,y, x_cols=[], y_cols=''):
  dfs = pd.DataFrame(data = X, columns = x_cols)
  dfs_1 = pd.DataFrame(data=y, columns=[y_cols])
  return pd.concat([dfs, dfs_1], axis=1)

x_cols = [x for x in backorddf.columns.to_list() if x != 'went_on_backorder']
y_cols = 'went_on_backorder'

newBackordDf = ConvertToDf(X, y, x_cols, y_cols)
newBackordDf.shape
newBackordDf.went_on_backorder.value_counts()<jupyter_output><empty_output><jupyter_text>So now We got the Balanced dataset<jupyter_code># Save the Preprocessed dataset

newBackordDf.to_csv('preprocessed.csv')
# Check the Correlation

plt.figure(figsize=(20,12))
sns.heatmap(newBackordDf.corr(), annot=True, cmap="YlGnBu")
plt.show()
# Checking the Correlation between Independant features

def PearsonCorrelation(data=None, thersold=0.5):
  cor_col = set()
  corr_matrix = data.corr()
  for i in range(len(corr_matrix.columns)):
    for j in range(i):
      if abs(corr_matrix.iloc[i,j]) > thersold:
        cor_col.add(corr_matrix.columns[i])
  return cor_col
PearsonCorrelation(newBackordDf, 0.25)<jupyter_output><empty_output><jupyter_text>Obeservations1. I am selecting a correlation thersold is 0.25 above and selecting those above thersold correlation features<jupyter_code># Checking The Information Gain for the features

"""
IG calculates the importance of each feature by measuring the increase in entropy when the feature is given vs. absent.

Algorithm: IG(S, a) = H(S) — H(S | a) Where IG(S, a) is the information for the dataset S for the variable a for a random variable, 
H(S) is the entropy for the dataset before any change (described above) and H(S | a) is the conditional entropy for the dataset in 
the presence of variable a.
"""

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import mutual_info_classif

def FeatureInfoGain(X, y, score, no_need_feat):
  bestfeatures = SelectKBest(score_func=mutual_info_classif, k=no_need_feat) # n is number of features you want to select
  fit = bestfeatures.fit(X,y)
  dfscores = pd.DataFrame(fit.scores_)
  dfcolumns = pd.DataFrame(X.columns)
  featureScores = pd.concat([dfcolumns,dfscores],axis=1)
  featureScores.columns = ['Specs','Score']
  return featureScores[featureScores.Score > score]

X = newBackordDf[[x for x in newBackordDf.columns.to_list() if x != 'went_on_backorder']]
y = newBackordDf['went_on_backorder']

FeatureInfoGain(X, y, 0.2, 6)<jupyter_output><empty_output><jupyter_text>1. If we use the Tree based Classification Algo we can use this technique to get the bset features based on the score.<jupyter_code>''' Seleting the features'''

predictors = ['forecast_6_month', 'forecast_9_month','sales_3_month', 'national_inv','sales_6_month',
              'sales_9_month', 'perf_6_month_avg','perf_12_month_avg', 'forecast_3_month']
target_c = 'went_on_backorder'<jupyter_output><empty_output><jupyter_text>As per Infogain and Correlation. i have selected 9 predictors<jupyter_code>def OutlierRatio(df):
  q1 = df.quantile(0.25)
  q3 = df.quantile(0.75)
  IQR = q3 - q1
  total_outliers = ((df < (q1 - 1.5 * IQR)) | (df > (q3 + 1.5 * IQR))).sum().sort_values(ascending=False)
  total_outl_per = ((total_outliers/ ((df < (q1 - 1.5 * IQR)) | (df > (q3 + 1.5 * IQR))).count()) * 100).sort_values(ascending=False)
  return pd.concat([total_outliers, total_outl_per], axis=1, keys=['Outliers Total', 'Percentage(%)'])

OutlierRatio(newBackordDf[predictors]).head(17)<jupyter_output><empty_output><jupyter_text>We need to scaling the Outlier datas .so we can apply the Robust Scalar<jupyter_code>from sklearn.preprocessing import RobustScaler
from sklearn.model_selection import train_test_split

X_train,x_test,Y_train,y_test = train_test_split(newBackordDf[predictors], newBackordDf[target_c], test_size=0.3, random_state=7)

transformer = RobustScaler().fit(X_train)
X_train = transformer.transform(X_train)
x_test = transformer.transform(x_test)<jupyter_output><empty_output><jupyter_text>In Robust scaling every values is scaled as below.1. Values = (value — median)/(75th percentile value — 25th percentile value)This approach standardizes the data with out considering outliers while scaling the data.<jupyter_code>try:
  from sklearn.metrics import make_scorer, f1_score, accuracy_score, precision_score, recall_score
  from sklearn.model_selection import cross_validate 
  from sklearn.linear_model import LogisticRegression
  from sklearn.svm import LinearSVC
  from sklearn.tree import DecisionTreeClassifier
  from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
  from sklearn.neighbors import KNeighborsClassifier
  from sklearn.naive_bayes import GaussianNB
  from xgboost import XGBClassifier
except Exception as e:
  raise str(e)

class EvalModel:
  def __init__(self, x=None, y=None, fold=5):
    self._x = x
    self._y = y
    self._fold = fold 
    self._clstree = {'logreg': LogisticRegression, 'svm': LinearSVC,
                     'dsctree': DecisionTreeClassifier, 'rndfclf': RandomForestClassifier, 
                     'adabst': AdaBoostClassifier, 'nvbys': GaussianNB,
                     'knn': KNeighborsClassifier, 'xgbst': XGBClassifier}
    self._scoring = {'accuracy':make_scorer(accuracy_score), 
           'precision':make_scorer(precision_score),
           'recall':make_scorer(recall_score), 
           'f1_score':make_scorer(f1_score)}

  @staticmethod
  def _has_classifier():
    return ['logreg', 'svm', 'dsctree', 'rndfclf', 'nvbys', 'adabst', 'knn', 'xgbst']

  def _has_classifier_names(self, name):
    return {
        'logreg': 'Logistic Regression',
        'svm': 'Support Vector Classifier',
        'dsctree': 'Decision Tree',
        'rndfclf':'Random Forest',
        'nvbys':'Gaussian Naive Bayes',
        'adabst': 'AdaBoost Classifier',
        'knn': 'KNearest Classifier',
        'xgbst': 'XGBoost Classifier'
    }.get(name)

  def _perform_validate(self,nmdl=[]):
    models_scores_table = {}
    try:
      for x in nmdl:
        if x not in self._has_classifier():
          return "{x} - key of Classifier Not Available. so please execute _has_classifier_names() method to see available classifiers".format(x=x)
        validated_obj = self._cross_validate(self._clstree.get(x)())
        models_scores_table.update({
            self._has_classifier_names(x) : [
                                            validated_obj['test_accuracy'].mean(),
                                            validated_obj['test_precision'].mean(),
                                            validated_obj['test_recall'].mean(),
                                            validated_obj['test_f1_score'].mean()                
                                    ]
        })
      models_scores_table = pd.DataFrame(models_scores_table, index=['Accuracy', 'Precision', 'Recall', 'F1 Score'])

      #Add 'Best Score' column
      models_scores_table['Best Score'] = models_scores_table.idxmax(axis=1)
    
      # Return models performance metrics scores data frame
      return models_scores_table
    except Exception as e:
      raise str(e)

  def _cross_validate(self, model):
    return cross_validate(model, self._x, self._y, scoring=self._scoring, cv=self._fold)
EvalModel._has_classifier()<jupyter_output><empty_output><jupyter_text>Available Models for checking what is the best modelIf you want select Model just pass their releated keys as a list to _perform_validate(['knn']) method{        'logreg': 'Logistic Regression',        'svm': 'Support Vector Classifier',        'dsctree': 'Decision Tree',        'rndfclf':'Random Forest',        'nvbys':'Gaussian Naive Bayes',        'adabst': 'AdaBoost Classifier',        'knn': 'KNearest Classifier',        'xgbst': 'XGBoost Classifier'    }<jupyter_code>EvalModel(X_train, Y_train, 10)._perform_validate(['logreg', 'xgbst', 'svm', 'rndfclf', 'knn', 'nvbys'])<jupyter_output><empty_output><jupyter_text>Observations1. As per i am selected Logesstic Regression, XGBoost, SVM, Randomforest, Knearesr, Naivebayes for Model Selection the Randomforest is better than other remaining.<jupyter_code>## Hyper Parameter Optimization

# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt','log2']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10,1000,10)]
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10,14]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4,6,8]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              'criterion':['entropy','gini']}


## Hyperparameter optimization using RandomizedSearchCV
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
# from xgboost import XGBClassifier

rf=RandomForestClassifier()
rf_randomcv=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,n_iter=10,cv=2,verbose=2,
                               random_state=100,n_jobs=20)
### fit the randomized model
rf_randomcv.fit(X_train,Y_train)
rf_randomcv.best_estimator_
rdf_clf = RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='entropy', max_depth=120, max_features='sqrt',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=600,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=3, warm_start=False)

rdf_clf.fit(X_train, Y_train)
y_pred = rdf_clf.predict(x_test)
# print confusion-matrix
from sklearn.metrics import confusion_matrix, classification_report
cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)

print('\nTrue Positives(TP) = ', cm[0,0])

print('\nTrue Negatives(TN) = ', cm[1,1])

print('\nFalse Positives(FP){type 1 error} = ', cm[0,1])

print('\nFalse Negatives(FN){type 2 error} = ', cm[1,0])
print('\n\n')

#-------------------------------visualization of confusion matrix -----------------------------------------------------------

# visualize confusion matrix with seaborn heatmap

cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')

#----------------------------------classification report ----------------------------------------------------------------------
print('classification report')
print(classification_report(y_test, y_pred))
print('\n\n')
#-----------------------------------classification accuracy -------------------------------------------------------------------
TP = cm[0,0]
TN = cm[1,1]
FP = cm[0,1]
FN = cm[1,0]

classification_accuracy = (TP + TN) / float(TP + TN + FP + FN)

print('Classification accuracy : {0:0.4f}'.format(classification_accuracy))
print('\n\n')

#----------------------------------------classification error------------------------------------------------------------------

classification_error = (FP + FN) / float(TP + TN + FP + FN)

print('Classification error : {0:0.4f}'.format(classification_error))
print('\n\n')

#----------------------------------------precision-----------------------------------------------------------------------------

precision = TP / float(TP + FP)

print('Precision : {0:0.4f}'.format(precision))
print('\n\n')


#--------------------------------------------Recall---------------------------------------------------------------------------
recall = TP / float(TP + FN)

print('Recall or Sensitivity : {0:0.4f}'.format(recall))
print('\n\n')
#--------------------------------------Truepositive rate------------------------------------------------------------------------

true_positive_rate = TP / float(TP + FN)

print('True Positive Rate : {0:0.4f}'.format(true_positive_rate))
print('\n\n')

#--------------------------------------false positive rate---------------------------------------------------------------------
false_positive_rate = FP / float(FP + TN)

print('False Positive Rate : {0:0.4f}'.format(false_positive_rate))
print('\n\n')
import pickle

#saving the model to the local file system
filename = 'finalized_model.pickle'
pickle.dump(rdf_clf, open(filename, 'wb'))
#saving the scalar to the local file system
filename = 'finalized_scalar.pickle'
pickle.dump(transformer, open(filename, 'wb'))
newBackordDf[predictors]
loaded_scalar = pickle.load(open('finalized_scalar.pickle', 'rb'))
loaded_model = pickle.load(open('finalized_model.pickle', 'rb'))
tranfm_f = loaded_scalar.transform([[16.0,32.0,	10.0,	1.0,	23.0,	33.0,	0.83,	0.85,	8.0]])
loaded_model.predict(tranfm_f)
newBackordDf[target_c]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1754952391375725, 'fraction_numerical': 0.026759330195906754, 'mean_word_length': 3.856497475418549, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 100, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '437174', 'license': 'MIT', 'n_tokens_mistral': 6312, 'n_tokens_neox': 5805, 'n_words': 1375, 'path': 'notebooks/final_model_bkrod_process.ipynb', 'repo_name': 'BALAVIGNESHDOSTRIX/Backorder-prediction-system', 'size': '18274'}