<jupyter_start><jupyter_text>$\Large{\text{Ensemble methods}}$<jupyter_code>import numpy as np #numpy package will be useful for most of the array operations in the code 
# from sklearn.datasets import load_wine
"""
Loading data set
"""
from sklearn.datasets import load_svmlight_file

myfeature, y = load_svmlight_file('diabetes')

"""
converting sparse matrix myfeature (which is in tuple form) to a dense matrix
"""

X = myfeature.todense()
X = np.array(X)
# print (y)
# print (X[0])
#Let us print the contents of the X_y variable 

print(type(X))
# print(X_y)
#So we found that the X_y variable is of type sklearn.utils.bunch
#further, we see that the X_y variable consists of dictionary-type structures named "data",  "target" and "target_names"

# Let us now segregate the data attributes and class labels into separate variables 

# X = X_y.data
# y = X_y.target

print('X shape:', X.shape, 'type(X):', type(X) )
print(' y shape:', y.shape, ' type(y):', type(y))
#Thus we see that X and y are numpy ndarrays. 
#We can print the unique labels available in y 
print(np.unique(y))
# print (y)<jupyter_output>[-1.  1.]<jupyter_text>Consider an input space $\mathcal{X} \subseteq {\mathbb{R}}^d$ and the output space $\mathcal{Y} = \{+1,-1\}$. Assume a fixed (but unknown) distribution $P(X,Y)$ over $\mathcal{X} \times \mathcal{Y}$. Let us assume that there are 15 base classifiers $C_1, C_2, \ldots, C_{15},$ where each classifier has an error rate of $\epsilon = 0.25$ on some sample from a fixed data distribution $P(X,Y)$. To predict the label for a test sample $\hat{x}$, we adopt the following inference procedure (called $\textbf{Ensemble Classifier}(\textbf{EC})$):1.   Predict $C_i(\hat{x})$ using each classifier $C_i, \ i \in \{1,2,\ldots,15\}$.2.   Predict the final label $\hat{y} = \arg\max_{y \in \mathcal{Y}} \sum_{i=1}^{15} \delta(y==C_i(\hat{x}))$ where $\delta(p)$ is the indicator function given by:  $  \delta(p) =   \begin{cases}  1 \text{ if } p \text{ is true. } \\   0 \text{ if } p \text{ is false. }    \end{cases}  $$\textbf{Question:}$ What would be error rate of the classifier obtained from the above inference algorithm $\textbf{EC}$?  $\textbf{One possible answer:}$Suppose the classifiers are assumed to be independent, then the $\textbf{EC}$ classifier would make an error only when more than half of the classifiers (i.e. more than 7 classifiers) make error in the prediction. Hence  we may write the error rate of $\textbf{EC}$ as:$\begin{align}\text{error}_{\textbf{EC}} = \sum_{i=8}^{15} \begin{pmatrix} 15 \\  i \end{pmatrix} {\epsilon}^{i} (1-\epsilon)^{15-i} \end{align}$which is approximately $0.017$. Note that this error rate is considerably smaller than the individual error rates of the classifiers.<jupyter_code>#compute the error rates of EC for different error rates of C_i
import math 
import matplotlib.pyplot as plt

def comb(n, k):
    return math.factorial(n) // math.factorial(k) // math.factorial(n - k)


epsilons = np.linspace(0,1,11)
errors = np.zeros(epsilons.shape)

num_classifiers = 15
eps_idx=0
for epsilon in epsilons:
  error_EC = 0
  for j in np.arange(np.ceil(num_classifiers/2),num_classifiers+1):
    err = comb(num_classifiers,j)*math.pow(epsilon,j)*math.pow(1-epsilon,num_classifiers-j)
    error_EC += err
  errors[eps_idx] = error_EC
  eps_idx+=1 


figure, ax = plt.subplots(1) 

plt.plot(epsilons, errors, marker='o')
ax.set_xlabel('$\epsilon$')
ax.set_ylabel('$Error_{EC}$')
ax.set_xlim(0, 1)
ax.set_ylim(errors.min()-0.1,errors.max()+0.1)
ax.set_xticks(epsilons)
ax.set_yticks(np.linspace(0,1,11))
ax.set_title('Error rate of Ensemble Classifier vs $\epsilon$')
plt.show()<jupyter_output><empty_output><jupyter_text>$\textbf{Important to note:}$ 1.   The base classifiers $C_1, C_2, \ldots, C_{15}$ are assumed to be independent. 2.   The error rate $\epsilon$ of each base classifier must be less than $0.5$ for the ensemble classifier to behave better. $\textbf{What is meant by a base classifier having an error rate less than } 0.5$? $\large{\text{Ways of building an ensemble classifier}}$$\textbf{Create multiple data partitions from training data}$1.   Resample the original training data $D$ (using sampling with replacement) and create different data paritions $D_1, D_2, \ldots, D_M$. 2.   Train different classifiers $C_i$ on respective data partition $D_i$, $i \in \{1,2,\ldots,M\}$.3.   For a test data point $\hat{x}$ predict the label $\hat{y}=\text{MajorityVote}(C_1(\hat{x}),C_2(\hat{x}), \ldots, C_M(\hat{x}))$.<jupyter_code>from urllib.request import urlopen
from PIL import Image

img = Image.open(urlopen('https://github.com/balamurugan-palaniappan-CEP/AIML_CEP_2021/raw/main/images/ensemble_classifier.png'))
img
np.random.seed(1000)
#Create an index array 
num_samples = X.shape[0]
# print (num_samples)
indexarr = np.arange(num_samples) #index array
np.random.shuffle(indexarr) #shuffle the indices 
#print('shuffled indices of samples:')
#print(indexarr)
#Use the samples corresponding to first 60% of indexarr for training 
num_train = int(0.8*num_samples)
#Use the remaining 20% samples for testing 
num_test = num_samples-num_train
print('num_train: ',num_train, 'num_test: ', num_test)
#Use the first 80% of indexarr to create the train data features and train labels 
train_X = X[indexarr[0:num_train]]
train_y = y[indexarr[0:num_train]]
print('shape of train data features:')
print(train_X.shape)
print('shape of train data labels')
print(train_y.shape)
#Use remaining 20% of indexarr to create the test data and test labels  
test_X = X[indexarr[num_train:num_samples]]
test_y = y[indexarr[num_train:num_samples]]
print('shape of test data features:')
print(test_X.shape)
print('shape of test data labels')
print(test_y.shape)
#Let us now use five different base classifiers 
from sklearn.svm import LinearSVC #import linear SVM
from sklearn.neighbors import KNeighborsClassifier
from sklearn import tree #decision tree from scikit learn 
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
# creating an object of LogisticRegression class

clf_list = [] 

clf_linearsvc = LinearSVC(C=0.01)
clf_neigh = KNeighborsClassifier(n_neighbors=3, metric='manhattan') #weights='uniform' (default) or 'distance'
clf_svc = SVC(kernel='rbf', gamma=0.01)
clf_tree = tree.DecisionTreeClassifier(criterion='gini')
clf_logit = LogisticRegression(C=1.0) # C is set to be large number in order to remove the inbuilt regularization 

clf_list.append(clf_linearsvc)
clf_list.append(clf_neigh)
clf_list.append(clf_svc)
clf_list.append(clf_tree)
clf_list.append(clf_logit)

clf_names = ['Linear SVM','Nearest Neighbors', 'Kernel SVM','Decision Tree','Logistic Regression']
num_classifiers = 5


for i in range(num_classifiers):
  print('classifier:',clf_list[i])
  indexarr = np.arange(train_X.shape[0]) #index array for train data
  np.random.shuffle(indexarr) #shuffle the indices 
  #we shall choose 70% of the data
  partition_prop = 0.7
  num_samples_partition = int(partition_prop*train_X.shape[0])

  X_partition = train_X[indexarr[0:num_samples_partition]]
  y_partition = train_y[indexarr[0:num_samples_partition]]

  base_clf = clf_list[i] 
  base_clf_model = base_clf.fit(X_partition,y_partition.ravel())


  #test accuracy
  from sklearn.metrics import accuracy_score
  test_y_predicted = base_clf_model.predict(test_X)
  test_acc = accuracy_score(test_y, test_y_predicted)
  print('test accuracy from classifier:',clf_names[i],' is:', test_acc)

  if i == 0:
    test_pred_all_clf = base_clf_model.predict(test_X)

  elif i ==1:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i==2:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )
  
  elif i ==3:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==4:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

test_pred_all_clf = np.array(test_pred_all_clf)
print(test_pred_all_clf.shape)
## test predictions by 5 base classifiers. The last column contains actual test label. 
print(np.column_stack( (test_pred_all_clf, test_y)) )

from scipy import stats

test_pred_all_clf = test_pred_all_clf.astype(int)

test_pred, counts = stats.mode(test_pred_all_clf, axis=1)
test_acc = accuracy_score(test_y, test_pred)
print('test accuracy from ensemble classifier is:', test_acc)



## creating confusing matrix

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(test_y, test_pred)
print(cm)

#Plotting the confusion matrix for better interpretation
import seaborn as sns
import matplotlib.pyplot as plt
ax = plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax)
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(['-1', '1'])
ax.yaxis.set_ticklabels(['-1', '1'])<jupyter_output><empty_output><jupyter_text>$\large{\text{Ways of building an ensemble classifier}}$$\textbf{Create multiple attribute partitions from training data}$1.   Resample the attributes from original training data $D$ (using sampling with replacement) and create different feature paritions $F_1, F_2, \ldots, F_M$. Note that the number of samples in these partitions might be same as that in $D$ or might be different.2.   Train different classifiers $C_i$ on respective feature partition $F_i$, $i \in \{1,2,\ldots,M\}$.3.   For a test data point $\hat{x}$ first create feature partitions based on $F_1, F_2, \ldots, F_M$ and predict the label $\hat{y}=\text{MajorityVote}(C_1(F_1(\hat{x})),C_2(F_2(\hat{x})), \ldots, C_M(F_M(\hat{x})))$.<jupyter_code>from urllib.request import urlopen
from PIL import Image

img = Image.open(urlopen('https://github.com/balamurugan-palaniappan-CEP/AIML_CEP_2021/raw/main/images/ensemble_classifier_RF.png'))
img
#Let us now use three different base classifiers and check the decision boundary 
from sklearn.ensemble import RandomForestClassifier

clf_list = [] 

clf_linearsvc = LinearSVC(C=1.0)
clf_neigh = KNeighborsClassifier(n_neighbors=2, metric='manhattan') #weights='uniform' (default) or 'distance'
clf_svc = SVC(kernel='rbf', gamma=0.01)
# clf_tree = tree.DecisionTreeClassifier(criterion='entropy')
clf_rf = RandomForestClassifier(n_estimators = 100,  random_state=0)
clf_logit = LogisticRegression(C=1.0) # C is set to be large number in order to remove the inbuilt regularization 

clf_list.append(clf_linearsvc)
clf_list.append(clf_neigh)
clf_list.append(clf_svc)
clf_list.append(clf_logit)
clf_list.append(clf_rf)

clf_names = ['Linear SVM','Nearest Neighbors', 'Kernel SVM','Logistic Regression','Random Forest']
num_classifiers = 5


for i in range(num_classifiers):
  print('classifier:',clf_list[i])
  indexarr = np.arange(train_X.shape[0]) #index array for train data
  np.random.shuffle(indexarr) #shuffle the indices 
  #we shall choose 70% of the data
  partition_prop = 0.7
  num_samples_partition = int(partition_prop*train_X.shape[0])

  X_partition = train_X[indexarr[0:num_samples_partition]]
  y_partition = train_y[indexarr[0:num_samples_partition]]

  base_clf = clf_list[i] 
  base_clf_model = base_clf.fit(X_partition,y_partition.ravel())


  #test accuracy
  from sklearn.metrics import accuracy_score
  test_y_predicted = base_clf_model.predict(test_X)
  test_acc = accuracy_score(test_y, test_y_predicted)
  print('test accuracy from classifier:',clf_names[i],' is:', test_acc)

  if i == 0:
    test_pred_all_clf = base_clf_model.predict(test_X)

  elif i ==1:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i==2:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )
  
  elif i ==3:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==4:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )
test_pred_all_clf = np.array(test_pred_all_clf)
print(test_pred_all_clf.shape)

from scipy import stats

test_pred_all_clf = test_pred_all_clf.astype(int)

test_pred, counts = stats.mode(test_pred_all_clf, axis=1)
test_acc = accuracy_score(test_y, test_pred)
print('test accuracy from ensemble classifier is:', test_acc)


## creating confusing matrix

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(test_y, test_pred)
print(cm)

#Plotting the confusion matrix for better interpretation
import seaborn as sns
import matplotlib.pyplot as plt
ax = plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax)
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(['-1', '1'])
ax.yaxis.set_ticklabels(['-1', '1'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12108854207513137, 'fraction_numerical': 0.013959689436122658, 'mean_word_length': 5.614107883817427, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '578604', 'license': 'CC0-1.0', 'n_tokens_mistral': 4666, 'n_tokens_neox': 4499, 'n_words': 1209, 'path': 'Ensemble Methods/EnsembleMethods_TA_session_13Nov2021_diabetes.ipynb', 'repo_name': 'arvind-maurya/AIML_CEP_2021', 'size': '12751'}