<jupyter_start><jupyter_text>___Enter Team Member Names here (*double click to edit*):- Name 1:<NAME>- Name 2:<NAME>- Name 3:<NAME>- Name 4:<NAME>___  In Class Assignment FourIn the following assignment you will be asked to fill in python code and derivations for a number of different problems. Please read all instructions carefully and turn in the rendered notebook (or HTML of the rendered notebook)  before the end of class (or right after class). The initial portion of this notebook is given before class and the remainder is given during class. Please answer the initial questions before class, to the best of your ability. Once class has started you may rework your answers as a team for the initial part of the assignment.  Contents* Loading KDDCup Data* KDDCup Evaluation and Cross Validation* More Cross Validation* Statistical Comparison**Before coming to class**, please make sure you have the latest version of `scikit-learn`. This notebook was created for version 0.18 and higher. ________________________________________________________________________________________________________Back to Top Loading KDDCup DataPlease run the following code to read in the "KDD Cup" dataset from sklearn's data loading module. It consists of examples of different simulated attacks for the 1998 DARPA Intrusion Detection System (IDS). This will load the data into the variable `ds`. `ds` is a `bunch` object with fields like `ds.data` and `ds.target`. The field `ds.data` is a numpy matrix of the continuous features in the dataset. **The object is not a pandas dataframe. It is a numpy matrix.** Each row is a set of observed instances, each column is a different feature. It also has a field called `ds.target` that is an integer value we are trying to predict (i.e., a specific integer represents a specific person). Each entry in `ds.target` is a label for each row of the `ds.data` matrix.<jupyter_code># fetch the dataset
from sklearn.datasets import  fetch_kddcup99
from sklearn import __version__ as sklearn_version

print('Sklearn Version:',sklearn_version)
ds =  fetch_kddcup99(subset='http')
import numpy as np
# get some of the specifics of the dataset
X = ds.data
y = ds.target != b'normal.'

n_samples, n_features = X.shape
n_classes = len(np.unique(y))

print("n_samples: {}".format(n_samples))
print("n_features: {}".format(n_features))
print("n_classes: {}".format(n_classes))
unique, counts = np.unique(y, return_counts=True)

np.asarray((unique, counts)).T<jupyter_output><empty_output><jupyter_text>___**Question 1:** How many instances are in the binary classification problem loaded above? How many instances are in each class? **Plot a pie chart or bar chart of the number of instances in each of the classes.**<jupyter_code>from matplotlib import pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

plt.pie(counts, labels=np.unique(y), startangle=90, autopct='%.1f%%')
plt.show()
#=== Fill in code below========
print('Number of instances in each class:',np.asarray((unique, counts)).T)<jupyter_output><empty_output><jupyter_text>Back to Top KDDCup Evaluation and Cross Validation<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold, ShuffleSplit
from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit

from sklearn.metrics import make_scorer, accuracy_score
from sklearn.metrics import precision_score, recall_score, f1_score

from sklearn.linear_model import LogisticRegression

# select model
clf = LogisticRegression(solver='lbfgs')
#select cross validation
cv = KFold(n_splits=10)
# select evaluation criteria
my_scorer = make_scorer(accuracy_score)
# run model training and cross validation
per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=cv,
                                    scoring=my_scorer
                                   )
plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])<jupyter_output><empty_output><jupyter_text>____**Question 2** Is the code above a proper separation of training and testing sets for the given dataset (i.e., using KFold)? Why or why not?  *Enter your answer here (double click)**Yes or No and why*NoThere are two reasons :1- because the data is split into 10 fold without making sure that each fold has the same percantage of the two classes like the orignal one2-the classfier should be trained on any 9 fold andtestedon the remaining one fold ___ **Question 3:** Is the evaluation metric chosen in the above code appropriate for the dataset (i.e., using accuracy)? Why or Why not? *Enter your answer here (double click)**Yes or No and why* Nobecause the prediction might have a high accuracy with a high variance , so we need to look at the accuracy variance and the bias___ **Exercise 1:** If the code above is not a proper separation of the train or does not use the proper evaluation criteria, fix the code in the block below to use appropriate train/test separation and appropriate evaluation criterion (criteria). **Mark changes in the code with comments.**<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold, ShuffleSplit
from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit

from sklearn.metrics import make_scorer, accuracy_score
from sklearn.metrics import precision_score, recall_score, f1_score

from sklearn.linear_model import LogisticRegression
# these imports above might help you



#=====Write your code below here=================
# select model
clf = LogisticRegression(solver='lbfgs')
#select cross validation
#cv = KFold(n_splits=10)
rs=StratifiedShuffleSplit(n_splits=10, random_state=1)
# select evaluation criteria
my_scorer = make_scorer(recall_score)
# run model training and cross validation
per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=rs,
                                    scoring=my_scorer
                                   )

plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])<jupyter_output><empty_output><jupyter_text>____**Question 4**: Does the learning algorithm perform well based on the evaluation criteria? Why or why not? Yes it does, because we presrved the percentage of samples for each class in each fold. Back to Top More Cross Validation**Exercise 2:** Does the code below contain any errors in the implementation of the cross validation? If so, fix the code below so that there are no longer any errors in the cross validation. there is two erorrs in the code :1- the best practice is to first scale data then do dimensionalty reduction and after that start our learning algrothem 2-use StratifiedShuffleSplit to presrved the percentage of samples for each class<jupyter_code>from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline


#======If there are errors, fix them below======
n_components = 1
pca = PCA(n_components=n_components)
Xpca = pca.fit_transform(X)

clf = Pipeline([('scl',StandardScaler()),
                ('clf',LogisticRegression(solver='lbfgs'))])

per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=Xpca,
                                    y=y,
                                    cv=cv,
                                    scoring=my_scorer
                                   )

#plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])

# =====fixed code======
# write the fixed code (if needed) below

clf = Pipeline([('scl',StandardScaler()),
                ('pca',PCA(n_components=n_components)),
                ('clf',LogisticRegression(solver='lbfgs'))])

per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=rs,
                                    scoring=my_scorer
                                   )

plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])<jupyter_output>C:\Anaconda\lib\site-packages\sklearn\utils\validation.py:595: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
C:\Anaconda\lib\site-packages\sklearn\utils\validation.py:595: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
C:\Anaconda\lib\site-packages\sklearn\utils\validation.py:595: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
C:\Anaconda\lib\site-packages\sklearn\utils\validation.py:595: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
C:\Anaconda\lib\site-packages\sklearn\utils\validation.py:595: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConv[...]<jupyter_text>___ Circumstances ChangeFor this question, the circumstances for the DARPA KDD99 cup are changed in the following way:- When the model for detecting attacks is deployed, we now think that it will often need to be retrained because new attacks will emerge. - DARPA anticipates that there will be a handful of different style attacks on their systems that have never been seen before. To detect these new attacks, they are employing programmers and analysts to find them manually every day.  - DARPA believes the perpetrators of these new attacks are more sophisticated, so finding the new attacks will take priority over detecting the older, known attacks. - DARPA wants to use your learning algorithm for detecting only these new attacks. However, they tell you that the amount of training data for the new attacks will be extremely small. That is, the analysts can only identify a handful of new style attacks each day (so you will only have about 3-5 examples of the attacks for training).- **DARPA asks you: Do you think its a good idea to employ retraining your model each day to find these new attacks?** They need an answer in the next 20 minutes.  **Question 5**: How would you change the method of cross validation to answer this question from DARPA? That is, how can you change your cross validation method to better mirror how your system will be used and deployed by DARPA when there are only 3-5 attack examples avaliable for training? *Note: you do not have access to these new training examples. You need to change you cross validation method with the existing data to answer this question.*  In order to accomodate these new attacks that are produced at a rate of about 3-5 a day, we would alter our cross-validation  in the following manner. We would need to create many, many more folds with substanstially less volume in each fold. Each fold would contain the 3-5 examples of the new attacks, along with 5 or so random samples from the larger dataset (to include old attacks and benign instances. Using this lower-volume-fold relationship, we would train our classifer and the result would essentially be an algorithm that skews heavily toward these new attacks, but would produce substantially more false positives. The decision of whether this would be an acceptable hindrance would be up to DARPA. They undoubtedly would consider false positives to be of less concern than false negatives, but if typical usage of the system slows to an intolerable crawl, other methods of detection may be necessary.One alternative could be to run two separate models, one trained heavily on the newer attacks and the other against the traditional, more conventional styled attacks. Depending on the accuracies of both classifiers, one could associate a confidence relationship that reduces false positives/negatives by combining their predictions via a weighted relationship based on accuracy.___<jupyter_code>#plotting function for use in next question
# takes input 'test_scores', axis labels, and an x-axis label
def plot_filled(test_scores,train_x_axis, xlabel=''):
    
    test_mean = np.percentile(test_scores,50, axis=1)
    test_max = np.percentile(test_scores,95, axis=1) 
    test_min = np.percentile(test_scores,5, axis=1) 

    plt.plot(train_x_axis, test_mean,
             color='blue', linestyle='--',
             marker='s', markersize=5,
             label='validation set')

    plt.fill_between(train_x_axis,
                     test_min,
                     test_max,
                     alpha=0.15, color='blue')

    plt.grid(True)
    plt.xlabel(xlabel)
    plt.ylabel('Evaluation Criterion')
    plt.legend(loc='lower right')
    plt.tight_layout()<jupyter_output><empty_output><jupyter_text>___DARPA is also concerned about how much training data they will need from the analysts in order to have a high performing model. They would like to use the current dataset to help answer that question. The code below is written for you to help answer DARPA's question about how many examples will be needed for training. Examine the code and the output that is already run for you, then answer the following question:**Question 6**: Based on the analysis graphed below, how many positive examples are required to have a good tradeoff between bias and variance for the given evaluation criteria? Why? *Note: the x-axis really is a percentage, so the value 0.1 is actually 0.1%.*<jupyter_code>clf = LogisticRegression(solver='lbfgs')

test_scores = []
train_sizes=np.linspace(5e-4,5e-3,10)

for size in train_sizes:
    cv = StratifiedShuffleSplit(n_splits=100,
                                train_size = size,
                                test_size = 1-size,
                               )
    test_scores.append(cross_val_score(estimator=clf,X=X,y=y,cv=cv,scoring=my_scorer))

plot_filled(np.array(test_scores), train_sizes*100, 'Percentage training data (%)')<jupyter_output><empty_output><jupyter_text>___*Enter you answer here (double click)*It seems that .3% (177 samples) is the best point, because the accuracy is the maxmum at this point and the variance in the lowesthowever as showen above after .2% the variance did not change at all, so if we are intrested in minmizing the data size , we can say that .2% is a good size in terms of accuracy since we only got a .002 % increase when going from .2% data size to .3% data size___ ___Back to Top Statistical ComparisonNow lets create a few different models and see if any of them have statistically better performances. We are creating three different classifiers below to compare to one another. For creating different training and testing splits, we are using stratified shuffle splits on the datasets.<jupyter_code>clf1 = LogisticRegression(C=100, solver='lbfgs')
clf2 = LogisticRegression(C=1, solver='lbfgs')
clf3 = LogisticRegression(C=0.1, solver='lbfgs')

train_size = 0.003 # small training size
cv = StratifiedShuffleSplit(n_splits=10,train_size=train_size,test_size=1-train_size)

evals1 = cross_val_score(estimator=clf1,X=X,y=y,scoring=my_scorer,cv=cv)
evals2 = cross_val_score(estimator=clf2,X=X,y=y,scoring=my_scorer,cv=cv)
evals3 = cross_val_score(estimator=clf3,X=X,y=y,scoring=my_scorer,cv=cv)<jupyter_output><empty_output><jupyter_text>**Question 7**: Given the code above, what statistical test is more appropriate for selecting confidence intervals, and **why**? Your options are:- **A**: approximating the evaluation criterion as a binomial distribution and bounding by the variance (the first option we used in the flipped lecture video)- **B**: approximating the bounds using the folds of the cross validation to get mean and variance (the second option we used in the flipped lecture video)- **C**: Either are acceptable statistical tests for obtaining confidence intervals The reason that I think (B) is correct is because the three classifiers are not independent but they all are correlated into each other, since they all using the same data this means that if classifier 1 did slightly better then classifier 2 would do slightly better and same goes to classfier 3   ___**Final Exercise:** With 95% confidence, perform the statistical test that you selected above. Is any model or set of models statistically the best performer(s)? Or can we not say if the models are different with greater than 95% confidence?If you chose option A, use a multiplier of Z=1.96. The number of instances used in testing can be calculated from the variable `train_size`.If you chose option B, use a multiplier of t=2.26 and k=10.<jupyter_code>#===================================================
# Enter your code below

clf1 = LogisticRegression(C=100, solver='lbfgs')
clf2 = LogisticRegression(C=1, solver='lbfgs')
clf3 = LogisticRegression(C=0.1, solver='lbfgs')

train_size = 0.003 # small training size
cv = StratifiedShuffleSplit(n_splits=10,train_size=train_size,test_size=1-train_size)

evals1 = cross_val_score(estimator=clf1,X=X,y=y,scoring=my_scorer,cv=cv)
evals2 = cross_val_score(estimator=clf2,X=X,y=y,scoring=my_scorer,cv=cv)
evals3 = cross_val_score(estimator=clf3,X=X,y=y,scoring=my_scorer,cv=cv)


test_scores1 = []
test_scores1.append(evals1)

test_scores2 = []
test_scores2.append(evals2)

test_scores3 = []
test_scores3.append(evals3)

e1=1-np.array(test_scores1)
e2=1-np.array(test_scores2)
e3=1-np.array(test_scores3)


t=2.26/np.sqrt(10)

# comparing E1 with E2----------------------
e1e2 = []
e1e2=e1[0]-e2[0]
e1e2_m=np.mean(e1e2)
std12=np.std(e1e2)
var1=np.mean(e1e2)
conf12=t*std12


print('Mean error: ', e1e2_m)
print ('Range of: ', var1-conf12, var1+conf12)
print('Because the mean of the difference is positive between and the confidence interval does not cross zero, this indicates that classfier 2 is more accurate than classfier 1.')
print('')

# comparing E2 with E3-------------------------
e2e3=e2[0]-e3[0]
e2e3_m=np.mean(e2e3)
std23=np.std(e1e2)
var2=np.mean(e2e3)



conf23=t*std23
print('Mean error: ', e2e3_m)
print ('Range of: ', var2-conf12, var2+conf12)
print('Because the mean of the difference is negative between and the confidence interval does not cross zero, this indicates that classfier 2 is more accurate than classfier 3.')
print(' ')



print('Model 2 has statistically the best accuracy (', np.mean(evals2), ') with 95% confidence.')
#===================================================<jupyter_output>Mean error:  0.00047481596611385066
Range of:  -0.0005025196295208773 0.0014521515617485787
Because the mean of the difference is positive between and the confidence interval does not cross zero, this indicates that classfier 2 is more accurate than classfier 1.

Mean error:  -0.00014346957249476987
Range of:  -0.0011208051681294978 0.000833866023139958
Because the mean of the difference is negative between and the confidence interval does not cross zero, this indicates that classfier 2 is more accurate than classfier 3.
 
Model 2 has statistically the best accuracy ( 0.999882149994022 ) with 95% confidence.
{'fraction_non_alphanumeric': 0.08080704629250307, 'fraction_numerical': 0.02212208111429742, 'mean_word_length': 4.289544962080173, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '86201', 'license': 'Unlicense', 'n_tokens_mistral': 5757, 'n_tokens_neox': 5369, 'n_words': 2185, 'path': 'ICA/ICA4_MachineLearning_3.23.19.ipynb', 'repo_name': 'wiggs555/cse7324project', 'size': '19555'}