<jupyter_start><jupyter_text>Abstract  In this notebook we will study the famed Iris data set.  This data set has been studied for many years by people doing machine learning.  In fact the dataset was first treated by <NAME>.  He is a rather handsom fellow, and we should be particularily impressed with his goatee!<img src="https://upload.wikimedia.org/wikipedia/commons/e/e0/Hermann_Emil_Fischer_c1895.jpg" style="width:300px">We chose this dataset because it is well studied but also well known to be non-trivial to get good answers from it. It is so well studied that there are even YouTube videos about it.  We can directly embed such a video in our notebook to liven up our presentation.<jupyter_code>from IPython.lib.display import YouTubeVideo
YouTubeVideo('hd1W4CyPX58')<jupyter_output><empty_output><jupyter_text>In fact all of the datasets that we have for this workshop come with their own descriptions.  Note this data set has been studied since 1936 (at least), and there are still open questions.  The data has many interesting features, such as 3 different classes, of varying difficulty and four dimensional features.<jupyter_code>! wget https://raw.githubusercontent.com/rcpaffenroth/DS595-Machine-Learning-for-Engineering-and-Science-Applications/main/examples/basic_python_demos/iris.txt
! wget https://raw.githubusercontent.com/rcpaffenroth/DS595-Machine-Learning-for-Engineering-and-Science-Applications/main/examples/basic_python_demos/iris.csv
print(open('iris.txt', 'r').read())<jupyter_output>1. Title: Iris Plants Database
	Updated Sept 21 by C.Blake - Added discrepency information

2. Sources:
     (a) Creator: <NAME>
     (b) Donor: <NAME> (<EMAIL>)
     (c) Date: July, 1988

3. Past Usage:
   - Publications: too many to mention!!!  Here are a few.
   1. Fisher,R.A. "The use of multiple measurements in taxonomic problems"
      Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions
      to Mathematical Statistics" (John Wiley, NY, 1950).
   2. Duda,R.O., & Hart,P.E. (1973) Pattern Classification and Scene Analysis.
      (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
   3. <NAME>. (1980) "Nosing Around the Neighborhood: A New System
      Structure and Classification Rule for Recognition in Partially Exposed
      Environments".  IEEE Transactions on Pattern Analysis and Machine
      Intelligence, Vol. PAMI-2, No. 1, 67-71.
      -- Results:
         -- very low misclassification rates (0% for t[...]<jupyter_text>Loading in the libraries.<jupyter_code>import numpy as np
import pandas as pa
import sklearn
import matplotlib.pylab as py
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.colors import ListedColormap
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Introduction The first thing that Randy does with data is to take a a look at this, so we thought that we would do the same thing.  So, we begin our analysis by reading in the data.   This was not as trivial as we have originally thought since we kept running into errors.  The problem was that: we needed to make sure the data was in the correct directoryThe eaisiest thing to do is have the data in the same directory at the Jupyter notebook.<jupyter_code>ls
data = pa.read_csv('iris.csv')<jupyter_output><empty_output><jupyter_text>The size of our data was<jupyter_code>data.shape<jupyter_output><empty_output><jupyter_text>Which means we have 150 total measurements and each measurement has 5 values.  For examples our first measurement is<jupyter_code>data.iloc[0,:]<jupyter_output><empty_output><jupyter_text>Observe that we want to predict the flower type from the predictors1. sepal length2. sepal width3. petal length4. petal width  Looking at the data<jupyter_code>X = np.array(data.iloc[:, :-1])
y = np.array(data.iloc[:, -1])
py.figure()
py.plot(X[y==0,0],X[y==0,1],'r.')
py.plot(X[y==1,0],X[y==1,1],'g.')
py.plot(X[y==2,0],X[y==2,1],'b.')
py.figure()
py.plot(X[y==0,2],X[y==0,3],'r.')
py.plot(X[y==1,2],X[y==1,3],'g.')
py.plot(X[y==2,2],X[y==2,3],'b.')
pa.plotting.scatter_matrix(data, alpha=0.2, figsize=(8,8));
fig = py.figure(figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)
ax.scatter(X[y==0, 0], X[y==0, 1], X[y==0, 2], c='r')
ax.scatter(X[y==1, 0], X[y==1, 1], X[y==1, 2], c='g')
ax.scatter(X[y==2, 0], X[y==2, 1], X[y==2, 2], c='b')<jupyter_output><empty_output><jupyter_text>Data processing  Asking the right question One of the most important aspects of data analysis is to ask the right question. So, we thought that we would preprocess our data to ask two different questions, one we think is easy, and one we think is hard.  The easy question we propose is tojust distinguish class 0 from classes 1 and 2.  Based upon our visualizations, we assume that this problem willl be very easy to solve using the petal lengths and petal widths<jupyter_code>py.figure()
py.plot(X[y==0,2],X[y==0,3],'r.')
py.plot(X[y!=0,2],X[y!=0,3],'g.')<jupyter_output><empty_output><jupyter_text>We also plan to test the sepal widths and the sepal lengths, even though that problem will likely be harder.<jupyter_code>py.figure()
py.plot(X[y==0,0],X[y==0,1],'r.')
py.plot(X[y!=0,0],X[y!=0,1],'g.')<jupyter_output><empty_output><jupyter_text>Finally we will test the full data set (all four columns) against all three classes.  However, plotting this data is rather difficult, so we needed to try something more advanced.  Randy suggested Principal Component Analysis, so we gave that a try.   Principle Component Analysis (PCA)<jupyter_code>from sklearn.decomposition import PCA

py.figure()
XPCA = PCA(n_components=3).fit_transform(X)
py.plot(XPCA[y==0,0],XPCA[y==0,1],'r.')
py.plot(XPCA[y==1,0],XPCA[y==1,1],'g.')
py.plot(XPCA[y==2,0],XPCA[y==2,1],'b.')<jupyter_output><empty_output><jupyter_text>Just as one can project from a high dimensional space to a two-dimensional space, one can also do the same thing to project to a three-dimensional space.  Our first classification tool, K-Nearest Neighbors. http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.htmln_neighbors : int, optional (default = 5)Number of neighbors to use by default for k_neighbors queries.weights : str or callable, optional (default = ‘uniform’)weight function used in prediction. Possible values:‘uniform’ : uniform weights. All points in each neighborhood are weighted equally.‘distance’ : weight points by the inverse of their distance. in this case, closer neighbors of a query point will have a greater influence than neighbors which are further away.[callable] : a user-defined function which accepts an array of distances, and returns an array of the same shape containing the weights.metric : string or DistanceMetric object (default = ‘minkowski’)the distance metric to use for the tree. The default metric is minkowski, and with p=2 is equivalent to the standard Euclidean metric. See the documentation of the DistanceMetric class for a list of available metrics. p : integer, optional (default = 2)Power parameter for the Minkowski metric. When p = 1, this is equivalent to using manhattan_distance (l1), and euclidean_distance (l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.metric_params : dict, optional (default = None)Additional keyword arguments for the metric function.<jupyter_code># # Import the K-NN solver
from sklearn import neighbors
clf = neighbors.KNeighborsClassifier(n_neighbors=3)<jupyter_output><empty_output><jupyter_text>http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.htmlC : float, optional (default=1.0)Penalty parameter C of the error term.loss : string, ‘hinge’ or ‘squared_hinge’ (default=’squared_hinge’)Specifies the loss function. ‘hinge’ is the standard SVM loss (used e.g. by the SVC class) while ‘squared_hinge’ is the square of the hinge loss.penalty : string, ‘l1’ or ‘l2’ (default=’l2’)Specifies the norm used in the penalization. The ‘l2’ penalty is the standard used in SVC. The ‘l1’ leads to coef_ vectors that are sparse.class_weight : {dict, ‘balanced’}, optionalSet the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))<jupyter_code># # Load in a classifier
#from sklearn import svm

#clf = svm.LinearSVC()<jupyter_output><empty_output><jupyter_text>http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.htmlsklearn.tree.DecisionTreeClassifiercriterion : string, optional (default=”gini”)The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain.max_depth : int or None, optional (default=None)The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.min_samples_split : int, float, optional (default=2)The minimum number of samples required to split an internal node:If int, then consider min_samples_split as the minimum number.If float, then min_samples_split is a percentage and ceil(min_samples_split * n_samples) are the minimum number of samples for each split.min_samples_leaf : int, float, optional (default=1)The minimum number of samples required to be at a leaf node:If int, then consider min_samples_leaf as the minimum number.If float, then min_samples_leaf is a percentage and ceil(min_samples_leaf * n_samples) are the minimum number of samples for each node.max_leaf_nodes : int or None, optional (default=None)Grow a tree with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.class_weight : dict, list of dicts, “balanced” or None, optional (default=None)Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))For multi-output, the weights of each column of y will be multiplied.Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified.<jupyter_code>## Decision tree classifier
#from sklearn import tree

#clf = tree.DecisionTreeClassifier()<jupyter_output><empty_output><jupyter_text>n_estimators : integer, optional (default=10)The number of trees in the forest.<jupyter_code># # Decision tree classifier
#from sklearn import ensemble

#clf = ensemble.RandomForestClassifier()<jupyter_output><empty_output><jupyter_text>http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.htmln_components : int, optionalNumber of components (< n_classes - 1) for dimensionality reduction.<jupyter_code># # Linear Disciminant Analysis
#from sklearn import discriminant_analysis

#clf = discriminant_analysis.LinearDiscriminantAnalysis()
# This is the one complicated bit of code in the whole demo.  There is no need to modify it, but please feel free if you want!
def runTest(clf,X,y,trainingPercent=0.66,plotTest=True):
    # A little cheat to make the pictures consistent
    np.random.seed(1234)
    
    # If there is one thing that I want to harp on, it is the difference
    # between testing and training errors!  So, here we create a training
    # set on which we computer the parameters of our algorithm, and a 
    # testing set for seeing how well we generalize (and work on real 
    # world problems).
    perm = np.random.permutation(len(y))
    n = X.shape[0]
    trainSize = int(trainingPercent*n)
    Xtrain = X[perm[:trainSize],0:2]
    Xtest = X[perm[trainSize:],0:2]

    yHat = y
    yHattrain = yHat[perm[:trainSize]]
    yHattest = yHat[perm[trainSize:]]

    # Run the calculation!
    clf.fit(Xtrain, yHattrain)
    
    # step size in the mesh for plotting the decision boundary.
    h = .02  
    # Plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, m_max]x[y_min, y_max].
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                             np.arange(y_min, y_max, h))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)

    py.figure(figsize=(8, 6))
    cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
    cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])
    py.pcolormesh(xx, yy, Z, cmap=cmap_light, shading='auto')

    # Plot also the training points
    py.scatter(Xtrain[:, 0], Xtrain[:, 1], c=yHattrain, cmap=cmap_bold,marker='o')
    if plotTest:
        py.scatter(Xtest[:, 0], Xtest[:, 1], c=yHattest, cmap=cmap_bold,marker='+')
    py.xlim(xx.min(), xx.max())
    py.ylim(yy.min(), yy.max())
    # py.show()

    # Print out some metrics
    print('training score',clf.score(Xtrain,yHattrain))
    print('testing score',clf.score(Xtest,yHattest))<jupyter_output><empty_output><jupyter_text>The problem we though would be easy was easy!<jupyter_code>yEasy = y.copy()
yEasy[y==2] = 1
runTest(clf,X[:,[2,3]],yEasy)<jupyter_output>training score 1.0
testing score 1.0<jupyter_text>The problem we thought would be a bit harder was still pretty easy.<jupyter_code>yEasy = y.copy()
yEasy[y==2] = 1
runTest(clf,X[:,[0,1]],yEasy)<jupyter_output>training score 0.98989898989899
testing score 1.0<jupyter_text>Testing the hypothesis that the problem we thought was hard actually is hard The hard problem is hard, and we pretty clearly overfit!<jupyter_code>runTest(clf,X[:,[0,1]],y,plotTest=False)<jupyter_output>training score 0.8585858585858586
testing score 0.8235294117647058<jupyter_text>However, Randy won't be mad since we figured out we were overfitting!  A better way of analyzing the data Thinking about what must have to been going on, we decided to try a different projection of the data.<jupyter_code>runTest(clf,X[:,[2,3]],y)<jupyter_output>training score 0.9797979797979798
testing score 0.9607843137254902<jupyter_text>This is much better!   The trainig and testing errors are much closer.    More advanced techniques and other explorations Getting just a bit fancier, we actually do PCA!!<jupyter_code>runTest(clf,XPCA[:,[0,1]],y)<jupyter_output>training score 1.0
testing score 0.9607843137254902<jupyter_text>What happens if we reduce the training data by a lot?<jupyter_code>runTest(clf,XPCA[:,[0,1]],y, trainingPercent = 0.2)<jupyter_output>training score 0.9666666666666667
testing score 0.9<jupyter_text>Of course, that makes the results much worse!<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10364593747917084, 'fraction_numerical': 0.025861494367793107, 'mean_word_length': 5.1265822784810124, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 88, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '509166', 'license': 'MIT', 'n_tokens_mistral': 5084, 'n_tokens_neox': 4750, 'n_words': 1677, 'path': 'examples/basic_python_demos/MachineLearning_different_methods.ipynb', 'repo_name': 'rcpaffenroth/-Machine-Learning-for-Engineering-and-Science-Applications', 'size': '15052'}