<jupyter_start><jupyter_text>The faces are stored in `faces_all.txt` as an array of integers indicating the grayscale on a 256 scale. The names of the person in each photo is in `labels_all.txt`. 1. Used `np.random.mulivariate_normal` to samples from 2+ multivariate gaussians in 2D space (for easy visualization)2. Ensured that clustering algorithm is able to correctly cluster the gaussians<jupyter_code>import numpy as np
import scipy.sparse as sp
import matplotlib
matplotlib.use("svg")
# AUTOLAB_IGNORE_START
%matplotlib inline

# AUTOLAB_IGNORE_STOP
import matplotlib.pyplot as plt
plt.style.use("ggplot")
# AUTOLAB_IGNORE_START
data = np.loadtxt("../hw5data/faces_all.txt")
with open("labels_all.txt") as f:
    labels = [line.rstrip() for line in f]

# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>Image Visualization It will be extremely valuable to be able to visualize the images and cluster means. So first you should create the following function, which plots a matrix of images in a grid structure. Each row of your data matrix is a vector of pixel values ranging from 0 to 255, as a grayscale value.  Specification* You can assume that the number of pixels is a square number* Use the matplotlib subplots to form a grid of images, and use `imshow` to plot the image. * For grading purposes, you should return a list of `matplotlib.image.AxesImage` objects. This is simply the result of calling the matplotlib `imshow` function. * The order of the returned objects should match the order of the matplotlib subplots. * You can assume that there are at least rows\*cols entries in X. If there are more than that many, just plot the first rows\*cols entries.<jupyter_code>import math

def plot_images(X, rows, cols):
    """ Plots the centers in a (row by col) grid
        Args: 
            centers (numpy 2D matrix) : matrix of centers, each row is a center
            rows (int) : number of rows to plot in a grid
            cols (int) : number of cols to plot in a grid
        Returns: 
            (list) : list of matplotlib.image.AxesImage objects
    """
    
    M,N = rows,cols
    fig, ax = plt.subplots(M,N)
    imgL = []
    pix = np.sqrt(X.shape[1])
    for j in range(M):
        for k in range(N):  
            imgL.append(ax[j][k].imshow(np.reshape(X[j*M+k],(pix,pix)), cmap=plt.get_cmap('gray')))
    return imgL;
    pass
#     digits = np.vstack([np.hstack([np.reshape(X[i*N+j,:],(256,256)) 
#                                    for j in range(N)]) for i in range(M)])
#     ax.imshow(255-digits, cmap=plt.get_cmap('gray'))
    
# AUTOLAB_IGNORE_START
plot_images(data, 10, 10)
plt.show()
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>K-means clustering (KMeans++)The first task here is to implement k-means clustering, as seen on slide 13 of the unsupervised learning lecture. We will use this to attempt to cluster the images into something more meaningful.   Specification* Use np.argmin when assigning clusters to get a consistent result with the grading script. * You can refer to the above visualization to write your plotting function* You should initialize your centers by using the KMeans++ initialization described in the slides. This typically gives significantly better results.<jupyter_code># def kmeans(X, k, max_iter=10):
# Mu = X[np.random.choice(X.shape[0],k),:] for i in range(max_iter):
#         D = (-2*X.dot(Mu.T) + np.sum(X**2,axis=1)[:,None] +
#              np.sum(Mu**2,axis=1))
#         C = np.eye(k)[np.argmin(D,axis=1),:]
# Mu = C.T.dot(X)/np.sum(C,axis=0)[:,None]
# loss = np.linalg.norm(X - Mu[np.argmin(D,axis=1),:])**2 return Mu, C, loss


import math
class KMeans:
    def init_centers(self, X, k):
        """ Initialize the starting k centers, using the KMeans++ algorithm. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                k (float) : number of clusters
            Return: 
                (numpy 2D matrix) : matrix of centers, each row is a center
        """
        
     
        Mu = np.zeros((k,X.shape[1]))
        Mu[0] = X[np.random.choice(X.shape[0],1),:]
        
        for i in range(1,k):
        
            pr =  (-2*X.dot(Mu.T) + np.sum(X**2,axis=1)[:,None] + np.sum(Mu**2,axis=1))
            pr = pr[:,:i]
           
            
            prmin = np.amin(pr,axis = 1)
            
            sum=np.sum(prmin)
            
            prob = prmin/sum
    

            Mu[i] = X[np.random.choice(X.shape[0],1,p=prob ),:]
           
        return Mu
            

        pass
    
    def assign_clusters(self, X, centers):
        """ Given the data and the centers, assign clusters to all the examples. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                centers (numpy 2D matrix) : matrix of centers, each row is a center
            Return: 
                (numpy 2D matrix) : 1 hot encoding of cluster assignments for each example
        """
        
        D = (-2*X.dot(centers.T) + np.sum(X**2,axis=1)[:,None] + np.sum(centers**2,axis=1))
        C = np.eye(len(centers))[np.argmin(D,axis=1),:]
        return C
        pass
    
    def compute_means(self, X, y):
        """ Given the data and the cluster labels, compute the new cluster centers. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                y (numpy 2D matrix) : 1 hot encoding of cluster assignments for each example
            Return: 
                (numpy 2D matrix) : matrix of centers, each row is a center
        """
        
        Mu = y.T.dot(X)/np.sum(y,axis=0)[:,None]
        return Mu
        pass
    
    def train(self, X, centers, niters=20):
        """ Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                centers (numpy 2D matrix) : initial matrix of centers, each row is a center
            Return: 
                (y, centers) : tuple of 1 hot encoding of cluster assignments for each example 
                               the resulting cluster centers
        """
        for i in range(niters):
            KM = KMeans()
            C= KM.assign_clusters(X, centers)
            
            centers = KM.compute_means( X, C)
        return (C,centers)
        pass<jupyter_output><empty_output><jupyter_text>Run your code on the given face images. Use the `plot25` function to visualize<jupyter_code># AUTOLAB_IGNORE_START
k = 25
P = np.random.permutation(data.shape[0])
# KM = KMeans()
# centers = KM.init_centers(data,k)
# centers
# y,centers = KM.train(data,centers)
print y
# train and plot your centers
# AUTOLAB_IGNORE_STOP


# AUTOLAB_IGNORE_START
# k = 5
# X= data
# max_iter = 10
# Mu = X[np.random.choice(X.shape[0],k),:] 
# print Mu.shape
# print Mu
# for i in range(max_iter):
#     D = (-2*X.dot(Mu.T) + np.sum(X**2,axis=1)[:,None] +
#     np.sum(Mu**2,axis=1))
#     print D
#     C = np.eye(k)[np.argmin(D,axis=1),:]
#     Mu = C.T.dot(X)/np.sum(C,axis=0)[:,None]
    
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>Running K-means with 25 clusters on a random permutation gives the following result with our implementation:    Cross-Entropy lossOne way to to measure the quality of your clusters (if we have labels for the true cluster) is to measure the purity of a cluster: if a cluster contains all of the same label, then it is a good cluster. One such measure is the cross entropy loss, which you've already seen when learning about decision trees (and in the MNIST homework if you were in 688). If $p_i$ is the proportion of cluster elements belonging to the true cluster $i$, then:$$\text{CrossEntropyLoss(cluster)}=-\sum_{i\in \text{cluster}} p_i \log p_i$$In other words, we take the proportions of all true clusters present in the estimated cluster, and calculate the cross entropy loss to see how good of a cluster it actually was. If everything in the cluster has the same label, then $p_i =1$ and so this sum is just 0, so 0 is a perfect score. The worst case is if the cluster is evenly distributed amongst $T$ clusters, and so $p_i = 1/T$ for $T$ clusters, resulting in a loss of $-\log 1/T$.  Specification* Compute the cross entropy loss according to the above equation. * The sum should only be over true labels that are actually present in the cluster. * The total entropy is just the sum of the cross entropy loss of all the estimated clusters.<jupyter_code>from collections import Counter

def entropy(cluster_labels):
    """ Calculate the entropy of a given cluster
        Args: 
            cluster_labels (list) : list of true cluster labels assigned to a cluster
        Return: 
            (float) : the cross entropy loss of this cluster
    """
    dict = Counter(cluster_labels) 
    k = 0 
    for i in set(cluster_labels):
        p = float(dict[i])/float(len(cluster_labels))
        k += p * np.log(p)
    return -k
    
    pass
    
    
def total_entropy(y, labels):
    """ Compute the total cross entropy loss. 
        Args: 
            y (numpy 2D array) : one hot encoding of the estimated cluster labels
            labels (list) : list of the true labels of each data point
    """
    sum = 0 
    clustno = 0
    clustdict = {}
    for i in y.T:
        
        ind = np.where(i ==1)
        k = ind[0].tolist()
        clustdict[clustno] = [labels[i] for i in k]
        clustno+=1
       
    for key, val in clustdict.iteritems():

        sum += entropy(val)
    return sum    
    pass

# AUTOLAB_IGNORE_START
print 
total_entropy(y, labels)
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>The following code splits the dataset into 25 roughly even clusters. You should expect that the total entropy of the output of your K-means algorithm to do better than the total entropy of a random clustering.<jupyter_code># AUTOLAB_IGNORE_START
y0 = np.zeros(len(P))
size = int(math.ceil(float(len(P))/25))

for i in range(k):
    end = min((i+1)*size, len(P))
    y0[P[i*size: end]] = i
print len(np.eye(len(P))[y0.astype(int),:][0])
total_entropy(np.eye(len(P))[y0.astype(int),:], labels)
# AUTOLAB_IGNORE_STOP<jupyter_output>4324<jupyter_text>Gaussian Mixture ModelIn this part, you will implement the softer cousin of K-means clustering: the Gaussian mixture model. For this problem, you should refer to the Anomaly detection slides for the details of the algorithm. We have provided an implementation of the multivariate normal log PDF function that works on an array of samples. Note that scipy has one as well, but it can only calculate the log pdf for one example at a time and is not as efficient.  Specification* Again, you can initialize your centers using the first k random indices from the permutation. * You should add a diagonal regularization to your covariance matrix to avoid singularity issues. * Your covariance matrix should be intialized as the sample covariance of the entire data matrix plus regularization using the unbiased estimater. You can use `np.cov` for this. * You can initialize  `phi` as a uniform discrete distribution over the clusters.<jupyter_code>def mv_normal_log_pdf(X, mu, Sig):
    return (-0.5*np.linalg.slogdet(2*np.pi*Sig)[1] - 0.5*np.sum((X-mu)*(np.linalg.inv(Sig).dot((X-mu).T)).T,axis=1))





class GaussianMixture:
    def init_parameters(self, X, P, k, reg):
        """ Initialize the parameters of means, covariances, and frequency counts. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                P (numpy 2D matrix) : Random permutation vector
                k (float) : number of clusters
                reg (float) : regularization parameter for the covariance matrix
            Returns: 
                mus (numpy 2D matrix) : matrix of initialzied means, chosen randomly by selection the first k elements of P
                Sigmas (list) : list of 2D covariance matrices corresponding to each cluster
                phi (numpy 1D vector) : vector of initialized frequencies
        """
        
        Sigmas = []
        ind = P[:k]
        mus = X[ind,:]
        
        Xinter = X + reg
        
        for i in range(k):
            Sigmas.append(np.cov(Xinter.T))

        phi = np.ones(k)/float(k)

        return mus,Sigmas,phi
        pass

    def Estep(self, X, mus, Sigmas, phi):
        """ Perform an E step and return the resulting probabilities. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                mus (numpy 2D matrix) : matrix of initialzied means, chosen randomly by selection the first k elements of P  
                Sigmas (list) : list of 2D covariance matrices corresponding to each cluster
                phi (numpy 1D vector) : vector of initialized frequencies  .... k
            Returns: 
                (numpy 2D matrix) : matrix of probabilities, where the i,jth entry corresponds to the probability of the
                                    ith element being in the jth cluster. ....4324,k
                                    
        """
        num = np.zeros((X.shape[0],mus.shape[0]))
        prob= np.zeros((X.shape[0],mus.shape[0]))
        
        i = 0
        for ind,ele in enumerate(Sigmas):
            num[:,i] = mv_normal_log_pdf(X, mus[i], Sigmas[i]) + np.log(phi[i])
            i = i+1

        for i in range(len(num)):
            numerator=np.exp(num[i,:]-np.max(num[i,:]))
            
            den = np.sum((np.exp(num[i,:]-np.max(num[i,:])))) 
            prob[i,:] = numerator/den
            
        return prob
        pass

    def Mstep(self, ps, X, reg):
        """ Initialize the parameters of means, covariances, and frequency counts. 
            Args: 
                ps (numpy 2D matrix) : matrix of probabilities, where the i,jth entry corresponds to the probability of the
                                       ith element being in the jth cluster. 
                X (numpy 2D matrix) : data matrix, each row is an example
                reg (float) : regularization parameter for the covariance matrix
            Returns: 
                (mus, Sigmas, phi) : 3 tuple of matrix of initialzied means, chosen randomly by selection the first 
                                     k elements of P, a list of 2D covariance matrices corresponding to each cluster, 
                                     and a vector of initialized frequencies
        """
        Sig = []
        phi = np.sum(ps,axis = 0)/X.shape[0]
#        print(phi)
#        print np.sum(phi)
#        print HI
        mus = np.zeros((ps.shape[1],X.shape[1]))
        for i in range(ps.shape[1]):
            mu =  (ps[:,i][:,None] * X)/np.sum(ps[:,i])
            mus[i,:] = np.sum(mu, axis = 0)
            #a1=ps[:,i][:,None] * X
            #a2=np.sum(a1,axis=0)
            #a3=np.sum(ps[:,i])
            #a4=a2/a3
            #mus[i,:] = a4
            

        dia = np.eye((X.shape[1]))
        diag = dia * reg

        for i in range(ps.shape[1]):
            
            num = (X-mus[i])*(ps.T[i][:,None])
            num2 = num.T.dot(X-mus[i])
            den = np.sum((ps[:,i]))
            num2 = num2/den
            num2 = num2+diag
#             p = np.zeros((X.shape[1],X.shape[1]))
            
# #             p2= np.zeros((X.shape[1],X.shape[1]))
# #            p=np.dot(X-mu.T,(X-mu))
# #            p=p+diag
# #            p=ps[:,i]*p
            
# #            X[:,i]-mus[i,:]
#             for j in range(ps.shape[0]):
# #                 if j%1000==0:
# #                     print j
#                 p += (np.outer((X[j,i]-mus[i,:]),(X[j,i]-mus[i,:]).T))
#                       *ps[j,i])/np.sum(ps[:,i])
#                 temp=((X[j,i]-mus[i,:])*(X[j,i]-mus[i,:]).T)
            
#             p=p+diag
            
#            p=p/np.sum(ps[:,i])
#            print(p2)
#            print(p)
#            print HI
#            p2 = p2 + diag
#            print(p.shape)
#            Sig.append(p)
            Sig.append(num2)

        return (mus,Sig, phi)
    
        pass
    
    def train(self, X, mus, Sigmas, phi, niters = 5, reg=1e-4):
        """ Train the model using the EM algorithm for a number of iterations. 
            Args: 
                X (numpy 2D matrix) : data matrix, each row is an example
                mus (numpy 2D matrix) : matrix of initialzied means, chosen randomly by selection the first k elements of P
                Sigmas (list) : list of 2D covariance matrices corresponding to each cluster
                phi (numpy 1D vector) : vector of initialized frequencies
                niters (int) : number of EM iterations to run
            Returns: 
                (mus, Sigmas, phi) : 3 tuple of matrix of initialzied means, chosen randomly by selection the first 
                                     k elements of P, a list of 2D covariance matrices corresponding to each cluster, 
                                     and a vector of initialized frequencies 
        
        """
        GM = GaussianMixture()

        for i in range(niters):
        
            # mus, Sigmas, phi = GM.init_parameters(data,P,k,reg)
  
            prob = GM.Estep( X, mus, Sigmas, phi)

            mus,Sigmas,phi = GM.Mstep(prob,X, reg)
        
        return (mus, Sigmas, phi)
        pass

# AUTOLAB_IGNORE_START
k = 5
reg = 1e-4
P = np.random.permutation(data.shape[0])
GM = GaussianMixture()

mus, Sigmas, phi = GM.init_parameters(data,P,k,reg)

prob = GM.Estep( data, mus, Sigmas, phi)
# # print prob.shape
mus,Sigmas,phi = GM.Mstep(prob,data, reg)


# train and plot your centers

#         Sigmas = np.dot((X-mus),(X-mus).T)
#         print len(Sigmas)
#         print mus.shape         /np.sum(ps[:,i])
#         print phi.shape
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>The Gaussian mixture model takes a lot longer to train than the K-means model. You may also get better results initializing your clusters with the results from the K-means model. Running the Gaussian mixture model with an intialization from the K-means clusters, regularization $10^{-4}$, and 10 iterations gives the following Gaussian means (for k=5) on our implementation:   Face Classification ContestIn general, facial classification in images is a difficult problem. Here, we will focus on a much simplified version of the problem: given images of faces with noise, can we classify a face as either <NAME> or <NAME>? We will designate <NAME> as +1, and <NAME> as -1.<jupyter_code># Load the data
# AUTOLAB_IGNORE_START
data = np.loadtxt("../hw5data/faces_noisy.txt")
with open("labels_all.txt") as f:
    labels = [line.rstrip() for line in f]

# plot_images(data,10,10)
# plt.show()
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>For your convenience, we've written some of the pipelining code for you which extracts only the images corresponding to George Bush and <NAME>. It also randomly splits the resulting data into a train and a test set.<jupyter_code># Extract the bush and powell images and create a random permutation
# AUTOLAB_IGNORE_START
bush_idxs = [i for i,l in enumerate(labels) if l=="George_W_Bush"]
powell_idxs = [i for i,l in enumerate(labels) if l=="Colin_Powell"]
all_data = data[bush_idxs + powell_idxs]
all_labels = np.hstack([np.ones(len(bush_idxs)), -np.ones(len(powell_idxs))])
P = np.random.permutation(len(all_labels))
# AUTOLAB_IGNORE_STOP
# Split into a training and a test set
# AUTOLAB_IGNORE_START
split = 400
tr_idx, te_idx = P[:split], P[split:]
data_tr, data_te = all_data[tr_idx], all_data[te_idx]
labels_tr, labels_te = all_labels[tr_idx], all_labels[te_idx]
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>SVM refresherAn easy first attempt at this problem is to just throw the data into an SVM. You can copy your SVM from the previous homework here, or use the one in the reference solution (which implements everything in 1-2 lines). What happens when you try to just straight up use an SVM?<jupyter_code>class SVM:
    def __init__(self, X, y, reg):
        """ Initialize the SVM attributes and initialize the weights vector to the zero vector. 
            Attributes: 
                X (array_like) : training data intputs
                y (vector) : 1D numpy array of training data outputs
                reg (float) : regularizer parameter
                theta : 1D numpy array of weights
        """
        self.X = X
        self.y = y
        self.reg = reg
        self.theta = np.zeros(X.shape[1])
        self.Xy = sp.diags(y).dot(sp.coo_matrix(X))
        
    def objective(self, X, y):
        """ Calculate the objective value of the SVM. When given the training data (self.X, self.y), this is the 
            actual objective being optimized. 
            Args:
                X (array_like) : array of examples, where each row is an example
                y (array_like) : array of outputs for the training examples
            Output:µ
                (float) : objective value of the SVM when calculated on X,y
        """
        obj = sum(np.maximum((1-np.multiply(y,np.dot(self.theta,X.T))),0)) + (self.reg/2) * sum (np.multiply(self.theta,self.theta)) 
        return obj
        pass
    
    def gradient(self):
        """ Calculate the gradient of the objective value on the training examples. 
            Output:
                (vector) : 1D numpy array containing the gradient
        """
        
        z = self.reg*self.theta
        t = (-self.Xy.T.dot(self.Xy.dot(self.theta) <= 1)) + self.reg*self.theta
    
        return t
        pass
    
    def train(self, niters=100, learning_rate=1, verbose=False):
        """ Train the support vector machine with the given parameters. 
            Args: 
                niters (int) : the number of iterations of gradient descent to run
                learning_rate (float) : the learning rate (or step size) to use when training
                verbose (bool) : an optional parameter that you can use to print useful information (like objective value)
        """
    
    
        for i in range(niters):
             self.theta -= (learning_rate * self.gradient())
        pass
            
    
    def rbf(X,mu,sig):
        sqdist = (-2*X.dot(mu.T) +
        np.sum(X**2,axis=1)[:,None] +
        np.sum(mu**2,axis=1))
        return np.exp(-sqdist/(2*sig**2))   
        pass          
                  
    def predict(self, X):
        """ Predict the class of each label in X. 
            Args: 
                X (array_like) : array of examples, where each row is an example
            Output:
                (vector) : 1D numpy array containing predicted labels
        """
        pred = X.dot(self.theta)
        pred[pred<0] = -1
        pred [pred>=0] = 1
        return (pred)
        pass
    
# AUTOLAB_IGNORE_START
svm = SVM(data_tr, labels_tr, 1e-3)
svm.train(verbose=True, niters=1000)
(svm.predict(data_te)==labels_te).mean()
# AUTOLAB_IGNORE_STOP<jupyter_output><empty_output><jupyter_text>Spoiler alert: the SVM gets stuck and ends up just predicting 1 for everything, so every image is <NAME>! This is because noisy images are even more nonlinear than normal images, and so the SVM has no hope at separating the two classes with a hyperplane. Without the noise (you can try this yourself), the the dataset is so nicely processed that Bush and Powell are actually linearly separable, but in real world images this is rarely ever the case.   Contest detailsFor this homework, your goal is to achieve at least 80% accuracy on the classification problem between <NAME> and <NAME>. You will need to write an end-to-end classifier that takes in training inputs, training outputs, and testing inputs. It should return predicted outputs for the testing outputs. The contest leaderboard will be ranked according to your accuracy.  Baseline approachIn lecture, Zico mentioned that a common technique when using RBF features is to use KMeans to select RBF centers, which help make nonlinear decision boundaries.  With a proper choice of parameters, this can do quite well on this task. Using an SVM with RBF features as described in lecture (with the median trick) and KMeans++, you can achieve full credit on this problem (our implementation achieves 81-82% accuracy).  However, similar to the last contest, you are free to do whatever you'd like and do not have to adhere to this solution. We have tried multiple other methodologies, which are also able to break the 80% barrier, and some of which are quite simple. With proper domain knowledge, you could even do this with just feature engineering and vanilla SVM! Specification: * Your function will receive the a labeled training set, and an unlabeled testing set, in the same format as that provided in this notebook. * There are no restrictions other than 1) you cannot use sklearn (we found it to be largely unhelpful, so you aren't missing much) and 2) you are restricted to the computing capabilities on Autolab. * You will receive full credit if you get above an 80% accuracy score. You will receive a scaled score for every % point you get above predicting the most frequent training example (e.g. predicting 1 for everything). * We've taken care of almost all the plumbing code for you in this assignment, so you just need to worry about predict_faces.<jupyter_code>def predict_faces(pixels, labels, pixels_te):
    """ Given some pixels and labels as training data, predict the labels of the testing set. 
        Args: 
            pixels (Numpy 2D array) : Array of pixels for each training example
            labels (Numpy 1D vector) : Vector of class labels for each training example
            pixels_te (Numpy 2D array) : Array of pixels for each testing example
        Return:
            (Numpy 1D vector) : Vector of predicted class labels for each testing example
    """
#     KM = KMeans()
#     centers = KM.init_centers(pixels,k=2)
#     pred = np.zeros(pixels_te.shape[0])
    
    
#     t,centers = KM.train(pixels,centers)
    
#     y,centers = KM.train(pixels_te,centers)
    
    
#     for idx, ele in enumerate(y):
#         if (ele[0] == 1):
#             pred[idx] = -1
#         else:
#             pred[idx] = 1
#     return pred
#     pred = [for i in ]
#     print y, labels_tr

    
    svm = SVM(pixels, labels, 1e-4)
    svm.train(verbose=True, niters=1000)
    pred = svm.predict(pixels_te)
    
    return pred 
    pass

# AUTOLAB_IGNORE_START
p_te = predict_faces(data_tr, labels_tr, data_te)
# print (labels_te==p_te).mean()

# AUTOLAB_IGNORE_STOP
import pickle
with open('../autograde/contest.pkl') as f:
    out = pickle.load(f)
print out
plot_images(out["Xte"], 5, 5)
out["Xtr"].shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09699096542970208, 'fraction_numerical': 0.01060255357276189, 'mean_word_length': 2.4320307495195386, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '283991', 'license': 'Apache-2.0', 'n_tokens_mistral': 8393, 'n_tokens_neox': 7897, 'n_words': 2946, 'path': 'unsupervised.ipynb', 'repo_name': 'bswalia/UnsupervisedLearning', 'size': '26842'}