<jupyter_start><jupyter_text>Práctica 6: Support Vector Machines--- Autores: Grupo 7<NAME> - 5º Doble Grado en Ingeniería Informática - Matemáticas   <NAME> - 4º Grado en Ingeniería Informática---**Fecha de entrega:** 13 de diciembre de 2018, 18.00h<jupyter_code>%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from scipy.io import loadmat
from scipy.optimize import minimize
from sklearn.svm import SVC
from process_email import email2TokenList
import codecs
from get_vocab_dict import getVocabDict
import glob
import sklearn.model_selection as ms
from sklearn.metrics import confusion_matrix
import time<jupyter_output><empty_output><jupyter_text>1. Support Vector Machines<jupyter_code>def plot_decision_line(X, y, svm):
    svm.fit(X, y.ravel())
    x1 = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
    x2 = np.linspace(X[:, 1].min(), X[:, 1].max(), 100)
    x1, x2 = np.meshgrid(x1, x2)
    yp = svm.predict(np.array([x1.ravel(),x2.ravel()]).T).reshape(x1.shape)
    pos = (y == 1).ravel()
    neg = (y == 0).ravel()
    plt.figure()
    plt.scatter(X[pos, 0], X[pos, 1], color='black', marker='+')
    plt.scatter(X[neg, 0], X[neg, 1], color='yellow',
    edgecolors='black', marker='o')
    plt.contour(x1, x2, yp)<jupyter_output><empty_output><jupyter_text>1.1 Kernel lineal<jupyter_code>data = loadmat('ex6data1.mat')
X, y = data['X'], data['y'].ravel()
svm = SVC(kernel='linear', C=1)
plot_decision_line(X, y, svm)
svm = SVC(kernel='linear', C=100)
plot_decision_line(X, y, svm)<jupyter_output><empty_output><jupyter_text>1.2 Kernel gaussiano<jupyter_code>data = loadmat('ex6data2.mat')
X, y = data['X'], data['y'].ravel()
C = 1.0
sigma = 0.1

svm = SVC(kernel='rbf', C=C, gamma=1 / (2 * sigma**2))
svm.fit(X, y.ravel())
plot_decision_line(X, y, svm)<jupyter_output><empty_output><jupyter_text>1.3 Elección de los parámetros $C$ y $\sigma$<jupyter_code>data = loadmat('ex6data3.mat')
X, y, Xval, yval = data['X'], data['y'].ravel(), data['Xval'], data['yval'].ravel()
n = 8
accuracy = np.empty((n,n))

C = 0.01
for c in range(n):
    sigma = 0.01
    for s in range(n):
        svm = SVC(kernel='rbf', C=C, gamma=1 / (2 * sigma**2))
        svm.fit(X, y.ravel())
        accuracy[c,s] = svm.score(Xval,yval)
        sigma *= 3
    C *= 3

print('Error mínimo: {}'.format(1-accuracy.max()))
C_opt, sigma_opt = accuracy.argmax()//n, accuracy.argmax()%n
print('C óptimo: {}   sigma óptimo: {}'.format(C_opt, sigma_opt))

svm = SVC(kernel='rbf', C=0.01*3**C_opt, gamma=1 / (2 * (0.01*3**sigma_opt)**2))
plot_decision_line(X, y, svm)<jupyter_output>Error mínimo: 0.030000000000000027
C óptimo: 4   sigma óptimo: 2<jupyter_text>2. Detección de spam En primer lugar, procesar los correos electrónicos para generar los datos de entrenamientoy validación.   El siguiente paso es convertir el texto del mensaje en un vector de atributos.<jupyter_code>corpus = getVocabDict()
# Process all spam mails
files = glob.glob('spam/*.txt')
Xspam = np.zeros((len(files), len(corpus)))
yspam = np.ones(len(files))
i = 0
for f in files:
    email_contents = codecs.open(f, 'r', encoding='utf-8', errors='ignore').read()
    tokens = email2TokenList(email_contents)
    words = filter(None,[corpus.get(x) for x in tokens])
    for w in words:
        Xspam[i, w-1] = 1
    i +=1
# Process all easy_ham mails
files = glob.glob('easy_ham/*.txt')
Xeasy_ham = np.zeros((len(files), len(corpus)))
yeasy_ham = np.zeros(len(files))
i = 0
for f in files:
    email_contents = codecs.open(f, 'r', encoding='utf-8', errors='ignore').read()
    tokens = email2TokenList(email_contents)
    words = filter(None,[corpus.get(x) for x in tokens])
    for w in words:
        Xeasy_ham[i, w-1] = 1
    i +=1
# Process all hard_ham mails
files = glob.glob('hard_ham/*.txt')
Xhard_ham = np.zeros((len(files), len(corpus)))
yhard_ham = np.zeros(len(files))
i = 0
for f in files:
    email_contents = codecs.open(f, 'r', encoding='utf-8', errors='ignore').read()
    tokens = email2TokenList(email_contents)
    words = filter(None,[corpus.get(x) for x in tokens])
    for w in words:
        Xhard_ham[i, w-1] = 1
    i +=1
    
Xspam.shape, Xeasy_ham.shape, Xhard_ham.shape<jupyter_output><empty_output><jupyter_text>Una vez hemos obtenido el dataset, debemos generar los datos de entrenamientoy validación, esto es, se deben evaluar distintas configuraciones del sistema de aprendizaje.   Típicamente se hace una división de los datos para train, test y validation del 60%, 20% y 20%, respectivamente.<jupyter_code>X = np.vstack((Xspam, Xeasy_ham, Xhard_ham))

yspam = [1]*len(Xspam)
yeasy_ham = [0]*len(Xeasy_ham)
yhard_ham = [0]*len(Xhard_ham)
y = np.r_[yspam, yeasy_ham, yhard_ham]

X.shape, y.shape<jupyter_output><empty_output><jupyter_text>Feature selectionThe number of different words in a document can be huge. The document representation vectors, althought sparse, may still have hundreds of nonzero components.   Most of those words are irrelevant to the classification task and can be dropped with no harm to the classifier performance.   The simplest measure is the document frequency, i.e, the number of documents containi each word. Experimenta evidence suggests that using only the top 10 percent of the most frequent words does not reduce the performance of classifiers.   Source:<NAME>., & <NAME>. (2006). The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge: Cambridge University Press. [doi:10.1017/CBO9780511546914](https://doi.org/10.1017/CBO9780511546914) Para comprobar el resultado anterior, haremos un estudio de precisión VS. eficiencia, comparando los resultados obtenidos con y sin reducción de atributos.  Con reducción de atributos<jupyter_code>def most_freq_words(X, frac):
    nwords = int(np.round(len(X[0])*frac)) # frac*100% of words
    freq = np.sort(X.sum(axis=0))[-nwords] # smaller frequece of the top %
    features = np.where(X.sum(axis=0) >= freq)[0] # top % of the most frequent words
    return X[:, features]<jupyter_output><empty_output><jupyter_text>Split data into train, validation and test sets.<jupyter_code># Feature selection/dimensionality reduction
X_freq = most_freq_words(X, 0.1)
# Split data into train, validation and test sets - 60% 20% 20%
X_train, X_test, y_train, y_test = ms.train_test_split(X_freq, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = ms.train_test_split(X_train, y_train, test_size=0.25, random_state=1)<jupyter_output><empty_output><jupyter_text>Fit the model. Hyperparameter tuning.<jupyter_code>n = 10
accuracy = np.empty((n,n))

tic = time.process_time()
C = 0.01
for c in range(n):
    sigma = 0.01
    for s in range(n):
        svm = SVC(kernel='rbf', C=C, gamma=1 / (2 * sigma**2))
        svm.fit(X_train, y_train)
        accuracy[c,s] = svm.score(X_val,y_val)
        sigma *= 3
    C *= 3

C_opt, sigma_opt = accuracy.argmax()//n, accuracy.argmax()%n
svm = SVC(kernel='rbf', C=0.01*3**C_opt, gamma=1 / (2 * (0.01*3**sigma_opt)**2))
svm.fit(X_train, y_train)
y_pred = svm.predict(X_test)
toc = time.process_time()

score_reduct = svm.score(X_test, y_test)
time_reduct = toc-tic

# Display results
print('Error mínimo: {}'.format(1-accuracy.max()))
print('C óptimo: {}   sigma óptimo: {}'.format(0.01*3**C_opt, 0.01*3**sigma_opt))
print('Matriz de confusión: ')
print(confusion_matrix(y_test, y_pred))
print('Precisión obtenida: {:.3f}%'.format(score_reduct*100))
print('Tiempo de ejecución: {}'.format(time_reduct))<jupyter_output>Error mínimo: 0.021212121212121238
C óptimo: 7.29   sigma óptimo: 7.29
Matriz de confusión: 
[[547   8]
 [  8  98]]
Precisión obtenida: 0.9758%
Tiempo de ejecución: 79.546875<jupyter_text>Sin reducción de atributos Split data into train, validation and test sets.<jupyter_code># Split data into train, validation and test sets - 60% 20% 20%
# Use the same random state to get the same sets as before
X_train, X_test, y_train, y_test = ms.train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = ms.train_test_split(X_train, y_train, test_size=0.25, random_state=1)<jupyter_output><empty_output><jupyter_text>Fit the model. Hyperparameter tuning.<jupyter_code>n = 10
accuracy = np.empty((n,n))

tic = time.process_time()
C = 0.01
for c in range(n):
    sigma = 0.01
    for s in range(n):
        svm = SVC(kernel='rbf', C=C, gamma=1 / (2 * sigma**2))
        svm.fit(X_train, y_train)
        accuracy[c,s] = svm.score(X_val,y_val)
        sigma *= 3
    C *= 3

C_opt, sigma_opt = accuracy.argmax()//n, accuracy.argmax()%n
svm = SVC(kernel='rbf', C=0.01*3**C_opt, gamma=1 / (2 * (0.01*3**sigma_opt)**2))
svm.fit(X_train, y_train)
y_pred = svm.predict(X_test)
toc = time.process_time()

score_full = svm.score(X_test, y_test)
time_full = toc-tic

# Display results
print('Error mínimo: {}'.format(1-accuracy.max()))
print('C óptimo: {}   sigma óptimo: {}'.format(0.01*3**C_opt, 0.01*3**sigma_opt))
print('Matriz de confusión: ')
print(confusion_matrix(y_test, y_pred))
print('Precisión obtenida: {:.3f}%'.format(score_full*100))
print('Tiempo de ejecución: {}'.format(time_full))<jupyter_output>Error mínimo: 0.015151515151515138
C óptimo: 21.87   sigma óptimo: 21.87
Matriz de confusión: 
[[551   4]
 [  9  97]]
Precisión obtenida: 0.9803%
Tiempo de ejecución: 749.03125
{'fraction_non_alphanumeric': 0.147277628032345, 'fraction_numerical': 0.04140161725067385, 'mean_word_length': 4.83029541169076, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '373295', 'license': 'MIT', 'n_tokens_mistral': 3868, 'n_tokens_neox': 3588, 'n_words': 891, 'path': 'P6.ipynb', 'repo_name': 'RaulMurillo/Aprendizaje-Automatico', 'size': '9310'}