<jupyter_start><jupyter_text>1. Sacamos la figura con la Toolbox  1.1. Importación de datos a RDD<jupyter_code>import urllib2
from pyspark.mllib.regression import LabeledPoint

response = urllib2.urlopen("https://archive.ics.uci.edu/ml/machine-learning-databases/statlog/satimage/sat.trn")
textdata = response.read()
RDD = sc.parallelize(textdata.split('\n')[:-1]).map(lambda line: line.split(" "))

RDD_labeled=RDD.map(lambda x: LabeledPoint(x[-1],x[0:-1]))

RDD_PCA = RDD_labeled.map(lambda x: x.features)<jupyter_output><empty_output><jupyter_text>1.2. Extraemos 15 componentes<jupyter_code>nprojections = 15

PCA_model = MVA('PCA','none','None',1e-5,nprojections,1e-8,1e-5,100,30)
PCA_model.fit(RDD_PCA)<jupyter_output>(15, 36)
(36, 36)
(36, 36)
Extracting projection vector 0 out of 15
Extracting projection vector 1 out of 15
Extracting projection vector 2 out of 15
Extracting projection vector 3 out of 15
Extracting projection vector 4 out of 15
Extracting projection vector 5 out of 15
Extracting projection vector 6 out of 15
Extracting projection vector 7 out of 15
Extracting projection vector 8 out of 15
Extracting projection vector 9 out of 15
Extracting projection vector 10 out of 15
Extracting projection vector 11 out of 15
Extracting projection vector 12 out of 15
Extracting projection vector 13 out of 15
Extracting projection vector 14 out of 15
(15, 36)
(36, 36)
(36, 36)
Extracting projection vector 0 out of 15
Extracting projection vector 1 out of 15
Extracting projection vector 2 out of 15
Extracting projection vector 3 out of 15
Extracting projection vector 4 out of 15
Extracting projection vector 5 out of 15
Extracting projection vector 6 out of 15
Extracting projection vector 7 out of 1[...]<jupyter_text>1.3. Dibujamos los vectores de proyección<jupyter_code>fig = plt.figure()

ax1 = fig.add_subplot(331)
ax1.plot(PCA_model._U[0,], 'b-')

ax1 = fig.add_subplot(332)
ax1.plot(PCA_model._U[1,], 'b-')

ax1 = fig.add_subplot(333)
ax1.plot(PCA_model._U[2,], 'b-')

ax1 = fig.add_subplot(334)
ax1.plot(PCA_model._U[3,], 'b-')

ax1 = fig.add_subplot(335)
ax1.plot(PCA_model._U[4,], 'b-')

ax1 = fig.add_subplot(336)
ax1.plot(PCA_model._U[5,], 'b-')

ax1 = fig.add_subplot(337)
ax1.plot(PCA_model._U[6,], 'b-')

ax1 = fig.add_subplot(338)
ax1.plot(PCA_model._U[7,], 'b-')

ax1 = fig.add_subplot(339)
ax1.plot(PCA_model._U[8,], 'b-')

plt.show()<jupyter_output><empty_output><jupyter_text>1.4. Evolución de la función objetivo al incrementar el número de componentes<jupyter_code>X_projected = np.array(PCA_model.predict(RDD_PCA).collect())
print X_projected.shape

X_pca = RDD_PCA.collect()
media = np.mean(X_pca, axis=0)
X_pca = X_pca-media

print X_pca.shape
from sklearn.linear_model import LinearRegression

mses = list()

n_targets = 36

for np2 in range(nprojections):
    mse = 0
    X_components = X_projected[:,:np2+1]
    for k in range(n_targets):
        targets = X_pca[:,k]
        
        #Entrenamos modelo
        lm = LinearRegression(fit_intercept=False)
        lm.fit(X_components, targets)
        predictions = lm.predict(X_components)
        
        MSE_iter = np.mean((targets-predictions)**2)
        mse += MSE_iter
    print 'MSE para ' + str(np2+1) + ' proyecciones: ' + str(mse)
    mses.append(mse)
plt.plot(map(lambda x: x+1, range(nprojections)), mses)
plt.xlabel('Number of projections')
plt.ylabel('Residual approximation error')
plt.title('Residual error for Sklearn projections')
plt.show()<jupyter_output><empty_output><jupyter_text>2. Para el cálculo distribuido de los MSE<jupyter_code>from pyspark.mllib.regression import LabeledPoint, LinearRegressionWithSGD, LinearRegressionModel

RDD_projected = prueba.predict(RDD_PCA)
mses = list()

n_targets = 36

for np in range(max_feat):
    nprojections = np+1
    mse = 0
    RDD_nprojections = RDD_projected.map(lambda x: x[:nprojections])
    for k in range(n_targets):
        RDD_target = RDD_PCA.map(lambda x: x[k])
        media = RDD_target.mean()
        RDD_target = RDD_target.map(lambda x: x - media)
        
        parsedData = RDD_target.zip(RDD_nprojections)
        
        parsedData = parsedData.map(lambda x: LabeledPoint(x[0], x[1:]))
        #print parsedData.first()
        #Entrenamos el modelo
        model = LinearRegressionWithSGD.train(parsedData, iterations=100, step=1e-5)
        # Evaluate the model on training data
        valuesAndPreds = parsedData.map(lambda p: (p.label, model.predict(p.features.T)))
        MSE_iter = valuesAndPreds.map(lambda (v, p): (v - p)**2).mean()
        mse += MSE_iter
    print 'MSE para ' + str(nprojections) + ' proyecciones: ' + str(mse)
    mses.append(mse)
num_feat=[1,2,3,4,5,6,7,8,9,10]
mses=[]
for i in range(len(num_feat)) :
    prueba = MVA('PCA','l1','None',1e-5,i+1, 0.01,1e-5,100,30)
    print num_feat[i+1]
    prueba.fit(RDD_PCA)
    U=prueba._U
    W=prueba._W
    mse=RDD_PCA.map(lambda x: np.mean(np.array(x - np.dot(W,np.dot(x, U.T)))**2)).mean()
    num_feat.append(mse)
    print mse 
    print 'First iteration for feature' + str(i+1) 
print num_feat
reg_param=[1e-8,1e-6,1e-5,1e-4, 1e-2, 1e-1,1]
Us_toolbox=[]
for i in range(len(reg_param)) :
    prueba = MVA('PCA','l1','None',1e-5,10, reg_param[i],1e-5,100,30)
    print reg_param[i]
    prueba.fit(RDD_PCA)
    Us_toolbox.append(prueba._U)
    print 'First iteration for ' + str(reg_param[i]) 

reg_param=[1e-8,1e-6,1e-5,1e-4, 1e-2, 1e-1,1]
Us_toolbox=[]
for i in range(len(reg_param)) :
    prueba = MVA('OPLS','l1','norm',1e-5,10,reg_param[i],1e-3,100)
    print reg_param[i]
    prueba.fit(RDD_sat)
    Us_toolbox.append(prueba._U)
    print 'First iteration for ' + str(reg_param[i]) 
print Us_toolbox[0]
from sklearn import decomposition
pca = decomposition.PCA(n_components=10)
pca.fit(RDD_PCA.collect())
from numpy import linalg as lin
import math
array_error=[]
for i in range(len(reg_param)-1) :
    result=math.sqrt(lin.norm(pca.components_ - Us_toolbox[i],'fro'))
    array_error.append(result)

print array_error
import matplotlib.pyplot as plt
plt.plot([1e-8,1e-6,1e-5,1e-4, 1e-2, 1e-1], [Us_toolbox[0],Us_toolbox[1],Us_toolbox[0],Us_toolbox[0],Us_toolbox[0],Us_toolbox[0]], 'ro')
prueba = MVA('OPLS','l1','norm',1e-5,10,1e-8,1e-3,100)
prueba.fit(RDD_labeled)
reg_param=[1e-8,1e-6,1e-5,1e-4, 1e-2, 1e-1,1]
Us_toolbox=[]
for i in range(len(reg_param)) :
    prueba = MVA('OPLS','l1','norm',1e-5,10,reg_param[i],1e-3,100)
    print reg_param[i]
    prueba.fit(RDD_labeled)
    Us_toolbox.append(prueba._U)
    print 'First iteration for ' + str(reg_param[i])<jupyter_output>1e-08
(5, 36)
(6, 36)
(6, 6)
Extracting projection vector 0 out of 5
Extracting projection vector 1 out of 5
Extracting projection vector 2 out of 5
Extracting projection vector 3 out of 5
Extracting projection vector 4 out of 5
(5, 36)
(6, 36)
(6, 6)
Extracting projection vector 0 out of 5
Extracting projection vector 1 out of 5
Extracting projection vector 2 out of 5
Extracting projection vector 3 out of 5
Extracting projection vector 4 out of 5
Frobenius norm error: 0.00417402062326
(5, 36)
(6, 36)
(6, 6)
Extracting projection vector 0 out of 5
Extracting projection vector 1 out of 5
Extracting projection vector 2 out of 5
Extracting projection vector 3 out of 5
Extracting projection vector 4 out of 5
Frobenius norm error: 0.000250911143384
(5, 36)
(6, 36)
(6, 6)
Extracting projection vector 0 out of 5
Extracting projection vector 1 out of 5
Extracting projection vector 2 out of 5
Extracting projection vector 3 out of 5
Extracting projection vector 4 out of 5
Frobenius norm error: 1[...]
{'fraction_non_alphanumeric': 0.12715003969304048, 'fraction_numerical': 0.05887801005557026, 'mean_word_length': 4.5175182481751825, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '183962', 'license': 'MIT', 'n_tokens_mistral': 3213, 'n_tokens_neox': 2958, 'n_words': 726, 'path': 'Sat_data_objetivo_toolbox.ipynb', 'repo_name': 'jeroarenas/pysparkMVA', 'size': '7558'}