<jupyter_start><jupyter_text>Analizemos los valores medios y la varianza de la data  Usar todas las columnas menos las columnas 'class' como características<jupyter_code>features = df.drop(['class'], axis=1)
target = df['class']

# Dividimos el dataset para el entrenamiento 80:20 %
f_train, f_dev, t_train, t_dev = train_test_split(features, target, test_size=0.2, random_state=42)
np.unique(target)<jupyter_output><empty_output><jupyter_text>Como se usan numeros reales, se necesita binarizar en dos clases: 0 normal 1 falla<jupyter_code># threshold classes into 0 and 1(failure)
lb = Binarizer()
lb.fit(t_train.values.reshape(-1, 1))
t_train_bi = lb.transform(t_train.values.reshape(-1, 1))
t_dev_bi = lb.transform(t_dev.values.reshape(-1, 1))
np.unique(t_train_bi)<jupyter_output><empty_output><jupyter_text>Para usar PCA necesitamos normalizar la data ya que se encuentra escalada<jupyter_code>from sklearn.preprocessing import scale
f_train_norm = scale(f_train)
f_dev_norm = scale(f_dev)
# Comparemos los datos con y sin normalización:
print('Antes de normalizar:\n', f_train[:2])
print('\nDespués de normalizar:\n', f_train_norm[:2])<jupyter_output>('Antes de normalizar:\n',         aa_000    ab_000   ac_000    ad_000    ae_000    af_000    ag_000  \
48572 -0.40625 -0.289062 -0.46875 -0.007812 -0.046875 -0.054688 -0.007812   
38696 -0.40625 -0.289062 -0.46875 -0.007812 -0.046875 -0.054688 -0.007812   

        ag_001    ag_002    ag_003    ...       ee_002    ee_003  ee_004  \
48572 -0.03125 -0.054688 -0.117188    ...    -0.382812 -0.382812  -0.375   
38696 -0.03125 -0.054688 -0.117188    ...    -0.382812 -0.382812  -0.375   

         ee_005  ee_006    ee_007    ee_008    ee_009    ef_000    eg_000  
48572 -0.351562 -0.3125 -0.195312 -0.304688 -0.171875 -0.023438 -0.023438  
38696 -0.351562 -0.3125 -0.195312 -0.304688 -0.171875 -0.023438 -0.023438  

[2 rows x 170 columns])
('\nDespu\xc3\xa9s de normalizar:\n', array([[-7.66026639e-01, -6.09221700e-01, -4.77337115e-01,
        -1.66564227e-02, -1.25460568e-01, -1.26119208e-01,
        -3.76524017e-02, -7.64080605e-02, -1.31434087e-01,
        -2.26290675e-01, -3.50083086e-01, -6[...]<jupyter_text>Gráfico de la varianza<jupyter_code># Varianza de los componentes principales
# plt.bar(pca.explained_variance_)

plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of components')
plt.ylabel('cumulative explained variance');
plt.bar(range(60),pca.explained_variance_ratio_)
from sklearn.decomposition import PCA
#Seleccionadmos los principales componentes de PCA
pca = PCA(n_components=60).fit(f_train_norm)

f_pca_train = pca.transform(f_train_norm)
f_pca_dev = pca.transform(f_dev_norm)<jupyter_output><empty_output><jupyter_text>Generamos el training and test set<jupyter_code># Separando data de entrenamiento y prueba
X_train = f_pca_train
X_dev = f_pca_dev
y_train = t_train_bi.reshape(-1,1)
y_dev = t_dev_bi.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>Entrenamos Linear Regression<jupyter_code>LR = LinearRegression()
LR.fit(X_train, y_train)
prediction = LR.predict(X_dev)
pred = (prediction > 0.5).astype(int)
print(classification_report(y_dev, pred))
resultado = confusion_matrix(y_dev, pred)
print(resultado)<jupyter_output>[[11772    16]
 [  134    78]]<jupyter_text>Resultado Linear Regression<jupyter_code>print "Falsos negativos",resultado[1,0]
print "Falsos positivos",resultado[0,1]
print "Costo total", (resultado[1,0]*500 + resultado[0,1]*10)<jupyter_output>Falsos negativos 134
Falsos positivos 16
Costo total 67160<jupyter_text>Entrenamos Regresión Logística<jupyter_code>LR = LogisticRegression()
LR.fit(X_train, y_train)

predictions = LR.predict(X_dev)
pred = (predictions > 0.5).astype(int)
print(classification_report(y_dev, pred))
resultado = confusion_matrix(y_dev, pred)
print(resultado)<jupyter_output>[[11760    28]
 [  115    97]]<jupyter_text>Resultado Regresión Logística<jupyter_code>print "Falsos negativos",resultado[1,0]
print "Falsos positivos",resultado[0,1]
print "Costo total", (resultado[1,0]*500 + resultado[0,1]*10)<jupyter_output>Falsos negativos 115
Falsos positivos 28
Costo total 57780
{'fraction_non_alphanumeric': 0.12503017137340092, 'fraction_numerical': 0.14120202751629254, 'mean_word_length': 4.945480631276901, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '353054', 'license': 'MIT', 'n_tokens_mistral': 2045, 'n_tokens_neox': 1706, 'n_words': 322, 'path': 'Regresion Logistica/experimentacion.ipynb', 'repo_name': 'ursulacervantes/scania-trucks', 'size': '4143'}