<jupyter_start><jupyter_text>Exercise 9 Mashable news stories analysisPredicting if a news story is going to be popular<jupyter_code>import pandas as pd

url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/mashable.csv'
df = pd.read_csv(url, index_col=0)
df.head()
df.shape
X = df.drop(['url', 'Popular'], axis=1)
y = df['Popular']
y.mean()
# train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)<jupyter_output><empty_output><jupyter_text>Exercise 9.1Estimate a Decision Tree Classifier and a Logistic RegresionEvaluate using the following metrics:* Accuracy* F1-Score<jupyter_code>import numpy as np

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

models = {'lr': LogisticRegression(),
          'dt': DecisionTreeClassifier(),
          }

# Train all the models
for model in models.keys():
    models[model].fit(X_train, y_train)
    
# predict test for each model
y_pred = pd.DataFrame(index=X_test.index, columns=models.keys())
for model in models.keys():
    y_pred[model] = models[model].predict(X_test)
    
# Evaluate each model
from sklearn.metrics import mean_squared_error
from sklearn import metrics

for model in models.keys():
    print(model," mse ",np.sqrt(mean_squared_error(y_pred[model], y_test)))
    print(model," f1 ",metrics.f1_score(y_pred[model], y_test))
    print(model," accuracy ",metrics.accuracy_score(y_pred[model], y_test))<jupyter_output>/Users/davidfajardo/anaconda3/lib/python3.5/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)<jupyter_text>Exercise 9.2Estimate 300 bagged samplesEstimate the following set of classifiers:* 100 Decision Trees where max_depth=None* 100 Decision Trees where max_depth=2* 100 Logistic Regressions<jupyter_code>#Estimate 300 bagged samples

np.random.seed(123)
n_samples = X_train.shape[0]
n_B = 300
samples = [np.random.choice(a=n_samples, size=n_samples, replace=True) for _ in range(1, n_B +1 )]
samples
#Estimate the following set of classifiers:
#100 Decision Trees where max_depth=None
#100 Decision Trees where max_depth=2
#100 Logistic Regressions

seeds = np.random.randint(1, 10000, size=n_B)

trees = {}
for i in range(0,100):
    trees[i] = DecisionTreeClassifier(max_features="sqrt", max_depth=None, random_state=seeds[i])
    trees[i].fit(X_train.iloc[samples[i]], y_train.iloc[samples[i]])

for i in range(100,200):
    trees[i] = DecisionTreeClassifier(max_features="sqrt", max_depth=2, random_state=seeds[i])
    trees[i].fit(X_train.iloc[samples[i]], y_train.iloc[samples[i]])
    
for i in range(200,300):
    trees[i] = LogisticRegression(random_state=seeds[i],solver='liblinear')
    trees[i].fit(X_train.iloc[samples[i]], y_train.iloc[samples[i]])
# Predict 
y_pred_df = pd.DataFrame(index=X_test.index, columns=list(range(n_B)))
for i in range(n_B):
    y_pred_df.iloc[:, i] = trees[i].predict(X_test)

y_pred_df.head()<jupyter_output><empty_output><jupyter_text>Exercise 9.3Ensemble using majority votingEvaluate using the following metrics:* Accuracy* F1-Score<jupyter_code>#Ensemble using majority voting
y_pred_df.sum(axis=1)[:10]
y_pred = (y_pred_df.sum(axis=1) >= (n_B / 2)).astype(np.int)

from sklearn import metrics
print("f1 score: ",metrics.f1_score(y_pred, y_test))
print("accuracy: ",metrics.accuracy_score(y_pred, y_test))<jupyter_output>f1 score:  0.6257839721254356
accuracy:  0.642<jupyter_text>Exercise 9.4Estimate te probability as %models that predict positiveModify the probability threshold and select the one that maximizes the F1-Score<jupyter_code>probability = y_pred_df.sum(axis=1) / n_B
probability.head()
step_nums = 20
nums = np.arange(0, 300, step=step_nums)
nums
probs = nums/n_B
probs
len(probs)
iter=[]
for i in range(15):
    y_pred_f1 = (y_pred_df.sum(axis=1)/n_B >= probs[i]).astype(np.int)
    iter.append([i,probs[i],metrics.f1_score(y_pred_f1, y_test)])
    
iter.sort(key=lambda x: x[2], reverse = True)
print(iter[0])
print ("\n")
print ("La probabilidad que maximiza el f-score es: Prob=0.2 , con un f1-score= 0.71")<jupyter_output>[3, 0.2, 0.7073552425665103]


La probabilidad que maximiza el f-score es: Prob=0.2 , con un f1-score= 0.71<jupyter_text>Exercise 9.5Ensemble using weighted voting using the oob_errorEvaluate using the following metrics:* Accuracy* F1-Score<jupyter_code>#Select each oob sample

samples_oob = []
# show the "out-of-bag" observations for each sample
for sample in samples:
    samples_oob.append(sorted(set(range(n_samples)) - set(sample)))
#Estimate the oob error of each classifier

errors = np.zeros(n_B)

for i in range(n_B):
    y_pred_ = trees[i].predict(X_train.iloc[samples_oob[i]])
    errors[i] = 1 - metrics.accuracy_score(y_train.iloc[samples_oob[i]], y_pred_)
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')

plt.scatter(range(n_B), errors)
plt.xlim([0, n_B])
plt.title('OOB error of each tree')
# Estimate $\alpha$
alpha = (1 - errors) / (1 - errors).sum()
weighted_sum_1 = ((y_pred_df) * alpha).sum(axis=1)
weighted_sum_1.head(20)
y_pred = (weighted_sum_1 >= 0.5).astype(np.int)

print("f1: ",metrics.f1_score(y_pred, y_test))
print("accuracy: ",metrics.accuracy_score(y_pred, y_test))<jupyter_output>f1:  0.6252616887648289
accuracy:  0.642<jupyter_text>Exercise 9.6Estimate te probability of the weighted votingModify the probability threshold and select the one that maximizes the F1-Score<jupyter_code>weighted=[]
for i in range(15):
    y_pred_w = (weighted_sum_1 >= probs[i]).astype(np.int)
    weighted.append([i,probs[i],metrics.f1_score(y_pred_w, y_test)])

weighted
weighted.sort(key=lambda x: x[2], reverse = True)
print(weighted[0])
print ("\n")
print ("La probabilidad que maximiza el f-score es: Prob=0.2 , con un f1-score= 0.71")<jupyter_output>[3, 0.2, 0.7055084745762711]


La probabilidad que maximiza el f-score es: Prob=0.2 , con un f1-score= 0.71<jupyter_text>Exercise 9.7Estimate a logistic regression using as input the estimated classifiersModify the probability threshold such that maximizes the F1-Score<jupyter_code>X_train_2 = pd.DataFrame(index=X_train.index, columns=list(range(n_B)))

for i in range(n_B):
    X_train_2[i] = trees[i].predict(X_train)
X_train_2.head()
from sklearn.linear_model import LogisticRegressionCV
lr = LogisticRegressionCV(cv = 5 )
lr.fit(X_train_2, y_train)
y_pred = lr.predict(y_pred_df)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
y_pred = (lr.predict_proba(y_pred_df)[:,1]>=0.5).astype(np.int)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
stacked=[]
for i in range(15):
    y_pred_s = (lr.predict_proba(y_pred_df)[:,1]>=probs[i]).astype(np.int)
    stacked.append([i,probs[i],metrics.f1_score(y_pred_s, y_test)])

stacked
stacked.sort(key=lambda x: x[2], reverse = True)
print(stacked[0])
print ("\n")
print ("La probabilidad que maximiza el f-score es: Prob=",stacked[0][1]," , con un f1-score= ",stacked[0][2])<jupyter_output>[5, 0.3333333333333333, 0.7126673532440783]


La probabilidad que maximiza el f-score es: Prob= 0.3333333333333333  , con un f1-score=  0.7126673532440783
{'fraction_non_alphanumeric': 0.13091504159279968, 'fraction_numerical': 0.04554752488749489, 'mean_word_length': 6.134241245136187, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '207327', 'license': 'Apache-2.0', 'n_tokens_mistral': 2996, 'n_tokens_neox': 2767, 'n_words': 605, 'path': 'E9-Bagging.ipynb', 'repo_name': 'dfajardo608/MachineLearningClass', 'size': '7333'}