<jupyter_start><jupyter_text>**Name:** \_\_\_\_\_**EID:** \_\_\_\_\_  CS4487 - Tutorial 4: Predicting Popularity of Online NewsIn this tutorial you will train regression models to predict the number of "shares" of a news article on Mashable.First we need to initialize Python.  Run the below cell.<jupyter_code>%matplotlib inline
import IPython.core.display         
# setup output image format (Chrome works best)
IPython.core.display.set_matplotlib_formats("svg")
import matplotlib.pyplot as plt
import matplotlib
from numpy import *
from sklearn import *
import glob
import os
import csv
import string
random.seed(100)<jupyter_output><empty_output><jupyter_text>1. Loading Data and Pre-processingNext we need to load the data.  Download `OnlineNewsPopularity.zip`, and **unzip** it in the same directory as this ipynb file.  Then run the following cell to load the data.<jupyter_code>filename = 'OnlineNewsPopularity/OnlineNewsPopularity.csv'

# read the data
allfeatnames = []
textdata      = []
with open(filename, 'r') as f:
    reader = csv.reader(f)
    for row in reader:
        if len(allfeatnames)==0:
            allfeatnames = row
        else:
            textdata.append(row)

# put the data into a np array
dataX = empty((len(textdata), len(allfeatnames)-3))
dataY = empty(len(textdata))
for i,row in enumerate(textdata):
    # extract features (remove the first 2 features and the last feature)
    dataX[i,:] = array([float(x) for x in row[2:-1]])
    # extract target (last entry)
    dataY[i] = float(row[-1])
    
# extract feature names
featnames = [x.strip() for x in allfeatnames[2:-1]]

# extract a subset of data
dataX = dataX[::6]
dataY = dataY[::6]

print(dataX.shape)
print(dataY.shape)<jupyter_output><empty_output><jupyter_text>There are 58 features for each article.  Here are the feature names, and an example entry.  The actual description of the features can be found in the `OnlineNewsPopularity-features.txt` file.<jupyter_code>print(featnames)

print("--- example article features---")
print(dataX[0])
print("--- example article target (# of shares)")
print(dataY[0])<jupyter_output><empty_output><jupyter_text>Now separate the data into training and testing sets.<jupyter_code># randomly split data into 50% train and 50% test set
trainX, testX, trainYo, testYo = \
  model_selection.train_test_split(dataX, dataY, 
  train_size=0.50, test_size=0.50, random_state=4487)

print(trainX.shape)
print(testX.shape)<jupyter_output><empty_output><jupyter_text>Finally, we normalize the input features.<jupyter_code># normalize feature values
# this makes comparing weights more meaningful
scaler = preprocessing.StandardScaler()  
trainXn = scaler.fit_transform(trainX)  
testXn  = scaler.transform(testX)<jupyter_output><empty_output><jupyter_text>Because the target value (number of shares) has a large dynamic range, we will transform the target values through the log function.<jupyter_code># map targets to log-space
trainY = log10(trainYo)
testY  = log10(testYo)

plt.figure(figsize=(10,3))
plt.subplot(1,2,1)
plt.hist(trainYo, 25);
plt.title('histogram of Y values')
plt.subplot(1,2,2)
plt.hist(trainY, 25);
plt.title("histogram of log(Y) values");<jupyter_output><empty_output><jupyter_text>2. Prediction with Linear RegressionFirst we will look at predicting the number of shares using simple linear regression models.  Use the training data to fit a linear model using Ordinary Least Squares and Ridge Regression.  Use cross-validation on the training set to select the optimal $\alpha$ parameter for ridge regression.<jupyter_code>### INSERT YOUR CODE HERE
## HINT: 
# 1. Ordinary Least Squares: linear_model.LinearRegression()
# 2. Ridge Regression: linear_model.Ridge(alphas= )
# 3. Rigge Regression with Cross-validation: linear_model.Ridge(alphas= )<jupyter_output><empty_output><jupyter_text>Compare the two models using the _average absolute error_ (AE) between the predictions and the true values.  Below is  code that will calculate AE for the training and test sets.<jupyter_code># ols is the linear regression model
trainAE = metrics.mean_absolute_error(trainY, ols.predict(trainXn))
testAE  = metrics.mean_absolute_error(testY, ols.predict(testXn))
print("OLS: train error =", trainAE)
print("OLS: test error =", testAE)
# rr is the ridge regression model
trainAE = metrics.mean_absolute_error(trainY, rr.predict(trainXn))
testAE  = metrics.mean_absolute_error(testY, rr.predict(testXn))
print("RR: train error =", trainAE)
print("RR: test error =", testAE)<jupyter_output><empty_output><jupyter_text>Which model has better prediction ability on the test set? Why?- **INSERT YOUR ANSWER HERE**  3. Which features are important?Next we will investigate which features are the most important for the prediction.  Use LASSO with cross-validation to learn the model, and print the training and testing error.<jupyter_code>### INSERT YOUR CODE HERE
## HINT
# 1. LASSO with Cross-validation: linear_model.LassoCV()<jupyter_output><empty_output><jupyter_text>Examine the LASSO coefficients by sorting them in descending order.<jupyter_code># sort coefficients from smallest to largest, then reverse it
inds = argsort(abs(las.coef_))[::-1]
# print out
print("weight : feature description")
for i in inds:
    print("{: .3f} : {}".format(las.coef_[i], featnames[i]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1033320853612879, 'fraction_numerical': 0.011606140022463497, 'mean_word_length': 5.452898550724638, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '260224', 'license': 'MIT', 'n_tokens_mistral': 1761, 'n_tokens_neox': 1679, 'n_words': 545, 'path': 'tutorial-4/Tutorial4.ipynb', 'repo_name': 'BigtoC/CS4487-JPYNBs', 'size': '5342'}