<jupyter_start><jupyter_text>Hacker News is a social news website focusing on computer science and entrepreneurship. It has a community where users can submit articles, and other users can upvote those articles. Like other website the articles with the most upvotes make it to the front page.This data set consists of submissions users made to Hacker News from 2006 to 2015. Developer <NAME> used the Hacker News API to scrape the data, which which can be  found in one of his GitHub repositories. https://github.com/arnauddri/hnhn_stories is a 3000 rows that was sampled from the data randomly, and it has only has four columns:* submission_time - When the article was submitted* upvotes - The number of upvotes the article received* url - The base URL of the article* headline - The article's headlineI'll be predicting the number of upvotes the articles received, based on their headlines. Upvotes are an indicator of popularity, I will try to discover which types of articles tend to be the most popular it this community. -------<jupyter_code>import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import mean_squared_error

# Reading the file to datframe
articals = pd.read_csv("hn_stories.csv")
articals.columns = ["submission_time", "upvotes", "url", "headline"]

# Exploring the data
articals.head()

articals.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2999 entries, 0 to 2998
Data columns (total 4 columns):
submission_time    2999 non-null object
upvotes            2999 non-null int64
url                2810 non-null object
headline           2989 non-null object
dtypes: int64(1), object(3)
memory usage: 93.8+ KB<jupyter_text>As we can see here, we have four columns three with an object data type and one numerical.There are some missing values, so I am going to remove those rows first<jupyter_code>articals = articals.dropna()<jupyter_output><empty_output><jupyter_text>-----Since I am going to train a linear regression model to predicts the number of upvotes a headline would receive, I will need to convert each headline to a numerical representation.There are several ways to do this, I will use the bag of words model where each piece of text is represented as a numerical vector.The first step in creating a bag of words model is tokenization. I am going to split each sentence into a list of individual words on the space character<jupyter_code># Creating a list for the tokens
headline_tokens = []

# Looping through the dataframe to split headlines and add words as a list 
for row in articals['headline']:
    headline_tokens.append(row.split())
    
headline_tokens[:2]<jupyter_output><empty_output><jupyter_text>----On my next step, I am going to remove punctuation lowercas all words.<jupyter_code># A list of punctuation to be removed from tokens
punctuation = [ "/", "-", "+", "&", "(", ")", ",", ":", ";", ".", "'", '"', "’", "?"]

# New list for the processed tokens
clean_tokens = []

# A loop to go through tokens and lower case each word and remove punctuation 
for tokens in headline_tokens:
    #list for each sentance
    tokens_list = []
    for token in tokens:
        token = token.lower()
        for p in punctuation:
            token = token.replace(p, '')
        tokens_list.append(token)
    clean_tokens.append(tokens_list)
    
clean_tokens[:2]<jupyter_output><empty_output><jupyter_text>-----Now I am going to create a dataframe for all the unique words to convert the sentences to their numerical representations.I will only keep tokens that occured more than one time, tokens that only occurred once don't add to the model's prediction power.<jupyter_code># A list for tokens that only occured once
single_tokens = []
# List for unique tokens
unique_tokens = []

# loop through the clean tokens list to add the unique once
for tokens in clean_tokens:
    for token in tokens:
        if token not in single_tokens:
            single_tokens.append(token)
        elif token not in unique_tokens:
            unique_tokens.append(token)

# creating a dataframe for all unique tokens as columns names and intialize it with 0 and same size as the clean_tokens list
tokens_df = pd.DataFrame(0, index=np.arange(len(clean_tokens)), columns = unique_tokens)

tokens_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2800 entries, 0 to 2799
Columns: 2310 entries, and to disaster
dtypes: int64(2310)
memory usage: 49.4 MB<jupyter_text>-----I will loop through the dataframe I just created and add 1 to the tokens if it is in the list of unique tokens<jupyter_code>for index, tokens in enumerate(clean_tokens):
    for token in tokens:
        if token in unique_tokens:
            tokens_df.iloc[index][token] += 1
            
tokens_df.iloc[:10,:20]<jupyter_output><empty_output><jupyter_text>To further inhance my prediciton, I am going to remove words that occured more than 100 times which should remove stopwords like 'and' and 'for', which occurs almost in every headline and words that occured less than 5 times.<jupyter_code># get the sum of each token occurrence 
word_counts = tokens_df.sum(axis = 0)

# remove tokens fewer than 5 and more than 100
tokens_df = tokens_df.loc[:, (word_counts >= 5) & (word_counts <= 100)]
tokens_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2800 entries, 0 to 2799
Columns: 661 entries, as to nike
dtypes: int64(661)
memory usage: 14.1 MB<jupyter_text>we reduced tokens from 2310 to 661 I will split the data into two sets: 80% training set and 20% test set, and I will use the train_test_split function from sckit-learn to do that<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(tokens_df, articals['upvotes'], test_size = 0.2)

# initializing the linear regression model
lr = LinearRegression()

# Fitting the model
lr.fit(x_train, y_train)

# Calculating predictions on the test data
predictions = lr.predict(x_test)



mean = np.mean(predictions)
std  = np.std(predictions)
mse = mean_squared_error(predictions, y_test)
rmse = np.sqrt(mse)

print('Mean = ', mean)
print('Srandard Deviation = ', std)
print('Mean Squared Error = ', mse)
print('Squared root of mse = ', rmse)<jupyter_output>Mean =  10.32407204900292
Srandard Deviation =  29.084076705842442
Mean Squared Error =  1815.5326272886325
Squared root of mse =  42.60906743040303<jupyter_text>Mean squared error is 1815 which is a fairly larg number. The mean  of upvotes is 10, and the standard deviation is 29.The square root of mse is 42.6 which means that the average error is 46 upvotes away from the true value which higher than the standard deviation, so my predictions are off-base.I will try a random forest model, maybe I can get better results.<jupyter_code># initializing the the model
rf = RandomForestClassifier(random_state = 1)

#fitting the model
rf.fit(x_train, y_train)

# Calculating predictions
predictions = rf.predict(x_test)

mean = np.mean(predictions)
std  = np.std(predictions)
mse = mean_squared_error(predictions, y_test)
rmse = np.sqrt(mse)

print('Mean = ', mean)
print('Srandard deviation = ', std)
print('Mean squared error = ', mse)
print('Squared root of mse = ', rmse)<jupyter_output>Mean =  6.1553571428571425
Srandard deviation =  36.77478474076486
Mean squared error =  2353.6696428571427
Squared root of mse =  48.51463328581535
{'fraction_non_alphanumeric': 0.077632282202031, 'fraction_numerical': 0.03674505611972207, 'mean_word_length': 4.536242603550296, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '423109', 'license': 'MIT', 'n_tokens_mistral': 2449, 'n_tokens_neox': 2207, 'n_words': 912, 'path': 'NLP Hackers News.ipynb', 'repo_name': 'Talawadhi/Machine-Learning', 'size': '7492'}