<jupyter_start><jupyter_text>Analyzing Real vs. Fake News Article Headlines 📰Author:[<NAME>](http://navierula.github.io)Data Source: [Randomly-Collected Fake News Dataset](https://github.com/BenjaminDHorne/fakenewsdata1)Resources Consulted: [Text Mining with R](http://tidytextmining.com)[R: Text Classification using a K Nearest Neighbour Model](http://garonfolo.dk/herbert/2015/05/r-text-classification-using-a-k-nearest-neighbour-model/)<jupyter_code># turn off warnings
options(warn=-1)
# import libraries
library(dplyr)
library(e1071)
library(ggplot2)
library(tidytext)
library(stringr)
library(RColorBrewer)
library(tm) 
library(class) 
library(SnowballC)
# load in dataset
mydata = read.csv("cleaned_data/headlines.txt",sep="\t",stringsAsFactors = FALSE,col.names=c("text", "status"),fill=TRUE)
# remove rows with empty values
mydata = mydata[!apply(mydata, 1, function(x) any(x=="")),] 
# preview the first five rows
# (mostly fake articles at the top)
head(mydata)
# preview the last five rows
# (mostly real articles at the bottom)
tail(mydata)
# calculate term/word frequency for words present in articles
news_words <- mydata %>%
  unnest_tokens(word, text) %>%
  count(status, word, sort = TRUE) %>%
  ungroup()

total_words <- news_words %>% 
  group_by(status) %>% 
  summarize(total = sum(n))

news_words <- left_join(news_words, total_words)

news_words<jupyter_output>Joining, by = "status"<jupyter_text>From the table above, we can see that the word "trump" is not only the most commonly used word in real news article headlines, but also the most commonly used world overall. This makes sense given the past election cycle. Out of 633 total words that appeared in real news article headlines, the word "trump" appeared 28 times, or rather 4.4% overall.In fake news article headlines, the most commonly used word was "obama," and following that, "trump" once again. These words appeared 20 out of 842 times and 19 out of 842 times. Respectively, 2.3% and 2.2%.<jupyter_code># visualize word counts in buckets
ggplot(news_words, aes(n/total, fill = status)) +
  geom_histogram(show.legend = TRUE,binwidth = 30,color="black") +
  facet_wrap(~status, ncol = 4)<jupyter_output><empty_output><jupyter_text>The visualization above simply counts the number of words present in each type of headline. For fake news headlines, the total number of words is 842. The total number of words for real news headlines is 633. Considering the fact that the particular dataset that I am using contains less real news headlines rather than fake news articles, the counts make sense.<jupyter_code>sprintf("The number of real news headlines in my dataset is: %d", str_count(mydata, "real")[2])
sprintf("The number of fake news headlines in my dataset is: %d", str_count(mydata, "fake")[2])
# calculate frequency by rank, using Zipf's law
freq_by_rank <- news_words %>% 
  group_by(status) %>% 
  mutate(rank = row_number(), 
         `term frequency` = n/total)

freq_by_rank<jupyter_output><empty_output><jupyter_text>The rank describes the rank of each word in the frequency table. It is plotted below, showing a constant negative slope.<jupyter_code>myColors <- c("gold4", "mediumorchid4")

# plot Zipf's law
freq_by_rank %>% 
  ggplot(aes(rank, `term frequency`, col=status)) + 
  geom_line(size = 3, alpha = 0.8) + 
  scale_x_log10() +
  scale_y_log10() + 
  scale_color_manual(values=myColors)<jupyter_output><empty_output><jupyter_text>From the above graph, we can see that words associate with real news headlines have a higher rank - which is not surprising. I will now use TF-IDF (Term Frequency–Inverse Document Frequency) to find the most relevant word for each article headline. According to [tidytextmining](http://tidytextmining.com/tfidf.htmlterm-frequency-in-jane-austens-novels), "the idea of tf-idf is to find the important words for the content of each document by decreasing the weight for commonly used words and increasing the weight for words that are not used very much in a collection or corpus of documents."TF-IDF may be a good method to use in regards to understanding contents of a document (or headline, in our case) because it finds words that are common, but not too common. This perhaps get rids of words that are unnecessary or irrelevant.<jupyter_code>news_words <- news_words %>%
  bind_tf_idf(word, status, n)
news_words<jupyter_output><empty_output><jupyter_text>We can see that tf-idf scores are ZERO for words that are very common. They appear in both types of news headlines. The idf will be low for such words and higher for words that appear in fewer headlines.<jupyter_code># order terms by highest tf-idf score
news_words %>%
  select(-total) %>%
  arrange(desc(tf_idf))
myColors <- c("rosybrown3", "darkseagreen4")

# plot top 30 words by tf-idf
plot_ <- news_words %>%
  arrange(desc(tf_idf)) %>%
  mutate(word = factor(word, levels = rev(unique(word))))

plot_ %>% 
  top_n(30) %>%
  ggplot(aes(word, tf_idf, fill = status)) +
  geom_bar(stat="identity") +
  scale_fill_manual(values=myColors) + #scale_fill instead of scale_col to fill color manually
  labs(x = "words", y = "tf-idf") +
  coord_flip() 
myColors <- c("lightpink1", "cornflowerblue")

# plot by grouping for top 25 words
plot_ %>% 
  group_by(status) %>% 
  top_n(25) %>% 
  ungroup %>%
  ggplot(aes(word, tf_idf, fill = status)) +
  geom_col(show.legend = FALSE) +
  labs(x = "word", y = "tf-idf") +
  facet_wrap(~status, ncol = 2, scales = "free") +
  scale_fill_manual(values=myColors) +
  coord_flip()<jupyter_output>Selecting by tf_idf<jupyter_text>News Classifier Using K-Nearest Neighbors Algorithm<jupyter_code># turn off warnings
options(warn=-1)
#install.packages("RTextTools") #try installing this as a package
# set seed value
set.seed(100)
# generate headlines corpus
headlines <- Corpus(VectorSource(mydata$text))
# clean headlines
headlines <- tm_map(headlines, content_transformer(tolower))
headlines <- tm_map(headlines, removeNumbers)
headlines <- tm_map(headlines, removeWords, stopwords("english"))
headlines <- tm_map(headlines, removePunctuation)
headlines <- tm_map(headlines, stripWhitespace)
headlines <- tm_map(headlines, stemDocument, language = "english")
# create document-term matrix
dtm <- DocumentTermMatrix(headlines)
# transforms document-term matrix to dataframe
mat.mydata <- as.data.frame(data.matrix(dtm), stringsAsfactors = FALSE)
# column bind on status
mat.mydata <- cbind(mat.mydata, mydata$status)
# Change name of new column to "status"
colnames(mat.mydata)[ncol(mat.mydata)] <- "status"
all <- 0
max = -Inf 
for (i in 1:1000)
{
 
  # split data into train and test sets
  train <- sample(nrow(mat.mydata), ceiling(nrow(mat.mydata) * .50))
  test <- (1:nrow(mat.mydata))[- train]
    
  # assign classifier
  classifier <- mat.mydata[, "status"]
  modeldata <- mat.mydata[,!colnames(mat.mydata) %in% "status"]
    
  # make predictions using knn algo
  knn_predictions <- knn(modeldata[train, ], modeldata[test, ], classifier[train])
    
  # create confusion matrix
  confusion_matrix <- table("Predictions" = knn_predictions, Actual = classifier[test])
  accuracy <- sum(diag(confusion_matrix))/length(test) * 100
  all = all + accuracy
  if (accuracy > max) {
      max <- accuracy # find max accuracy
      print(max)
      print(confusion_matrix)
    }
}

all/1000<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11133768352365415, 'fraction_numerical': 0.011827079934747145, 'mean_word_length': 5.182352941176471, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 48, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '20731', 'license': 'MIT', 'n_tokens_mistral': 2518, 'n_tokens_neox': 2339, 'n_words': 753, 'path': 'analysis.ipynb', 'repo_name': 'navierula/language-in-real-and-fake-news', 'size': '7363'}