<jupyter_start><jupyter_text>Automatically Summarize Trump’s State of the Union Address  Importing Libraries<jupyter_code>from bs4 import BeautifulSoup
from urllib.request import urlopen
from nltk.tokenize import sent_tokenize
import numpy as np
import pandas as pd
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import cosine_similarity
import networkx as nx
import nltk
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.<jupyter_text>TextRank with NLTK  **TextRank is an unsupervised text summarization technique that uses the intuition behind the PageRank algorithm to rank sentences. When using NLTK for our project, we have the following steps:*** Fetch the State of Union Address on the internet.* Basic text cleaning.* Find vector representation (word embeddings) for each and every sentence.* Similarities between sentence vectors are then calculated and stored in a matrix.* The similarity matrix is converted into a graph, with sentences as vertices and similarity scores as edges.* Apply the PageRank algorithm over this sentence graph for sentence rank calculation.* print out several top ranked sentences.  Load and Read the data<jupyter_code>def get_only_text(url):
    """ 
    return the title and the text of the article
    at the specified url
    """
    page = urlopen(url)
    soup = BeautifulSoup(page, "lxml")
    text = ' '.join(map(lambda p: p.text, soup.find_all('p')))
  
    print ("=====================")
    print (text)
    print ("=====================")
 
    return soup.title.text, text    
 
     
url="https://www.whitehouse.gov/briefings-statements/remarks-president-trump-state-union-address-2/"
text = get_only_text(url)<jupyter_output>=====================

				Remarks			 
Issued on:
February 6, 2019
 February 5, 2019
9:07 P.M. EST THE PRESIDENT:  <NAME>, Mr. Vice President, Members of Congress, the First Lady of the United States — (applause) — and my fellow Americans: We meet tonight at a moment of unlimited potential.  As we begin a new Congress, I stand here ready to work with you to achieve historic breakthroughs for all Americans. Millions of our fellow citizens are watching us now, gathered in this great chamber, hoping that we will govern not as two parties but as one nation.  (Applause.) The agenda I will lay out this evening is not a Republican agenda or a Democrat agenda.  It’s the agenda of the American people. Many of us have campaigned on the same core promises: to defend American jobs and demand fair trade for American workers; to rebuild and revitalize our nation’s infrastructure; to reduce the price of healthcare and prescription drugs; to create an immigration system that is safe, lawful, mo[...]<jupyter_text>A look on few sentences<jupyter_code>sentences = []
for s in text:
    sentences.append(sent_tokenize(s))

sentences = [y for x in sentences for y in x]
sentences[30:40]<jupyter_output><empty_output><jupyter_text>We will use pre-trained word vectors to create vectors for the sentences  We will use pre-trained word vectors to create vectors for the sentences in the State of Union address. I have downloaded the data from Glove and saved in my working directory.<jupyter_code># Extract word vectors
word_embeddings = {}
f = open('glove.6B.100d.txt', encoding='utf-8')
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    word_embeddings[word] = coefs
f.close()
len(word_embeddings)<jupyter_output><empty_output><jupyter_text>Basic Preprocessing and Text cleanining<jupyter_code># remove punctuations, numbers and special characters
clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")

# change to lowercase
clean_sentences = [s.lower() for s in clean_sentences]
stop_words = stopwords.words('english')
# function to remove stopwords
def remove_stopwords(sen):
    sen_new = " ".join([i for i in sen if i not in stop_words])
    return sen_new
clean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]<jupyter_output><empty_output><jupyter_text>**In the following code scripts, we create vectors for the sentences. We first fetch vectors (each of size 100 elements) for the constituent words in a sentence and then take average of those vectors to arrive at a consolidated vector for the sentence. Finally print out top 15 ranked sentences as the summary representation.**<jupyter_code>sentence_vectors = []
for i in clean_sentences:
    if len(i) != 0:
        v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()])/(len(i.split())+0.001)
    else:
        v = np.zeros((100,))
    sentence_vectors.append(v)
# Create an empty similarity matrix
sim_mat = np.zeros([len(sentences), len(sentences)])
for i in range(len(sentences)):
    for j in range(len(sentences)):
        if i != j:
            sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,100), sentence_vectors[j].reshape(1,100))[0,0]
nx_graph = nx.from_numpy_array(sim_mat)
scores = nx.pagerank(nx_graph)
ranked_sentences = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)
# Extract top 15 sentences as the summary representation
for i in range(10):
    print(ranked_sentences[i][1])<jupyter_output>After 24 months of rapid progress, our economy is the envy of the world, our military is the most powerful on Earth, by far, and America — (applause) — America is again winning each and every day.
Millions of our fellow citizens are watching us now, gathered in this great chamber, hoping that we will govern not as two parties but as one nation.
I hope you can pass the USMCA into law so that we can bring back our manufacturing jobs in even greater numbers, expand American agriculture, protect intellectual property, and ensure that more cars are proudly stamped with our four beautiful words: “Made in the USA.”  (Applause.)
When Grace completed treatment last fall, her doctors and nurses cheered — they loved her; they still love her — with tears in their eyes as she hung up a poster that read: “Last day of chemo.”  (Applause.)
We do not know whether we will achieve an agreement, but we do know that, after two decades of war, the hour has come to at least try for peace.
Perhaps — (applause[...]<jupyter_text>Gensim<jupyter_code>import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
from gensim.summarization import summarize
from gensim.summarization import keywords
import urllib.request
from urllib.request import urlopen
from bs4 import BeautifulSoup
import requests
text = requests.get('https://www.whitehouse.gov/briefings-statements/remarks-president-trump-state-union-address-2/').text

print('Summary:')
print(summarize(text, ratio=0.01))

print('\nKeywords:')
print(keywords(text, ratio=0.01))<jupyter_output>2019-02-12 16:20:44,425 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2019-02-12 16:20:44,439 : INFO : built Dictionary(1371 unique tokens: ['bno', 'classnam', 'document', 'documentel', 'replac']...) from 539 documents (total 3371 corpus positions)<jupyter_text>Sumy **Sumy is a Python library for extracting summary from HTML pages or plain texts.**<jupyter_code>from __future__ import absolute_import
from __future__ import division, print_function, unicode_literals

 
from sumy.parsers.html import HtmlParser
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lsa import LsaSummarizer
from sumy.nlp.stemmers import Stemmer
from sumy.utils import get_stop_words
 
from sumy.summarizers.luhn import LuhnSummarizer
from sumy.summarizers.edmundson import EdmundsonSummarizer
from sumy.summarizers.lex_rank import LexRankSummarizer<jupyter_output>Collecting sumy
[?25l  Downloading https://files.pythonhosted.org/packages/2f/0e/30ebd2fb0925537a3b2f9fccf0a13171ba557e9450b1702d278159d3c592/sumy-0.7.0-py2.py3-none-any.whl (78kB)
[K    100% |████████████████████████████████| 81kB 7.4MB/s 
[?25hRequirement already satisfied: requests>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from sumy) (2.18.4)
Collecting breadability>=0.1.20 (from sumy)
  Downloading https://files.pythonhosted.org/packages/ad/2d/bb6c9b381e6b6a432aa2ffa8f4afdb2204f1ff97cfcc0766a5b7683fec43/breadability-0.1.20.tar.gz
Requirement already satisfied: nltk>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from sumy) (3.2.5)
Requirement already satisfied: docopt<0.7,>=0.6.1 in /usr/local/lib/python3.6/dist-packages (from sumy) (0.6.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.7.0->sumy) (2018.11.29)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (fr[...]<jupyter_text>**LsaSummarizer. ** - Latent Semantic Analysis which combines term frequency with singular value decomposition.<jupyter_code>LANGUAGE = "english"
SENTENCES_COUNT = 10
url="https://www.whitehouse.gov/briefings-statements/remarks-president-trump-state-union-address-2/"
parser = HtmlParser.from_url(url, Tokenizer(LANGUAGE))
print ("--LsaSummarizer--")    
summarizer = LsaSummarizer()
summarizer = LsaSummarizer(Stemmer(LANGUAGE))
summarizer.stop_words = get_stop_words(LANGUAGE)
for sentence in summarizer(parser.document, SENTENCES_COUNT):
    print(sentence)<jupyter_output>--LsaSummarizer--
In June, we mark 75 years since the start of what General D<NAME>enhower called the “Great Crusade” — the Allied liberation of Europe in World War II.
Now is the time for Congress to show the world that America is committed to ending illegal immigration and putting the ruthless coyotes, cartels, drug dealers, and human traffickers out of business.
Meanwhile, working-class Americans are left to pay the price for mass illegal migration: reduced jobs, lower wages, overburdened schools, hospitals that are so crowded you can’t get in, increased crime, and a depleted social safety net.
Tens of thousands of innocent Americans are killed by lethal drugs that cross our border and flood into our cities, including meth, heroin, cocaine, and fentanyl.
In the last two years, our brave ICE officers made 266,000 arrests of criminal aliens, including those charged or convicted of nearly 100,000 assaults, 30,000 sex crimes, and 4,000 killings or murders.
I hope you can pass the [...]<jupyter_text>**LuhnSummarizer.** -  A naive approach based on TF-IDF and looking at the “window size” of non-important words between words of high importance. It also assigns higher weights to sentences occurring near the beginning of a document.<jupyter_code>print ("--LuhnSummarizer--")     
summarizer = LuhnSummarizer() 
summarizer = LuhnSummarizer(Stemmer(LANGUAGE))
summarizer.stop_words = ("I", "am", "the", "you", "are", "me", "is", "than", "that", "this")
for sentence in summarizer(parser.document, SENTENCES_COUNT):
    print(sentence)<jupyter_output>--LuhnSummarizer--
Many of us have campaigned on the same core promises: to defend American jobs and demand fair trade for American workers; to rebuild and revitalize our nation’s infrastructure; to reduce the price of healthcare and prescription drugs; to create an immigration system that is safe, lawful, modern, and secure; and to pursue a foreign policy that puts America’s interests first.
Human traffickers and sex traffickers take advantage of the wide-open areas between our ports of entry to smuggle thousands of young girls and women into the United States and to sell them into prostitution and modern-day slavery.
Elvin says that, “If I can make sure these young girls get their justice, I’ve [really] done my job.” Thanks to his work, and that of his incredible colleagues, more than 300 women and girls have been rescued from the horror of this terrible situation, and more than 1,500 sadistic traffickers have been put behind bars.
The border city of El Paso, Texas used to have extre[...]<jupyter_text>**EdmundsonSummarizer. ** - When using EdmundsonSummarizer, we need to enter bonus_words which are the words we want to see in summary and are significant, stigma_words which are unimportant, null_words which are stop words.<jupyter_code>print ("--EdmundsonSummarizer--")     
summarizer = EdmundsonSummarizer() 
words1 = ("economy", "fight", "trade", "china")
summarizer.bonus_words = words1
     
words2 = ("another", "and", "some", "next")
summarizer.stigma_words = words2
    
words3 = ("another", "and", "some", "next")
summarizer.null_words = words3
for sentence in summarizer(parser.document, SENTENCES_COUNT):
    print(sentence)<jupyter_output>--EdmundsonSummarizer--
February 5, 2019
9:07 P.M. EST
We meet tonight at a moment of unlimited potential.
As we begin a new Congress, I stand here ready to work with you to achieve historic breakthroughs for all Americans.
Millions of our fellow citizens are watching us now, gathered in this great chamber, hoping that we will govern not as two parties but as one nation.
(Applause.)
The agenda I will lay out this evening is not a Republican agenda or a Democrat agenda.
It’s the agenda of the American people.
There is a new opportunity in American politics, if only we have the courage, together, to seize it.
Tonight, I am also asking you to join me in another fight that all Americans can get behind: the fight against childhood cancer.
END 10:29 P.M. EST<jupyter_text>**LexRankSummarizer.** -  Unsupervised approach inspired by algorithms PageRank. It finds the relative importance of all words in a document and selects the sentences which contain the most of those high-scoring words.<jupyter_code>print ("--LexRankSummarizer--")   
summarizer = LexRankSummarizer()
summarizer = LexRankSummarizer(Stemmer(LANGUAGE))
summarizer.stop_words = ("I", "am", "the", "you", "are", "me", "is", "than", "that", "this")
for sentence in summarizer(parser.document, SENTENCES_COUNT):
    print(sentence)<jupyter_output>--LexRankSummarizer--
Many of us have campaigned on the same core promises: to defend American jobs and demand fair trade for American workers; to rebuild and revitalize our nation’s infrastructure; to reduce the price of healthcare and prescription drugs; to create an immigration system that is safe, lawful, modern, and secure; and to pursue a foreign policy that puts America’s interests first.
Tonight, I am asking you to defend our very dangerous southern border out of love and devotion to our fellow citizens and to our country.
And thank you for being here.
When Elvin — (applause) — thank you.
To build on — (applause) — thank you.
I have great respect for President Xi, and we are now working on a new trade deal with China.
But we must do more.
As we make progress in these negotiations, we will be able to reduce our troop’s presence and focus on counterterrorism.
It’s time.
It’s the Americans!” (Applause.)
{'fraction_non_alphanumeric': 0.09077470939998657, 'fraction_numerical': 0.02176980447490425, 'mean_word_length': 5.095004095004095, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 50, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '130461', 'license': 'Apache-2.0', 'n_tokens_mistral': 4639, 'n_tokens_neox': 4327, 'n_words': 1774, 'path': "Summarize-Trump's-Remarks.ipynb", 'repo_name': 'VeerendraPappala/Analyise-Summarize-Trump-s-remarks-on-state-of-the-Union-Houses', 'size': '14896'}