<jupyter_start><jupyter_text>HOW TO SUMMARIZE IN PYTHONFollowing [this tutorial!](https://stackabuse.com/text-summarization-with-nltk-in-python/) | 10-13-19 STEP 1: GET THE DATA!!  Step 1a: Import libraries<jupyter_code>import bs4 as bs
import urllib.request
import re<jupyter_output><empty_output><jupyter_text>Step 1b: Use the libraries to scrape the WHOLE INTERNET!! (jk just this page)<jupyter_code># url = 'https://en.wikipedia.org/wiki/Lizard'
# url = 'https://en.wikipedia.org/wiki/cat'
url = 'https://en.wikipedia.org/wiki/Naive_Bayes_classifier'
# url = 'https://en.wikipedia.org/wiki/Machine_learning' # good at 20 words
# url = 'https://en.wikipedia.org/wiki/Artificial_intelligence' # good at 30 words
# scraped_data = urllib.request.urlopen('https://en.wikipedia.org/wiki/Artificial_intelligence')
# scraped_data = urllib.request.urlopen('https://en.wikipedia.org/wiki/Harry_Potter_and_the_Philosopher%27s_Stone')
scraped_data = urllib.request.urlopen(url)
article = scraped_data.read()
parsed_article = bs.BeautifulSoup(article,'lxml')<jupyter_output><empty_output><jupyter_text>Step 1c: Use `find_all` from `BeautifulSoup` to get all of the p tags<jupyter_code>paragraphs = parsed_article.find_all('p')
article_text = ""
for p in paragraphs:
    article_text += p.text
article_text[:1000]<jupyter_output><empty_output><jupyter_text>STEP 2: CLEAN (& preprocess) THE DATA!!   Step 2a: Use regex and `re.sub` to remove square brackets and extra spaces from ORIGINAL article_text<jupyter_code>article_text = re.sub(r'\[[0-9]*\]', '', article_text)
article_text = re.sub(r'\s+', ' ', article_text)
article_text[:1000]<jupyter_output><empty_output><jupyter_text>Step 2b: Use regex and `re.sub` to remove extra characters and digits for a new FORMATTED_TEXT variable<jupyter_code>formatted_article_text = re.sub('[^a-zA-Z]', ' ', article_text)
formatted_article_text = re.sub(r'\s+', ' ', formatted_article_text)
formatted_article_text[:1000]<jupyter_output><empty_output><jupyter_text>STEP 3: TOKENIZE SENTENCES!!<jupyter_code>import nltk
sentence_list = nltk.sent_tokenize(article_text)
sentence_list[:5]<jupyter_output><empty_output><jupyter_text>STEP 4: FIND WORD FREQUENCY, WEIGHTED!!  Step 4a: Remove Stopwords<jupyter_code>stopwords = nltk.corpus.stopwords.words('english')<jupyter_output><empty_output><jupyter_text>Step 4b: Tokenize Words & DIY Frequency Distribution<jupyter_code>word_frequencies = {}
for word in nltk.word_tokenize(formatted_article_text):
    if word not in stopwords:
        if word not in word_frequencies.keys():
            word_frequencies[word] = 1
        else:
            word_frequencies[word] += 1<jupyter_output><empty_output><jupyter_text>Step 4c: Calculate Weighted Frequency<jupyter_code>max_frequency = max(word_frequencies.values())
for word in word_frequencies.keys():
    word_frequencies[word] = (word_frequencies[word]/max_frequency)<jupyter_output><empty_output><jupyter_text>STEP 5: CALCULATE SENTENCE SCORES<jupyter_code>## ILLUSTRATIVE EXAMPLE
## Nothing removed
for sent in sentence_list[:1]:
    for word in nltk.word_tokenize(sent.lower()):
        print(word)
## ILLUSTRATIVE EXAMPLE
## Stopwords etc. removed
## We are ONLY assigning values/weights to the words in the sentences that are inside our freq dist!

for sent in sentence_list[:1]:
    for word in nltk.word_tokenize(sent.lower()):
        if word in word_frequencies.keys():
            print(word)
sentence_scores = {}
for sent in sentence_list:
    for word in nltk.word_tokenize(sent.lower())[:50]:
        if word in word_frequencies.keys():
            if len(sent.split(' ')) < 30:
                if sent not in sentence_scores.keys():
                    sentence_scores[sent] = word_frequencies[word]
                else:
                    sentence_scores[sent] += word_frequencies[word]
sorted_sentences = sorted(sentence_scores.items(), key=lambda kv: kv[1], reverse=True)
sorted_sentences[:10]
summary = [sent[0] for sent in sorted_sentences[:5]]
''.join(summary)
''.join(summary).strip()
summary_2 = [sent[0] for sent in sentence_scores.items() if sent[1] > 3]
''.join(summary_2).strip()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12469966362325804, 'fraction_numerical': 0.014175876982220087, 'mean_word_length': 5.4743390357698285, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '664742', 'license': 'MIT', 'n_tokens_mistral': 1533, 'n_tokens_neox': 1444, 'n_words': 291, 'path': 'assets/all_html/2019_10_13_HW2_summarization_v2_wikipedia.ipynb', 'repo_name': 'dskw1/dskw1.github.io', 'size': '4162'}