<jupyter_start><jupyter_text>Text Summarization<jupyter_code>#We need to tokenize the article into sentences
#Sentence tokenization

sentence_list = nltk.sent_tokenize(clean_text)
#Weighted Frequency of Occurrence

stopwords = nltk.corpus.stopwords.words('english')

word_frequencies = {}
for word in nltk.word_tokenize(clean_text):
    if word not in stopwords:
        if word not in word_frequencies.keys():
            word_frequencies[word] = 1
        else:
            word_frequencies[word] += 1
maximum_frequncy = max(word_frequencies.values())

for word in word_frequencies.keys():
    word_frequencies[word] = (word_frequencies[word]/maximum_frequncy)
sentence_scores = {}
for sent in sentence_list:
    for word in nltk.word_tokenize(sent.lower()):
        if word in word_frequencies.keys():
            if len(sent.split(' ')) < 30:
                if sent not in sentence_scores.keys():
                    sentence_scores[sent] = word_frequencies[word]
                else:
                    sentence_scores[sent] += word_frequencies[word]
sentence_scores<jupyter_output><empty_output><jupyter_text>10 Key Ideas<jupyter_code>import heapq
summary_sentences = heapq.nlargest(10, sentence_scores, key=sentence_scores.get)

summary = ' '.join(summary_sentences)
print(summary)<jupyter_output>On the other hand, industrial demand is down, and in some areas, commercial demand in particular is off sharply. Thus, while not a huge deal, it’s enough to add materially to the company. On a valuation basis, Waste Management isn’t particularly cheap, as it sells with a price-earnings ratio of around 26. They send their trucks around periodically to your house, take the trash to the landfill, and that’s that. A Strong Competitive Advantage Waste Management has earned above-average profits, and thus delivered stellar shareholder returns, because its business is protected from many competitive pressures. The economy is experiencing rapid transformation right now, but there’s little sign it will change humans’ basic desire to consume vast quantities of goods and services. It, in turn, sells this landfill capacity to locally operated third parties that run their own disposal routes. It’s hard to argue that WM stock is especially cheap, but it’s a fine pick as a defensive play with a decen[...]<jupyter_text>15 Key Ideas<jupyter_code>import heapq
summary_sentences_2 = heapq.nlargest(15, sentence_scores, key=sentence_scores.get)

summary_2 = ' '.join(summary_sentences_2)
print(summary_2)<jupyter_output>On the other hand, industrial demand is down, and in some areas, commercial demand in particular is off sharply. Thus, while not a huge deal, it’s enough to add materially to the company. On a valuation basis, Waste Management isn’t particularly cheap, as it sells with a price-earnings ratio of around 26. They send their trucks around periodically to your house, take the trash to the landfill, and that’s that. A Strong Competitive Advantage Waste Management has earned above-average profits, and thus delivered stellar shareholder returns, because its business is protected from many competitive pressures. The economy is experiencing rapid transformation right now, but there’s little sign it will change humans’ basic desire to consume vast quantities of goods and services. It, in turn, sells this landfill capacity to locally operated third parties that run their own disposal routes. It’s hard to argue that WM stock is especially cheap, but it’s a fine pick as a defensive play with a decen[...]
{'fraction_non_alphanumeric': 0.06207482993197279, 'fraction_numerical': 0.005668934240362812, 'mean_word_length': 4.619426751592357, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '503046', 'license': 'MIT', 'n_tokens_mistral': 984, 'n_tokens_neox': 923, 'n_words': 414, 'path': 'Waste Management Inc/3. Waste Management Key Aspects.ipynb', 'repo_name': 'Wealthy-Waste/Financial-Analysis', 'size': '3528'}