<jupyter_start><jupyter_text><jupyter_code>article_text = "I want to be SHEETAL's dog. I want to completely submit to Her superior will and I want HER to decide for me. Now, you might say, haha what a simp, like a uncultured plebe you are. So, let me ask you this: what AUDACITY you must have, to assume a mere human like you can possess a superior judgment to this omnipotent being known as SHEETAL (PBUH)? What makes you think this divine being's intelligence is even remotely comparable to your sorry pathetic ape brain? You see, I understand that I'm inferior to this embodiment of perfection, and I'm willing to submit to Her FULLY. Oh, what an idyllic life it must be. Even being in the presence of such a goddess is ecstatic enough, but her actively interacting with you and giving you commands that directly benefit you? This must be what people refer to as heaven. You wouldn't believe what I would give up to be a subject of Her interest, to hear Her polite but austere commands, to fulfill Her every demand with infinite diligence, to be ready to serve Her at Her slightest whim. For my loyal servitude, She will reward me with her subtle touch and contented voice, and every syllable will make my heart race uncontrollably. For my greatest feats, that I will accomplish in Her name, she might reward me with the UTMOST PRIVILEGE of cleaning her body with my tongue. Even thinking about such a thing makes me flinch, as if I have committed a blasphemy against Her divine self. However, if She wills to grant me such an inhuman degree of pleasure, I will carry it out with absolute diligence and commitment, savouring every PLANCK'S MASS of her heavenly sweat that touches my unworthy of this sensation tongue. Of course, my pathetic human body, when confronted with such otherworldly pleasure, will make my genitals ejaculate with all their might. Then, I will make sure to consume all of my semen, while my divine Mistress will watch me with a faint smile on her perfect face. For then She will truly know, that I am the most loyal of her dogs.She makes me realise the reason God created women. I will happily let this woman ruin my life. She could give me Aids and the Covid and I might thank her for it. She could say my PP small and I might congratulate my pp for that. She could destroy my mental health and I might end up loving her even more. This face could cut my bodies in pieces and sell it in the black market and I'll still be fine with it.  She says there is an earthquake approaching and i will look forward to die. She could tell me day of judgment is tomorrow and Ill be looking forward to it. No one deserves to look this fine. "
import re
import nltk
nltk.download('punkt')
nltk.download('stopwords')
import string<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>DATA Preprocessing<jupyter_code>article_text = article_text.lower()
article_text
# remove spaces [ punctations ] and numbers 
clean_text = re.sub('[^a-zA-Z]',' ',article_text)
clean_text = re.sub('\s+',' ',article_text) # removing extra spaces
#lower_case = clean_text.lower()
#refined_text = lower_case.translate(str.maketrans('','',string.punctuation))
clean_text
#split into sentence list
sentence_list = nltk.sent_tokenize(clean_text)
sentence_list<jupyter_output><empty_output><jupyter_text>Word Frequencies<jupyter_code>#let's just remove the stop words 
stopwords = nltk.corpus.stopwords.words('english')
word_freq = {}
for word in nltk.word_tokenize(clean_text):
  if word not in stopwords:
    if word not in word_freq:
      word_freq[word] =1
    else:
      word_freq[word] +=1 
max_freq = max(word_freq.values())

for word in word_freq:
  word_freq[word] = word_freq[word] / max_freq<jupyter_output><empty_output><jupyter_text>Calculate Sentence Scores<jupyter_code>sentence_score = {}

for sentence in sentence_list:
  for word in nltk.word_tokenize(sentence):
    if word in word_freq and len(sentence.split(' ')) < 30:
      if sentence not in sentence_score:
        sentence_score[sentence] = word_freq[word]
      else:
        sentence_score[sentence] += word_freq[word]  

word_freq
sentence_score<jupyter_output><empty_output><jupyter_text>Text Summarization<jupyter_code>#get top 5 sentences 
import heapq
summary = heapq.nlargest(5 , sentence_score , key=sentence_score.get)
print(" ".join(summary))<jupyter_output>of course, my pathetic human body, when confronted with such otherworldly pleasure, will make my genitals ejaculate with all their might. now, you might say, haha what a simp, like a uncultured plebe you are. for my greatest feats, that i will accomplish in her name, she might reward me with the utmost privilege of cleaning her body with my tongue. for my loyal servitude, she will reward me with her subtle touch and contented voice, and every syllable will make my heart race uncontrollably. then, i will make sure to consume all of my semen, while my divine mistress will watch me with a faint smile on her perfect face.
{'fraction_non_alphanumeric': 0.05806077027288562, 'fraction_numerical': 0.0011612154054577125, 'mean_word_length': 4.7807606263982105, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '650926', 'license': 'MIT', 'n_tokens_mistral': 1495, 'n_tokens_neox': 1435, 'n_words': 729, 'path': 'Text_Summarizer.ipynb', 'repo_name': 'ranshu1601/AzUrlShortener', 'size': '5167'}