<jupyter_start><jupyter_text>Lousy Fairytale Plot Generator | <NAME>*scroll to the end and click run to check the presentation first!*IdeaFor this project, I spent a lot of time browsing through project gutenberg. While I was doing that, I got a an urge to look up what kind of children's books are on the website, inspired by a phone call with my niece. My niece is a very curious child, and I can only imagine that by the time she starts getting interested in stories and fairytales, her curiosity will mean that her parents will run out of made-up stories to tell her. So, I wanted to create a tool that helps them, well kind of...I wanted to capture that confused tone that parents have sometimes when they make up a story as they go. ResearchWhat makes a fairytale? A google search led to Missouri Southern State University's [website](https://libguides.mssu.edu/c.php?g=185298&p=1223898:~:text=The%20basic%20structure%20of%20a,a%20solution%20can%20be%20found.). To summarize, the page explains that a fairy tale's main elements are:*   Characters*   The Moral Lesson*   Obstacles*   Magic* A Happily Ever AfterSo, these elements formed the structure of my output. Elements*(More specific comments are in the code)*CharactersFor the characters, I scraped a few pages from a website called World of Tales. I chose it over Project Gutenberg as I found the HTML easier to work with. The program picks a random fairytale from the ones I have in an array, parses the HTML, finds all paragraphs and cleans them from new line and trailing characters before appending them to a list. Afterwards, to extract the main character's name I used NLTK and a function that counts proper nouns to find the most repeated proper noun which becomes my main character. As you can imagine, this isn't a perfect method since we could get "I" or "King's". I tried to do my best to limit such results, but it isn't perfect. The Moral LessonHere, I scraped a random blog post that lists 101 valuable life lessons. I thought this would add a humorous tone to the work as some of the lessons are completely unrelated to the fairytale world, e.g. "Don’t go into bad debt (debt taken on for consumption)". Then, I parsed and cleaned the text, and sliced the results to remove any paragraph elements that weren't part of the list. For this part I also used Regular Expressions to remove list numbers ("1."). Unfortunately, there's a bit of hardcoded replacements and removals in this part. I found it difficult to figure out the pronouns, such as turning all you's into a neutral they resulting in "theyself"...etc. I left some of the failed RegEx attempts commented if anyone has any suggestions! This part picks a random lesson from the list everytime. ObstacleMy obstacle was basically a villain the main character needs to fight. Which I got from a webpage that lists 10 fairytale villains. The process was similar, I parsed the page and cleaned the strings using replace() and regEx then randomly chose one of the villains listed. MagicUsing an approach almost identical to the one for villains. I scraped a webpage that lists a bunch of superpowers and magical ability, and made a list out of them to pick a random one when the program runs. I relied on random web pages to compliment that "confused" "humorous" tone of scrambling to make up a random story on the spot!The Happily Ever AfterI made sure to end the text generation with a reference to the happily ever after and the fact that the hero beats the villain using the specific superpower. PresentationI made use of the Sleep() function to create a rhythm for the conversation. I embedded my generated text into human sounding text to really make it feel like you're talking to someone. Obviously, some outputs make it very obvious that this is a bot, such as when the main character is called "I", but I think the general feel is quite natural. ImpactI would say the impact is more entertaining and maybe humorous than functional, although it is quite functional sometimes and suggests ideas that could make great fairytales!Final ReflectionWhile working I realized that RegEx really need a lot of practice, and that the more I do it the more it'll make sense...so I'm excited to do that! However,This was a really fun assignment to work on! It was very rewarding to gradually go through my ideas and realize that there's something in our "toolbox" that can help me achieve what I have in mind.<jupyter_code>import requests
from bs4 import BeautifulSoup
import random
import nltk
import re
from time import sleep
nltk.download('averaged_perceptron_tagger')
from nltk.tag import pos_tag
fairytaleLinks = ["Charles_Perrault/Little_Thumb.html#gsc.tab=0","Brothers_Grimm/Margaret_Hunt/The_Story_of_the_Youth_who_Went_Forth_to_Learn_What_Fear_Was.html#gsc.tab=0",
                  "Charles_Perrault/THE_FAIRY.html#gsc.tab=0","Hans_Christian_Andersen/Andersen_fairy_tale_17.html#gsc.tab=0","Brothers_Grimm/Grimm_fairy_stories/Cinderella.html#gsc.tab=0"
                  "Brothers_Grimm/Margaret_Hunt/Little_Snow-white.html#gsc.tab=0",
                  "Hans_Christian_Andersen/Andersen_fairy_tale_17.html#gsc.tab=0","Charles_Perrault/THE_MASTER_CAT,_OR_PUSS_IN_BOOTS.html#gsc.tab=0",
                  "Brothers_Grimm/Grimm_household_tales/The_Sleeping_Beauty.html#gsc.tab=0", "Hans_Christian_Andersen/Andersen_fairy_tale_31.html#gsc.tab=0",
                  "Hans_Christian_Andersen/Andersen_fairy_tale_47.html#gsc.tab=0","Brothers_Grimm/Grimm_fairy_stories/Snow-White_And_Rose-Red.html#gsc.tab=0"
                  "Brothers_Grimm/Margaret_Hunt/Hansel_and_Grethel.html#gsc.tab=0","Brothers_Grimm/RUMPELSTILTSKIN.html#gsc.tab=0"
                  "Brothers_Grimm/THE%20ELVES%20AND%20THE%20SHOEMAKER.html#gsc.tab=0","Brothers_Grimm/THE%20JUNIPER-TREE.html#gsc.tab=0","Brothers_Grimm/THE%20GOLDEN%20GOOSE.html#gsc.tab=0",
                  "Brothers_Grimm/Margaret_Hunt/The_Frog-King,_or_Iron_Henry.html#gsc.tab=0","Brothers_Grimm/Grimm_fairy_stories/Snow-White_And_Rose-Red.html#gsc.tab=0"]
fairytaleIndex = random.randint(0,(len(fairytaleLinks))-1)

fairytaleTargetUrl = "https://www.worldoftales.com/fairy_tales/" + fairytaleLinks[fairytaleIndex]
reqFairytale = requests.get(fairytaleTargetUrl)

moralLessonTargetUrl = "https://daringtolivefully.com/life-lessons"
reqMoralLesson = requests.get(moralLessonTargetUrl)

superpowerTargetUrl = "http://www.superheronation.com/2007/12/30/list-of-superpowers/"
reqSuperpower = requests.get(superpowerTargetUrl)

villainTargetUrl = "https://beat102103.com/life/top-10-fairytale-villains-ranked/"
reqVillain = requests.get(villainTargetUrl)


#FOR CHARACTER
soupFairytale = BeautifulSoup(reqFairytale.content, 'html.parser')
story = soupFairytale.find_all("p")
paragraphs = []
for i in story:
  content = str(i.text)
  content = content.replace("\r"," ")
  content = content.replace("\n","")
  paragraphs.append(content)
  

#print(paragraphs)

story = "  "
story = story.join(paragraphs)
#print(story)
#Code from StackOverflow: https://stackoverflow.com/questions/17669952/finding-proper-nouns-using-nltk-wordnet
tagged_sent = pos_tag(story.split())
propernouns = [word for word,pos in tagged_sent if pos == 'NNP']

#print(propernouns)
highestCount = 0
#A loop that looks for the most repeated proper noun
for i in propernouns:
  currentCount = propernouns.count(i)
  if currentCount > highestCount:
    highestCount = currentCount
  

countDictionary = {i:propernouns.count(i) for i in propernouns}



#Had an issue with this at first as there's no direct function to get the key using the value, so I researched solutions and used this page:
# https://www.geeksforgeeks.org/python-get-key-from-value-in-dictionary/
def get_key(val):
    for key, value in countDictionary.items():
         if val == value:
             return key

        
  
characterName = get_key(highestCount)
#to eliminate instances like "king's"
characterName = characterName.replace("'", "")
characterName = characterName.replace('"', "")
#print(characterName)




#FOR MORAL LESSON
soupLesson = BeautifulSoup(reqMoralLesson.content, 'html.parser')
lessonsHTML = soupLesson.find_all("p")
lessons = []
for i in lessonsHTML:
  content = str(i.text)
  content = content.replace("\r"," ")
  content = content.replace("\n","")
  lessons.append(content)


#used .index() to figure out the slicing here:   
del lessons[0]
del lessons[101:len(lessons)+1]

# to make things more convenient :)
toRemove = ['2. Don’t postpone joy.',
'8. Pay yourself first: save 10% of what you earn.',
'10. Don’t go into bad debt (debt taken on for consumption).',
'15. Remember <NAME>’s admonishment: “Whether you think you can or whether you think you can’t, you’re right.”',
'19. Don’t smoke. Don’t abuse alcohol. Don’t do drugs.',
'36. Don’t take action when you’re angry. Take a moment to calm down and ask yourself if what you’re thinking of doing is really in your best interest.',
'38. Worry is a waste of time; it’s a misuse of your imagination.',
'44. Don’t gossip. Remember the following quote by Eleanor Roosevelt: “Great minds discuss ideas, average minds discuss events, small minds discuss people.”',
'52. Don’t procrastinate; procrastination is the thief of time.',
 '61. Don’t take yourself too seriously.',
'62. During difficult times remember this: “And this too shall pass.”',
'63. When things go wrong remember that few things are as bad as they first seem.',
'64. Keep in mind that mistakes are stepping stones to success. Success and failure are a team; success is the hero and failure is the sidekick. Don’t be afraid to fail.',
'70. If you don’t know the answer, say so; then go and find the answer.',
'77. Don’t renege on your promises, whether to others or to yourself.',
'80. Don’t worry about what other people think.',
'89. Every time you fall simply get back up again.',
'95. Don’t argue for your limitations.',
 '97. Listen to Eleanor Roosevelt’s advice: “No one can make you feel inferior without your consent.”',
'99. Remember the motto: “You catch more flies with honey.”']

for i in toRemove:
  lessons.remove(i)

#print(lessons)
x = 0
strippedLessons = []

for i in lessons:
  lessons[x] = lessons[x].strip()
  #remove any digits + the period after the digits 
  lessons[x] = re.sub("\d+""[.]", "", i)

  #lessons[x] = re.sub("n'^Dot", "not to", lessons[x]) // attempt to turn all don't(s) at the beginning of the sentence to "not to"

  #remove periods ONLY at the end
  lessons[x] = re.sub("\.$", " ", lessons[x])
  
  lessons[x] = lessons[x].replace("theirs","others'").replace("your","their").replace("you","they").replace("theyself","themselves").lower()
  strippedLessons.append(lessons[x])

  x+=1



#specifics = {"you've":"they've","theirself":"themself","theirs":"others'"} // #an attempt to fix any awkward results due to the replace function above

#y = 0
#for word in strippedLessons[y]:
 #   if word.lower() in specifics:
  #      srippedLessons = text.replace(word, specifics[word.lower()])
   #     y+=1


randomLessonIndex = random.randint(0, len(strippedLessons)-1)
chosenMoralLesson = strippedLessons[randomLessonIndex]
#print(chosenMoralLesson)


#FOR SUPERPOWER
soupSuperpower = BeautifulSoup(reqSuperpower.content, 'html.parser')
superpowers = soupSuperpower.find_all("li")
allSuperpowers = []
for i in superpowers:
  content = (str(i.text)).strip()
  content = content.replace("\r"," ")
  content = content.replace("\n"," ")
  content = content.replace("\t","")
  allSuperpowers.append(content)

#allSuperpowers.index('Superstrength')
#removing all non-Superpower elements
del allSuperpowers[67:len(allSuperpowers)+1]
del allSuperpowers[0:5]
toRemove2 = ['Skills and/or knowledge  Popular categories: science, mechanical, computer/electronics, weapons-handling/military, driving, occult/magical.',
             'Popular categories: science, mechanical, computer/electronics, weapons-handling/military, driving, occult/magical.',
             'Resourcefulness (“I’m never more than a carton of baking soda away from a doomsday device”)']

for i in toRemove2:
  allSuperpowers.remove(i)

randomSuperpowerIndex = random.randint(0, len(allSuperpowers)-1)
chosenSuperpower = allSuperpowers[randomSuperpowerIndex].lower()
#print(chosenSuperpower)

#FOR VILLAIN:
soupVillain = BeautifulSoup(reqVillain.content, 'html.parser')
villainsHTML = soupVillain.find_all("strong")

villains = []
for i in villainsHTML:
  content = str(i.text)
  content = content.replace("\r"," ")
  content = content.replace("\n","")
  content = content.replace("\xa0"," ")
  villains.append(content)

x = 0
for v in villains:
  villains[x] = re.sub("\d+""[.]", "", v)
  villains[x].strip()
  x+=1

randomVillainIndex = random.randint(0, len(villains)-1)
chosenVillain = villains[randomVillainIndex].lower()
#print(chosenVillain)
print(u"\U0001F5E8"+ " " +"Oh? You're out of bedtime stories to tell?")
sleep(1.5)
print(u"\U0001F5E8"+ " " +"hmmm...")
sleep(2)
print(u"\U0001F5E8"+ " " +"how about you tell a story about")
sleep(1)
print(u"\U0001F5E8"+ " " +".....")
sleep(3)
print(u"\U0001F5E8"+ " " +characterName+"?")
sleep(2)
print(u"\U0001F5E8"+ " " +"yeah...yeah, tell a story about " + characterName +" " +  "and how they learnt to...")
sleep(1.5)
print(u"\U0001F5E8"+ " " +"I don't know...like..")
sleep(3)
print(u"\U0001F5E8"+ " " +"how they learnt to..." + chosenMoralLesson)
sleep(1.5)
print(u"\U0001F5E8"+ " " +"Yes! that sounds good I guess.")
sleep(2)
print(u"\U0001F5E8"+ " " +"and of course it isn't that easy...it isn't all rainbows and sunshine you know?")
sleep(3)
print(u"\U0001F5E8"+ " " +"I don't know... maybe talk about their struggles with...")
sleep(4)
print(u"\U0001F5E8"+ " " +"with" + chosenVillain  + "...yikes")
sleep(4)
print(u"\U0001F5E8"+ " " +"but we need a happily ever after, so maybe say that " + characterName + " was able to defeat" + chosenVillain + " somehow...")
sleep(5)
print(u"\U0001F5E8"+ " " +"like by " + chosenSuperpower + " or something...does that make sense?")
sleep(2)
print(u"\U0001F5E8"+ " " +"I mean even if it doesn't...that's all I can give you tonight")
sleep(3)
print(u"\U0001F5E8"+ " " +"you should practice being imaginative or something...")
sleep(2)
print(u"\U0001F5E8"+ " " +"anyways, it's way past bedtime. Go tell your story!")
print("\n\n\nERROR:chat disconnected")<jupyter_output>🗨 Oh? You're out of bedtime stories to tell?
🗨 hmmm...
🗨 how about you tell a story about
🗨 .....
🗨 Gerda?
🗨 yeah...yeah, tell a story about Gerda and how they learnt to...
🗨 I don't know...like..
🗨 how they learnt to... develop their listening skills 
🗨 Yes! that sounds good I guess.
🗨 and of course it isn't that easy...it isn't all rainbows and sunshine you know?
🗨 I don't know... maybe talk about their struggles with...
🗨 with the evil queen from snow white...yikes
🗨 but we need a happily ever after, so maybe say that Gerda was able to defeat the evil queen from snow white somehow...
🗨 like by supersenses  sight/hearing/smell/taste/touch sensing danger (spider-sense) sensing other types of events (dishonesty, murder, etc.) or something...does that make sense?
🗨 I mean even if it doesn't...that's all I can give you tonight
🗨 you should practice being imaginative or something...
🗨 anyways, it's way past bedtime. Go tell your story!



ERROR:chat disconnected
{'fraction_non_alphanumeric': 0.10929319371727748, 'fraction_numerical': 0.018979057591623036, 'mean_word_length': 4.618014705882353, 'pattern_counts': {'":': 3, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '7251', 'license': 'MIT', 'n_tokens_mistral': 5120, 'n_tokens_neox': 4749, 'n_words': 1742, 'path': 'LousyFairytaleGenerator_Assemblage_Project1_.ipynb', 'repo_name': 'sarahalyahya/SoftwareArt-Text', 'size': '15292'}