<jupyter_start><jupyter_text>Import & examine data<jupyter_code>import pandas as pd
import os
from pathlib import Path

cwd = Path(os.getcwd())
cmts_csv = cwd / 'Data' / 'Comments_20210607_234843.csv'
posts_csv = cwd / 'Data' / 'Posts_20210607_234843.csv'

cmts_df = pd.read_csv(cmts_csv,sep=',',index_col=0)
cmts_df.head(15)<jupyter_output><empty_output><jupyter_text>Exploring comments...Let's print the first 10 comments in full to see how it looks<jupyter_code>for i in cmts_df['body'][0:9]:
    print(i+"\n")<jupyter_output>everyone! Here's a conpendium with seven powerful, fearsome dragons for your high-level campaign! This is an attempt at creating monsters powerful enough to serve as effective solo bosses, given how common it is for this type of encounters to fall under the power of action economy jaja
  
  ey're probably a bit overtuned, but they are high CR monsters intended to be thrown at players at a lever where game balance goes out of the window, so I had some fun imagining and creating this thematic dragons with fun abilities and give them interesting mechanics to interact with your players.
  
   a bonus, there's some lore and history about the campaign where they'll inaugurate, as well as their personality and interactions with other dragons.
  
   course, all art credit is at the last page, and my personal thanks to my girlfriend who helped me edit this.
  
  pe you like this conpendium! If you have any opinions , questions or feedback, I'll be eagerly reading them in the comments below!
[...]<jupyter_text>Extracting the GMBinder and/or Homebrewery linksWe can see we have some different varieties of potentially useful links, including /share/ and /edit/ links. There are also some from unrelated domains (Google Docs, Wikipedia articles, etc.) and some direct links to PDF files that lack the HTML source we want to collect.We also have three varieties of link text: Raw linksURLs with no special formating around them.> Gmbinder Link: https://www.gmbinder.com/share/-MW5xKc1QPj4TXEgBzQm Markdown linksThese take the form of a display text portion in brackets immediately followed by a URL in parenthesis.> [Display text](url)Moreover, in some cases, the display text is also a url and in at least one case early on the url is incorrect in the display text portion, but correct in the actual url segment of the link code.> [https://homebrewery.naturalcrit.com/share/MhTK9\_7zHuB6](https://homebrewery.naturalcrit.com/share/MhTK9_7zHuB6)Since, the url portion is more likely to be correct in most cases, we'll go exclusively with raw links and the (url) portion of markdown code.To do this we can use a set of regular expressions. After a few trial and errors, I arrived at the following: Homebrewery> ?:https?:\/\/)?(?:www\.)?homebrewery\.naturalcrit\.com\/[\w\-][\w\-]*\/?[\w\-]*\/?[\w\-]* GMBinder> (?:https?:\/\/)?(?:www\.)?gmbinder\.com\/[\w\-][\w\-]*\/?[\w\-]*\/?[\w\-]*[\s\)]<jupyter_code>cmts_df['url'] = cmts_df['body'].str.findall(r'[\s\(](?:https?:\/\/)?(?:www\.)?homebrewery\.naturalcrit\.com\/[\w\-][\w\-]*\/?[\w\-]*\/?[\w\-]*[\s\\)]|[\s\(](?:https?:\/\/)?(?:www\.)?gmbinder\.com\/[\w\-][\w\-]*\/?[\w\-]*\/?[\w\-]*[\s\)]')
for i in cmts_df['url'][0:14]:
    print(i)<jupyter_output>['(https://www.gmbinder.com/share/-Ma-uM6qJVQrOQCqAW_h)']
[]
['(https://homebrewery.naturalcrit.com/share/5pAZ7yalcycb)']
['(https://www.gmbinder.com/share/-MbXDsntDZfWFZTSvI1u)']
['(https://www.gmbinder.com/share/-MY93ONkI97gQojUCamq)']
['(https://homebrewery.naturalcrit.com/share/MuXv0wp7T3VP)']
[' https://homebrewery.naturalcrit.com/share/bwlZ\\']
['(https://homebrewery.naturalcrit.com/share/1akc6sy-yLCYkmdigekOAI7MeXX2vUvmxL2M60vRkIjRE)']
['(https://www.gmbinder.com/share/-Mb0TEktVmDXVBJp1o75)']
['(https://www.gmbinder.com/share/-MaUdYa0vZ3L5rs10J8B)', '(https://www.gmbinder.com/share/-MaFl4cOGqS5DpWWlGK_)']
[]
['(https://www.gmbinder.com/share/-MaGpUOUtN5708lGugMr)']
['(https://www.gmbinder.com/share/-MaiiTks5XL8QJDoreAT)']
['(https://homebrewery.naturalcrit.com/share/1PKVJd1CVvEvSPUJtF3Y0f2lr_G60Sw96EbCjzmP6VhS0)']<jupyter_text>Final cleaning steps...As expected, some entries have multiple urls, while others now lack any entries.Many of our strings also having some hanging portions at the ends from our Regex expression:* '('* ')'* '\n'* ' 'For our final data, we'd like to get everything in a format where each row represets one URL rather than one post and each URL is unique and has been cleaned.Lastly, we'll remove anything with 'profile' or 'pdf' in the url, as these URLs require significantly more processing to handle and would not be worth the effort (especially the PDFs)Luckily, we have a few convient tools at our disposal, so we will takle this in the following manner:1) Expand each entry into multiple rows for each URL in the dataset. To do this we will use Pandas' .explode() function designed for this very purpose.2) Remove rows that are now empty (i.e. posts that only contained direct links to the homebrewery/gmbinder homepage) by dropping NA values3) Each each individual string to remove the extraneous portions left over using the map function.4) Remove duplicates using the drop_duplicates() function in Pandas.5) Lastly, we'll remove any rows that contain the string 'pdf' or 'profile'<jupyter_code>cmts_df = cmts_df.explode('url')
cmts_df = cmts_df.dropna()
cmts_df['url'] = cmts_df['url'].map(lambda x : x.strip('(').strip(')').strip('\n').strip(' '))
cmts_df.drop_duplicates(subset=['url'],keep='first',inplace=True)
cmts_df = cmts_df[cmts_df['url'].str.contains('pdf') == False]
cmts_df = cmts_df[cmts_df['url'].str.contains('profile') == False]
cmts_df = cmts_df[cmts_df['url'].str.contains('edit') == False]
cmts_df.reindex()
cmts_df.head(25)<jupyter_output><empty_output><jupyter_text>Collect Markdown/Source CodeNext, we'll import the post data and collect the mixed html/css/markdown-style source code used by Homebrewery/GMBinder.For this, I've defined two functions in dnd_scraper_tools that I'll be importing.The first function simply checks for a source code button. This seems to be standard on Homebrewery, but can be optionally disabled on GMBinder.In that event, we'll try to get the cleanest text we can from the HTML available on the main display page, so the function will return the original URL if it can't find a button. This is done only for the GMBinder links, as Homebrewery's mcontained directain display page is rendered by React JS and collecting the text via BeautifulSoup is not possible. My initial exploration of the links suggests that this does not occur. If it does, the script will return an value of None that we can later filter out. Note, I performed some manual curation of the post data to add UnearthedArcana style descriptive flair tags to the DnDHomebrew posts, that way this datset is initially complete with respect to that column.<jupyter_code>from dnd_scraper_tools import grab_src_url, collect_text, remove_html

posts_df = pd.read_csv(posts_csv,sep=',',index_col=0)

posts_df = posts_df[posts_df['url'].str.contains('pdf') == False]
posts_df = posts_df[posts_df['url'].str.contains('profile') == False]
posts_df = posts_df[posts_df['url'].str.contains('edit') == False]
posts_df.head(10)
posts_df['src_url'] = posts_df['url'].map(grab_src_url)
posts_df.head(10)
cmts_df['src_url'] = cmts_df['url'].map(grab_src_url)
cmts_df.head(10)
cmts_df['Text'] = cmts_df['src_url'].map(collect_text)
posts_df['Text'] = posts_df['src_url'].map(collect_text)
cmts_df.head(10)
posts_df.head(10)<jupyter_output><empty_output><jupyter_text>Removing residual HTMLA quick scan reveals that some HTML tags in the source text (extra styling elements, etc) did not get removed. Let's strip all of the HTML one more time.<jupyter_code>cmts_df['Text'] = cmts_df['Text'].map(remove_html)
posts_df['Text'] = posts_df['Text'].map(remove_html)
posts_df.head(10)
cmts_df.head(10)
processed_cmts = cwd / 'Data' / 'CommentsProcessed.csv'
processed_posts = cwd / 'Data' / 'PostsProcessed.csv'

posts_df.to_csv(processed_posts,sep=',',encoding='utf-8')
cmts_df.to_csv(processed_cmts,sep=',',encoding='utf-8')
cmts_df.rename(columns={"post_flair":"flair"},inplace=True)
cmts_df.rename(columns={"post_title":"title"},inplace=True)
all_data = pd.concat([posts_df[["src_url","sub","flair","title","Text"]], cmts_df[["src_url","sub","flair","title","Text"]]])
all_data.drop_duplicates(subset=["src_url"], inplace=True)
processed = cwd / 'Data' / 'TestProcessed.csv'

all_data = all_data[["sub","flair","title","Text","src_url"]]

all_data.to_csv(processed, sep=',', index=False,encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13442356761772212, 'fraction_numerical': 0.013801902110879146, 'mean_word_length': 5.557414448669202, 'pattern_counts': {'":': 2, '<': 23, '<?xml version=': 0, '>': 28, 'https://': 16, 'lorem ipsum': 0, 'www.': 13, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '155672', 'license': 'MIT', 'n_tokens_mistral': 2992, 'n_tokens_neox': 2867, 'n_words': 876, 'path': 'ProcessingTestData.ipynb', 'repo_name': 'nkuehnle/COMP_SCI_349_MachineLearning_Final', 'size': '8622'}