<jupyter_start><jupyter_text>Fix FullText CleanupNoticed that I was getting comments as well as main text from Medium.com scraping! Fix to avoid getting comments.<jupyter_code>import os, time, re, pickle
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from datetime import timedelta, date
import urllib
import html5lib
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup, SoupStrainer<jupyter_output><empty_output><jupyter_text>CLEAN UP FULLTEXT note: fixed to avoid getting comments as well as main text<jupyter_code>fhtml = open('/Users/clarencecheng/Dropbox/~Insight/skimr/datasets/fullhtml_20170606_00-54-27_isolate.txt','r')

id_ft = []
fullt = []

# article ID #1005 from 00-54-27 has blockquote highlight! (url: https://themission.co/the-pain-of-being-a-pro-1a41f802614)
# Later, look into using html tags as features?

# num = 0
for line in fhtml:
    text = line.strip().split('\t')
    fullh = text[2]
    fullt_line = []

    soup = BeautifulSoup(fullh,'lxml')                            #,parse_only=content)
    txt0 = soup.find('div',attrs={'data-source':'post_page'})     #class_='postArticle-content')
    if not txt0:
        print('error! skipping '+text[1])
        continue
    txt1 = txt0.find_all(class_='graf')#'p',class_='graf')
    id_ft.append(text[1])

    for line in txt1:
        txt2 = re.sub('<[^>]+>', '', str(line) )
        fullt_line.append(txt2)
#     num+=1
#     if num == 10:
#         break

#     print(fullt_line)
    
    fullt.append( fullt_line )

print(id_ft[1006])
print(fullt[1006])

print(id_ft[953])
print(fullt[953])<jupyter_output>1005
['https://pixabay.com/en/ball-mini-golf-ball-hole-putting-1271974/', 'The Hard Truth About Mastering Any\xa0Skill', 'Like all skills that don’t come naturally, practice makes perfect. The key to building anything right is repetition. Mastery does not require you to be perfect, but it does require you to pursue perfection.', 'You can’t “smartcut” your way to mastery. But you can work your way to the top. Excellence is borne not of any particular innate ability, but of practice. Success in any field is more about commitment to a process than it is about finding one magic trick.', 'Professionals stick to the schedule. Everyone else lets life get in the way. Professionals work towards mastery with purpose. You don’t need a giant leap of faith or big breaks to be a professional at what you do.', '<NAME> explains it perfectly:', 'If you want to be a pro, you’re going to have to break this terrible amateur habit of looking at what people have without paying attention to what they did[...]<jupyter_text>CLEAN UP FULLTEXT for second dataset<jupyter_code>fhtml2 = open('/Users/clarencecheng/Dropbox/~Insight/skimr/datasets/fullhtml_20170606_10-45-58_edit_isolate.txt','r')

id_ft2 = []
fullt2 = []

# num = 0
for line in fhtml2:
    text = line.strip().split('\t')
    fullh = text[2]
    fullt_line = []

    soup = BeautifulSoup(fullh,'lxml')                            #,parse_only=content)
    txt0 = soup.find('div',attrs={'data-source':'post_page'})     #class_='postArticle-content')
    if not txt0:
        print('error! skipping '+text[1])
        continue
    txt1 = txt0.find_all(class_='graf')#'p',class_='graf')
    id_ft2.append(str(int(text[1])+2384))

    for line in txt1:
        txt2 = re.sub('<[^>]+>', '', str(line) )
        fullt_line.append(txt2)

#     print(fullt_line)
    fullt2.append( fullt_line )
#     num+=1
#     if num == 22:
#         break

print(id_ft2[9])
print(fullt2[9])
print(id_ft2)
print(str(len(id_ft2)))
print(str(len(fullt2)))<jupyter_output>error! skipping 20
error! skipping 68
error! skipping 70
error! skipping 121
error! skipping 180
error! skipping 237
error! skipping 257
error! skipping 266
error! skipping 287
error! skipping 298
error! skipping 302
error! skipping 317
error! skipping 333
error! skipping 380
error! skipping 382
error! skipping 400
error! skipping 422
error! skipping 439
error! skipping 441
error! skipping 450
error! skipping 482
error! skipping 523
error! skipping 532
error! skipping 544
error! skipping 556
error! skipping 584
error! skipping 594
error! skipping 615
error! skipping 638
error! skipping 660
error! skipping 684
error! skipping 705
error! skipping 707
error! skipping 708
error! skipping 731
error! skipping 739
error! skipping 771
error! skipping 776
error! skipping 843
error! skipping 863
error! skipping 873
error! skipping 881
error! skipping 890
error! skipping 904
error! skipping 924
error! skipping 946
error! skipping 949
error! skipping 951
error! skipping 962
error! skipping 970
err[...]<jupyter_text>COMBINE DATASETS 1 AND 2<jupyter_code>id_ft_all = id_ft + id_ft2
fullt_all = fullt + fullt2

print(str(len(id_ft)))
print(str(len(id_ft2)))
print(str(len(id_ft_all)))
print(str(len(fullt)))
print(str(len(fullt2)))
print(str(len(fullt_all)))

keys_fullt = id_ft_all
vals_fullt = fullt_all
dict_fullt = dict(zip(keys_fullt,vals_fullt))<jupyter_output>2276
2065
4341
2276
2065
4341<jupyter_text>Use prior 'uniqified' ids to remove duplicates from new cleaned data    (data_pd initially saved from http://localhost:8888/notebooks/skimr/jupyter/2_Data_Exploration.ipynb)<jupyter_code># retrieve with pickle
data_temp = pickle.load(open('/Users/clarencecheng/Dropbox/~Insight/skimr/data_pd','rb'))

print(data_temp.head())

ids = data_temp['ids']
intersect = set(ids) & set(id_ft_all)
# print(intersect)

# print(len(ids))
# print(len(id_ft_all))
# print(len(intersect))

interx = list(map(int,intersect))
interx.sort()
interx = list(map(str,interx))
# print(interx)

sorted(interx) == sorted(ids)

keys_f = []
vals_f = []
for i in interx:
    keys_f.append(i)
    vals_f.append(dict_fullt[i])

keys_h = data_temp['ids']
vals_h = data_temp['highlights']

print(sorted(keys_h) == sorted(keys_f))

dict_h = dict(zip(keys_h,vals_h))
dict_f = dict(zip(keys_f,vals_f))

# print(len(keys_h))
# print(len(vals_h))
# print(len(keys_f))
# print(len(vals_f))

vals_all = zip(vals_h, vals_f)
dict_all_new = dict(zip(keys_h, vals_all))
print(dict_all_new['2'])

data_new = pd.DataFrame({'ids':keys_h, 'highlights':vals_h, 'text':vals_f})<jupyter_output>highlights ids  \
0  Until you appreciate what you currently have, ...   2   
1  And let’s not feel terrified, but full of rene...   4   
2  Hard work is doing the work other people don’t...   5   
3  He’d made up his mind to skip the dog thing al...  10   
4  Realizing that our actions, feelings and behav...  12   

                                                text  
0  [When you have five minutes of down-time, how ...  
1  [The purpose of terrorism is to spread terror....  
2  [Designers, programmers, tech entrepreneurs, a...  
3  [REPORTING FROM HERTFORDSHIRE, ENGLAND, 6/3/17...  
4  [I spent several years working through Cogniti...  
True
('Until you appreciate what you currently have, more won’t make your life better.', ['https://unsplash.com/?photo=pFqrYbhIAXs', '35 Things You Need to Give Up to Be Successful', '1. Wasting Five\xa0Minutes', 'When you have five minutes of down-time, how do you spend that time? Most people use it as an e[...]<jupyter_text>Save dataframe and dict with pickle<jupyter_code># save dataframe with pickle
# data = pd.DataFrame({'ids':keys_h, 'highlights':vals_h, 'text':vals_f})
fdata = open('/Users/clarencecheng/Dropbox/~Insight/skimr/data_pd_new','wb')
pickle.dump(data_new,fdata)

# # save dict_all with pickle
fdict_all = open('/Users/clarencecheng/Dropbox/~Insight/skimr/dict_all_new','wb')
pickle.dump(dict_all_new, fdict_all)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11846415935001965, 'fraction_numerical': 0.04298257109160005, 'mean_word_length': 4.389830508474576, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '385095', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2919, 'n_tokens_neox': 2627, 'n_words': 768, 'path': 'jupyter/2b_Fix_FullText_Cleanup.ipynb', 'repo_name': 'cyucheng/skimmr', 'size': '7635'}