<jupyter_start><jupyter_text>WenXuan -- Rare Char Replacement Function<jupyter_code>from WenXuan import WenXuan

wenshuan = WenXuan("2018-07-10", "MF")
wenshuan.load_htmls()<jupyter_output>[Info] Stop at loading data/wenshuan_0582.html.
[Info] Total length of the data is 582.<jupyter_text>Replacement<jupyter_code># if the wenshuan_rare_char.json does not exist
import os
if not os.path.exists("{}_rare_char.json".format(wenshuan.bookname)):
    driver_path = '/Users/cicero/Documents/Physics/PYTHON/Scraping/chromedriver'
    wenshuan.extract_rare_chars(driver_path) # this line would take a very long time, be careful before you execute it
    wenshuan.write_rare_chars()<jupyter_output>[Error] invalid literal for int() with base 16: '', no such search result for 厂林石
[Error] invalid literal for int() with base 16: '', no such search result for 厂林石
[Error] invalid literal for int() with base 16: '', no such search result for 厂林石
[Error] invalid literal for int() with base 16: '', no such search result for 厂林石
[Error] invalid literal for int() with base 16: '', no such search result for 厂林石
[Error] invalid literal for int() with base 16: '', no such search result for <jupyter_text>What `extract_rare_chars` do is writing each html into a temp file `temp.html` with sinca [normalization javascript API](http://char.iis.sinica.edu.tw/API/ics.js). And then, open the html with `selenium` and scrape the rare chars identified by the API. Finally, using the sinica normalization search to identify the unicode of the rare chars.Because sinica normalization API would block out the massive amount of search, sometime the `extract_rare_chars` function would appear some errors.<jupyter_code># if the wenshuan_rare_char.json exist
wenshuan.update_rare_chars()<jupyter_output><empty_output><jupyter_text>Yeah, 👆this line update the html source page to recover the UNICODE of the rare characters. Therefore, rare char components in the `self.flat_bodies` would have been converted to UNICODEs.<jupyter_code>wenshuan.flat_rare_chars[-4]
print("\ueda4")
wenshuan.pretty_print(-4)<jupyter_output><empty_output><jupyter_text>See! 鄭玄周禮注曰：壙，謂冢中也。棺或為，非也. Corrected converted.  Extract Again using Updated Rare Chars<jupyter_code># organize the text files 
wenshuan.extract_paths()         # extract the bookmarks
wenshuan.get_author_bag()        # get the bag of author names and comments
wenshuan.extract_meta()          # extract the meta data
wenshuan.passages2tuples()       # get the passsage into (text, comment) tuples
wenshuan.heads2tuples()          # get headers into (head, comment, ...) tuples
wenshuan.extract_commentators()  # append commentators to metadata
wenshuan.extract_sound_glosses() # append all sound glosses in comments into a list and remove them from the self.flat_passages
wenshuan.flat_passages[-4][-4]<jupyter_output><empty_output><jupyter_text>Save to CSV<jupyter_code>import pandas as pd
df = pd.DataFrame({key : values
    for rare_chars in wenshuan.flat_rare_chars
    for key,values in rare_chars.items()
}).T
df.columns = ['UNICODE', 'UTF-8']
df.to_csv("WenXuan_rare_char.csv")
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09854706253948199, 'fraction_numerical': 0.011054958938723942, 'mean_word_length': 5.556935817805383, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '339644', 'license': 'MIT', 'n_tokens_mistral': 1163, 'n_tokens_neox': 1115, 'n_words': 326, 'path': 'Han-Ji/tutorials/WenXuan -- Rare Char Replacement Function.ipynb', 'repo_name': 'jibanCat/DigitalHumanities', 'size': '3166'}