<jupyter_start><jupyter_text>TXT and RTF Processing and AnalysisCopyright (C) 2021 ServiceNow, Inc.This notebook handles the special-case processing of .txt and .rtf files in the GEOSCAN dataset. It also contains analysis of these files.<jupyter_code>import pandas as pd
fname = 'rtf_filelist_assessment_121216.xlsx'

rtf_list = pd.read_excel(fname, engine='openpyxl')
rtf_list.head()
rtf_list.Useful.value_counts()
fname = 'txt_filelist_assessment_121216.xlsx'

txt_list = pd.read_excel(fname, engine='openpyxl')
txt_list.head()
txt_list.Useful.value_counts()<jupyter_output><empty_output><jupyter_text>Document content analysis  Reading the rtfs<jupyter_code>from striprtf.striprtf import rtf_to_text

texts = []
for irow, row in rtf_list[rtf_list.Useful.str.strip() == 'Yes'].iterrows():
    with open(f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/wp_rtf/{row.Filename}", 'r') as f:
        data = f.read()
        
    text2 = rtf_to_text(data)
    texts.append(text2)

        
texts<jupyter_output><empty_output><jupyter_text>**analysis** There are few RTFs. These all use \n as the newline character. One contains a \x00 character.<jupyter_code>rtf_list_useful = rtf_list[rtf_list.Useful.str.strip() == 'Yes']
rtf_list_useful['text'] = texts
rtf_list_useful.shape
texts[7].split('\n')<jupyter_output><empty_output><jupyter_text>Reading the txts<jupyter_code>texts_txts = []
for irow, row in txt_list[txt_list.Useful.str.strip() == 'Yes'].iterrows():
    #text = []
    import codecs
    with codecs.open(f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/txt/{row.Filename}", encoding='latin-1') as f:
        data = f.read()

    texts_txts.append(data)
    #print(data)
texts_txts
text_list_useful = txt_list[txt_list.Useful.str.strip() == 'Yes']
text_list_useful['text'] = texts_txts
for x in texts_txts[0].split('\n'):
    print('-' + x)
for x in texts_txts[2].split('\n'):
    print('-' + x)<jupyter_output>-                  SCOTIAN SHELF MESOZOIC MAPPING PROJECT
-<NAME>
-                        ATLANTIC GEOSCIENCE CENTRE
-
-Introduction:
-
-The Geological Survey of Canada at the Atlantic Geoscience Centre, Dartmouth, Nova Scotia undertook a 5 year project in 1984 to re-interpret the Mesozoic geology of Scotian Basin.  Since publication of a Scotian Basin synthesis almost ten years earlier by the G.S.C. (Jansa and Wade, 1975), a vast quantity of well and seismic information had been gathered by the petroleum industry and much of these data were available through the Halifax regional office of the Canada Oil and Gas Lands Administration (now the Canada-Nova Scotia Offshore Petroleum Board) and the Canada-Newfoundland Offshore Petroleum Board.
-
-The results of this study have been published in Volume 2 of the Geology of Canada Series, entitled: 'Geology of the Continental Margin of Eastern Canada' (Wade and MacLean, 1990).  They will also be presented in the Scotian Shelf an[...]<jupyter_text>**analysis** These documents contain \r\n as newline characters. Depending on the document, some have newlines according to a document's lines (linebreaks mid sentences), while some have newlines only at the end of paragraphs.  Statistical analysis<jupyter_code>import matplotlib.pyplot as plt
text_list_useful.text.str.len().plot.hist(bins=20)
plt.title('Length of texts in characters')
text_list_useful.text.str.split().str.len().plot.hist(bins=20)
plt.title('Length of texts in words (simple space tokenization)')
print(f'Total number of words {text_list_useful.text.str.split().str.len().sum()}')
text_list_useful.text.str.split('\n').str.len().plot.hist(bins=20)
plt.title('Length of texts in lines')
import re
text_list_useful['text_split'] = text_list_useful.text.str.split('\r\n') # + '\n' #apply(lambda x: re.split(r'(\r\n)', x))#str.split(r'(\r\n)')
text_list_useful_split = text_list_useful.explode('text_split') + '\n'# = text_list_useful['text_split'] + '\n'
text_list_useful_split['file'] = f'/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/txt/' + text_list_useful_split['Filename']
text_list_useful_split['text'] = text_list_useful_split['text_split']
#text_list_useful_split[['file', '']]

text_list_useful_split[['file', 'text']]<jupyter_output><empty_output><jupyter_text>Write txt to file<jupyter_code>import codecs
import numpy as np
import tqdm

df = txt_list[txt_list.Useful.str.strip() == 'Yes']
for irow, row in tqdm.tqdm(df.iterrows(), total=df.shape[0]):
    #text = []
    
    newfilename = pathlib.Path(row.Filename).stem + '.csv'
    with codecs.open(f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/txt/{row.Filename}", encoding='latin-1') as f:
        data = f.read()
        
    data = data.split('\r\n')
    newdata = pd.DataFrame({'text': data})
    newdata['text'] = newdata['text'] + "\n"                           
    newdata['file'] = f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/txt/{row.Filename}"
    newdata['obj_type'] = np.nan
    newdata['pg'] = np.nan
    newdata['pos_x0'] = np.nan
    newdata['pos_y0'] = np.nan
    newdata['pos_x1'] = np.nan
    newdata['pos_y1'] = np.nan
    #display(newdata)
    
    outputfile = f"/nrcan_p2/data/02_intermediate/20201117/geoscan/txt/{newfilename}"
    newdata = newdata[['obj_type', 'pg', 'pos_x0', 'pos_y0', 'pos_x1', 'pos_y1', 'text', 'file']]
    newdata.to_csv(outputfile)
    #break
    

    #texts_txts.append(data)<jupyter_output>100%|██████████| 35/35 [00:00<00:00, 86.92it/s]<jupyter_text>Write rtf to file<jupyter_code>import numpy as np
import tqdm
import pathlib

df = rtf_list[rtf_list.Useful.str.strip() == 'Yes']
for irow, row in tqdm.tqdm(df.iterrows(), total=df.shape[0]):
    #text = []
    
    newfilename = pathlib.Path(row.Filename).stem + '.csv'
    with codecs.open(f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/wp_rtf/{row.Filename}") as f:
        data = f.read()
        
    data = rtf_to_text(data)
    data = data.split('\n')
    newdata = pd.DataFrame({'text': data})
    newdata['text'] = newdata['text'] + "\n"                           
    newdata['file'] = f"/nrcan_p2/data/01_raw/20201117/geoscan/raw/extracted/wp_rtf/{row.Filename}"
    newdata['obj_type'] = np.nan
    newdata['pg'] = np.nan
    newdata['pos_x0'] = np.nan
    newdata['pos_y0'] = np.nan
    newdata['pos_x1'] = np.nan
    newdata['pos_y1'] = np.nan
    #display(newdata)
    
    outputfile = f"/nrcan_p2/data/02_intermediate/20201117/geoscan/wp_rtf/{newfilename}"
    newdata = newdata[['obj_type', 'pg', 'pos_x0', 'pos_y0', 'pos_x1', 'pos_y1', 'text', 'file']]
    newdata.to_csv(outputfile)
    #break<jupyter_output>100%|██████████| 9/9 [00:00<00:00, 106.59it/s]
{'fraction_non_alphanumeric': 0.14623493975903615, 'fraction_numerical': 0.029969879518072288, 'mean_word_length': 4.347020933977456, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '566748', 'license': 'MIT', 'n_tokens_mistral': 2635, 'n_tokens_neox': 2468, 'n_words': 504, 'path': 'project_tools/notebooks/Preprocessing_and_Dataset_Analysis/Txt and RTF Processing.ipynb', 'repo_name': 'feedzor/Geoscience_Language_Models', 'size': '6646'}