<jupyter_start><jupyter_text>Find an estimate for ${\alpha}$ (Term to Abbreviation Ratio)<jupyter_code>import Abbreviation_and_NC_Extraction
import re
import pandas as pd
from string import punctuation<jupyter_output><empty_output><jupyter_text>Load PURE Data from file:<jupyter_code>pure_data = pd.read_csv('pure_data.CSV', names=["dataset", "id", "req_texts"], sep='\t', encoding='utf8')
ids = list(pure_data['id'].values)
reqs = list(pure_data['req_texts'].values)
dataset = list(pure_data['dataset'].values)<jupyter_output><empty_output><jupyter_text>The Main Function: Independent compilation of the set of NCs and the set of abbreviations. Calculation of the ratio of their cardinalities as the final result.<jupyter_code>set_of_detected_ncs = set()
for req in reqs:
    set_of_detected_ncs = set_of_detected_ncs.union(Abbreviation_and_NC_Extraction.nc_detect(req))
for term in list(set_of_detected_ncs)[:20]:
    print(term)
print(len(set_of_detected_ncs))
set_of_detected_abbreviations = set()
for req in reqs:
    extracted_abbreviations = Abbreviation_and_NC_Extraction.abbv_detect(req)
    for abbv in extracted_abbreviations:
        cleaned_abbv = re.sub(r"[\([{})\]]", "", abbv)
        set_of_detected_abbreviations.add(cleaned_abbv.strip(punctuation))
print(len(set_of_detected_abbreviations))<jupyter_output>138<jupyter_text>Final result:<jupyter_code>ratio = len(set_of_detected_ncs)/len(set_of_detected_abbreviations)
print(ratio)<jupyter_output>23.152173913043477
{'fraction_non_alphanumeric': 0.1112609575185435, 'fraction_numerical': 0.015509103169251517, 'mean_word_length': 8.512820512820513, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '487567', 'license': 'MIT', 'n_tokens_mistral': 582, 'n_tokens_neox': 534, 'n_words': 94, 'path': '4) Section_5.4 -- Find an Estimate for Alpha, the Term to Abbreviation Ratio.ipynb', 'repo_name': 'AEPForGTE/ILLOD', 'size': '1483'}