<jupyter_start><jupyter_text>A subset of columnsA subset of columns whose values can be potentially used as traning text for embeddings.<jupyter_code>cols = ['LOINC_NUM', 'COMPONENT', 'PROPERTY', 'TIME_ASPCT', 'SYSTEM', 'SCALE_TYP', 'METHOD_TYP', 'CLASS',
       'SHORTNAME', 'LONG_COMMON_NAME', 'RELATEDNAMES2']
df_loinc[cols].sample(5)<jupyter_output><empty_output><jupyter_text>The final set of columns for embeddingsThe goal is collect short sentences that can be used to generate the embeddings.The following columns contain values that can be used directly as sentences for generating embeddings:     ['COMPONENT', 'SHORTNAME', 'LONG_COMMON_NAME', 'SYSTEM', 'CLASS']The column `RELATEDNAMES` must split by ";" into multiple sentences.The final out put will be a data frame of 3 columns: `['LOINC_NUM', 'SENTENCE', 'ORG_COLUMN']`* LOINC_NUM -- The LOINC number is the label* SENTENCE -- The sentences that will be used to generate the embeddings* ORG_COLUMN -- The original column containing the sentence<jupyter_code>df_component = df_loinc[['LOINC_NUM', 'COMPONENT']]
df_short_name = df_loinc[['LOINC_NUM', 'SHORTNAME']]
df_long_name = df_loinc[['LOINC_NUM', 'LONG_COMMON_NAME']]
df_system = df_loinc[['LOINC_NUM', 'SYSTEM']]
df_class = df_loinc[['LOINC_NUM', 'CLASS']]
df_related = df_loinc[['LOINC_NUM', 'RELATEDNAMES2']]
# Split the related names by ';' and save them into separate rows
rows = []
for index, row in df_related.iterrows():
    loinc_num = row['LOINC_NUM']
    names = row['RELATEDNAMES2']
    for name in names.split(';'):
        name = name.strip()
        new_row = {
            'LOINC_NUM': loinc_num,
            'RELATED_NAME': name
        }
        rows.append(new_row)
df_related = DataFrame(rows)
df_related.shape
df_related.head(5)
# Make sure the data frame is correctly generated from the rows
rows[:10]<jupyter_output><empty_output><jupyter_text>Remove duplicates<jupyter_code>df_component = df_component.drop_duplicates()
df_short_name = df_short_name.drop_duplicates()
df_long_name = df_long_name.drop_duplicates()
df_system = df_system.drop_duplicates()
df_class = df_class.drop_duplicates()
df_related = df_related.drop_duplicates()<jupyter_output><empty_output><jupyter_text>Save the results<jupyter_code>df_component.to_csv('data/loinc-component.csv', index=False)
df_short_name.to_csv('data/loinc-short-name.csv', index=False)
df_long_name.to_csv('data/loinc-long-name.csv', index=False)
df_system.to_csv('data/loinc-system.csv', index=False)
df_class.to_csv('data/loinc-class.csv', index=False)
df_related.to_csv('data/loinc-related-name.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12338648443432043, 'fraction_numerical': 0.0030372057706909645, 'mean_word_length': 6.258953168044077, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '562006', 'license': 'Apache-2.0', 'n_tokens_mistral': 930, 'n_tokens_neox': 904, 'n_words': 206, 'path': 'loinc-0-extract-columns.ipynb', 'repo_name': 'neurite/med-codes', 'size': '2634'}