<jupyter_start><jupyter_text>This notebook plots the top UMLS semantic types and groups that are represented in search queries from users on a web site. It takes the output from the process that queries NLM's Metamap (https://metamap.nlm.nih.gov/) which maps search queries from end-users to UMLS Semantic Terms. It also uses a lookup table to map the semantic terms to semantic groups.*Coded by: VCid, NLM Codeathon, October 2019*  Reading the input Metamap data<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>The input file is the output from the process that queries MetaMap with search terms. It should look like:```text_000N_50478|MMI|124.36|Medical Subject Headings|C1135584|[inpr]|["MeSH"-tx-1-"mesh"-noun-0,"MeSH Headings"-tx-1-"mesh headings"-noun-0]|TX|[523/4],[826/4],[1008/4],[1503/4],[1978/4];653/4,751/8|L01.453.245.945.700.500text_000N_50478|MMI|100.68|GDF15 protein, human|C1431343|[aapp,bacs]|["PDF"-tx-1-"pdf"-noun-0]|TX|[29/3],[150/3],[590/3],[1087/3],[1356/3],[1560/3],[1934/3]|x.x.x.xtext_000N_50478|MMI|79.67|Penicillin G|C0030827|[antb,orch]|["penicillin"-tx-1-"penicillin"-noun-0]|TX|[461/10],[570/10],[772/10],[1309/10]|D02.065.589.099.750.750;D02.886.108.750.750;D03.633.100.300.750.750text_000N_50478|MMI|69.61|Yellow color|C0221205|[qlco]|["YELLOW"-tx-1-"yellow"-adj-0]|TX|[12/6],[37/6],[547/6],[624/6],[877/6],[1169/6],[1251/6],[1329/6],[1449/6],[1543/6],[1733/6],[1917/6]|text_000N_50478|MMI|58.09|Homo sapiens|C0086418|[humn]|["Human"-tx-1-"human"-noun-0]|TX|[415/5],[859/5]|B01.050.150.900.649.313.988.400.112.400.400text_000N_50478|MMI|51.41|Hippocratic Oath|C0019565|[rnlw]|["Hippocratic Oath"-tx-1-"hippocratic oath"-noun-0]|TX|[61/16],[390/16]|K01.752.566.479.068.400;K01.752.566.479.171.132.750.551;N05.350.213.400;N05.350.340.162.500.500```<jupyter_code>input_file = 'data/input_data.txt' # Change this filename to something else if needed<jupyter_output><empty_output><jupyter_text>The output files are CSV files that contain the number of times Semantic types and semantic groups were seen in the input data.<jupyter_code>PToutput_file = 'data/preferred_term_counts_output.txt'  # preferred termscounts
SToutput_file = 'data/semantic_type_counts_output.txt'   # Semantic types counts
SGoutput_file = 'data/semantic_group_counts_output.txt'  # Semantic groups counts<jupyter_output><empty_output><jupyter_text>Charts are saved in PNG format for future use<jupyter_code>PTchart_file = 'data/UMLS_PrefTerm_Counts.png'  # bar chart of UMLS Preferred term counts
STchart_abbr_file = 'data/UMLS_SemType_abbr_Counts.png'  # Bar chart with all Semantic Types counts (ST abbreviations)
STchart_TopN_abbr_file = 'data/UMLS_SemType_TopN_abbr_Counts.png'  # Bar chart with Top 'N' Semantic Types counts (ST abbreviations)
STchart_file = 'data/UMLS_SemType_Counts.png'  # Bar chart with all Semantic Types counts (full ST names)
STchart_TopN_file = 'data/UMLS_SemType_TopN_Counts.png'  # Bar chart with Top 'N' Semantic Types counts (full ST names)
SGchart_file = 'data/UMLS_SemGroup_counts.png'  # Bar chart of all semantig group counts<jupyter_output><empty_output><jupyter_text>The following are lookup tables used to translate UMLS Semantic Terms into readable text and for mapping them to UMLS Semantic groups<jupyter_code>rec_count = 0  # number of records read
terms_count = 0  # number of UMLS terms found in the records
# retrieving the query information (future use), UMLS Semantic Type, UMLS Preferred Term
data_df = pd.DataFrame(columns=['query', 'UMLS_ST', 'UMLS_PT'])
with open(input_file) as fp:
    for line in fp:
        rec_count += 1
        elems = line.rstrip().split('|')
        query = elems[6]  # query data
        pt = elems[3]  # UMLS preferred term
        ST_string = elems[5]  # list of UMLS semantic types
        ST_string = ST_string.replace("[", "")
        ST_string = ST_string.replace("]", "")
        ST_list = ST_string.split(",")
        for elem in ST_list:
            terms_count += 1
            data_df.loc[len(data_df)] = [query, elem, pt]
print('Records read: {}'.format(rec_count))
print('Terms found: {}'.format(terms_count))

data_df.head()<jupyter_output><empty_output><jupyter_text>Notice that each dataframe record has the query used as input to Metamap (not used in this code) and the abreviation of the UMLS Semantic Type that Metamap associated to that query. There is one record per UMLS semantic query--if Metamap associated more than one UMLS ST to the same query, then there will be multiple records for the same query.  Counting frequency of UMLS Preferred Terms<jupyter_code>PT_sum = data_df.groupby('UMLS_PT').count()
PT_sum.drop(["query"], axis = 1, inplace = True)
PT_sum.rename(columns={"UMLS_ST": "count"}, inplace=True)
PT_sum = PT_sum.sort_values('count', ascending=False)
PT_sum<jupyter_output><empty_output><jupyter_text>Plotting UMLS Prefered Term counts<jupyter_code>plt.style.use('ggplot')
fig, ax = plt.subplots(figsize=(7,12))
PT_sum.plot(kind='barh', legend = False, ax=ax)
ax.set_xlabel('Count')
ax.set_ylabel('UMLS Preferred Term')
ax.invert_yaxis()
ax.set(title='UMLS Preferred Terms represented in search queries\n');
PT_sum.rename(columns={"query": "count"}, inplace=True)<jupyter_output><empty_output><jupyter_text>saving chart to PNG file<jupyter_code>fig.savefig(PTchart_file, bbox_inches='tight')
print('Chart with UMLS Preferred Terms counts was saved in "{}"'.format(PTchart_file))<jupyter_output>Chart with UMLS Preferred Terms counts was saved in "data/UMLS_PrefTerm_Counts.png"<jupyter_text>Saving Preferred Terms data<jupyter_code>PT_sum.to_csv(PToutput_file)
print('Preferred Term frequency counts written to file "{}"'.format(PToutput_file))<jupyter_output>Preferred Term frequency counts written to file "data/preferred_term_counts_output.txt"<jupyter_text>The output format is:```UMLS_PT,count, , ...```  Counting UMLS Semantic Type frequencies<jupyter_code>summary = data_df.groupby('UMLS_ST').count()<jupyter_output><empty_output><jupyter_text>Renaming the query column to what it has become: counts<jupyter_code>summary.rename(columns={"query": "count"}, inplace=True)<jupyter_output><empty_output><jupyter_text>Sorting by counts in descending order<jupyter_code>sum_sorted = summary.sort_values('count', ascending=False)<jupyter_output><empty_output><jupyter_text>Now the dataframe has the number of times each abreviation was seen in the data<jupyter_code>sum_sorted.iloc[:10]<jupyter_output><empty_output><jupyter_text>Plotting UMLS Semantic Type counts<jupyter_code>plt.style.use('ggplot')
fig, ax = plt.subplots(figsize=(7,12))
sum_sorted.plot(y='count', kind='barh', legend = False, ax=ax)
ax.set_xlabel('Count')
ax.set_ylabel('UMLS Semantic Type')
ax.invert_yaxis()
ax.set(title='UMLS Semantic Types abbreviations represented in search queries\n');<jupyter_output><empty_output><jupyter_text>Saving chart in a PNG file<jupyter_code>fig.savefig(STchart_abbr_file, bbox_inches='tight')
print('Chart with UMLS Semantic Type abbreviations counts was saved in "{}"'.format(STchart_abbr_file))<jupyter_output>Chart with UMLS Semantic Type abbreviations counts was saved in "data/UMLS_SemType_abbr_Counts.png"<jupyter_text>Plotting only the top 5<jupyter_code>top_n = 5  # change this number to whaever is convenient

p = sum_sorted.iloc[:top_n].plot(kind='barh', legend = False);
p.invert_yaxis()
p.set(ylabel='UMLS Semantic Type')
p.set(xlabel='Counts')
p.set(title='Top {} UMLS Semantic Types abbreviations represented in search queries\n'.format(top_n));<jupyter_output><empty_output><jupyter_text>Saving chart in a PNG file<jupyter_code>p.figure.savefig(STchart_TopN_abbr_file, bbox_inches='tight')
print('Chart with Top {} UMLS Semantic Type abbreviation counts was saved in "{}"'.format(top_n, STchart_TopN_abbr_file))<jupyter_output>Chart with Top 5 UMLS Semantic Type abbreviation counts was saved in "data/UMLS_SemType_TopN_abbr_Counts.png"<jupyter_text>Mapping UMLS ST Abreviations to UMLS ST Names<jupyter_code>lookup_table = 'data/umls_semantic_types.csv'
mycolumns=['index', 'TUI', 'abbr', 'name']
maptable_df = pd.read_csv(lookup_table, index_col=0, names=mycolumns)
maptable_df.head()
maptable_df.loc['acty', 'name']
for index, row in sum_sorted.iterrows():
    #print(index, row[0])
    sum_sorted.loc[index, 'name'] = maptable_df.loc[index, 'name']
sum_sorted.head()
sum_sorted.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 37 entries, inpr to clas
Data columns (total 3 columns):
count      37 non-null int64
UMLS_PT    37 non-null int64
name       37 non-null object
dtypes: int64(2), object(1)
memory usage: 2.4+ KB<jupyter_text>Saving UMLS Semantic Type frequency tableThe outout will be a simple CSV file that can be used for additional post-processing.<jupyter_code>sum_sorted.to_csv(SToutput_file)
print('Semantic Type frequency counts written to file "{}"'.format(SToutput_file))<jupyter_output>Semantic Type frequency counts written to file "data/semantic_type_counts_output.txt"<jupyter_text>The output format is:```UMLS_ST,count,name, , , , ...```  Plotting UMLS Semantic Types frequencies, in descending order<jupyter_code>fig, ax = plt.subplots(figsize=(7,12))
sum_sorted.plot(kind='barh', x='name', y='count', legend = False, ax=ax)
ax.set_xlabel('Count')
ax.set_ylabel('UMLS Semantic Type')
ax.invert_yaxis()
ax.set(title='UMLS Semantic Types represented in search queries\n');<jupyter_output><empty_output><jupyter_text>Saving charty in a PNG file<jupyter_code>fig.savefig(STchart_file, bbox_inches='tight')
print('Chart with UMLS Semantic Types counts was saved in "{}"'.format(STchart_file))<jupyter_output>Chart with UMLS Semantic Types counts was saved in "data/UMLS_SemType_Counts.png"<jupyter_text>**Note:** a single query can match multiple semantic types, therefore this chart can show more rows than queries. Plotting only the top ones only<jupyter_code>top_n = 5   # change this to the number you need

fig = plt.figure(figsize=(9, 11))
fig = sum_sorted.iloc[:top_n].plot(kind='barh', x='name', y='count', legend = False);
fig.invert_yaxis()
fig.set(ylabel='UMLS Semantic Type')
fig.set(xlabel='Counts')
fig.set(title='Top {} UMLS Semantic Types represented in search queries\n'.format(top_n));
fig.figure.savefig(STchart_TopN_file, bbox_inches='tight')
print('Chart with Top {} UMLS Semantic Type counts was saved in "{}"'.format(top_n, STchart_TopN_file))<jupyter_output>Chart with Top 5 UMLS Semantic Type counts was saved in "data/UMLS_SemType_TopN_Counts.png"<jupyter_text>Mapping UMLS Semantic Types to Groups Reading table that maps UMLS semantic types to semantic groups<jupyter_code>st2grp_mapping_file = 'data/UMLS_type_group_mapping.txt'
st2grp_mapping_df = pd.read_csv(st2grp_mapping_file, sep='\t')
st2grp_mapping_df.head(10)<jupyter_output><empty_output><jupyter_text>Doing the mapping by looking up the semantic types in the table to get their corresponding semantic group<jupyter_code>for index, row in sum_sorted.iterrows():
    sum_sorted.loc[index, 'group'] = st2grp_mapping_df[st2grp_mapping_df['SemanticType'].str.contains(row['name'])].iloc[0]['SemanticGroup']
sum_sorted.head()<jupyter_output><empty_output><jupyter_text>Plotting Semantic Groups counts<jupyter_code>groups = sum_sorted.groupby('group')['count'].sum()
groups = groups.sort_values(ascending=False)
groups
fig, ax = plt.subplots(figsize=(7,12))
groups.plot(kind='barh', x='name', y='count', legend = False, ax=ax)
ax.set_xlabel('Count')
ax.set_ylabel('UMLS Semantic Group')
ax.invert_yaxis()
ax.set(title='UMLS Groups represented in search queries\n');
ax.figure.savefig(SGchart_file, bbox_inches='tight')
print('Semantic Group frequency counts chart written to file "{}"'.format(SGchart_file))<jupyter_output>Semantic Group frequency counts chart written to file "data/UMLS_SemGroup_counts.png"<jupyter_text>Saving UMLS Semantic Group frequency table to fileThe output will be a simple CSV file that can be used for additional post-processing.<jupyter_code>groups.to_frame().to_csv(SGoutput_file)
print('Semantic Group frequency counts written to file "{}"'.format(SGoutput_file))<jupyter_output>Semantic Group frequency counts written to file "data/semantic_group_counts_output.txt"
{'fraction_non_alphanumeric': 0.12536752695197648, 'fraction_numerical': 0.0419797451813133, 'mean_word_length': 6.972005208333333, 'pattern_counts': {'":': 3, '<': 96, '<?xml version=': 0, '>': 96, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '400641', 'license': 'MIT', 'n_tokens_mistral': 4817, 'n_tokens_neox': 4405, 'n_words': 987, 'path': 'webapp/Plotting_ST_Counts.ipynb', 'repo_name': 'NCBI-Codeathons/Digital-and-Customer-Feedback-Analytics-Data-Environment', 'size': '12244'}