<jupyter_start><jupyter_text>I. Installing the Unified Medical Language System (UMLS)<jupyter_code>%load_ext autoreload
%autoreload 2

import sys
sys.path.insert(0,'../../trove')<jupyter_output><empty_output><jupyter_text>A. Download the UMLS Release FilesTrove requires access to the [Unified Medical Language System (UMLS)](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html) which is freely available after signing up for an account with the National Library of Medicine (NLM). Visit the link below and download the latest "UMLS Metathesaurus Files" release [2020AB](https://download.nlm.nih.gov/umls/kss/2020AB/umls-2020AB-metathesaurus.zip). This file is quite large (5.3 GB compressed), so it may take some time to download.  **Please note, "full" release zip files are currently not supported.**Alternatively, if you have an existing API KEY you can use the following script to download the zip file from the command line. See https://documentation.uts.nlm.nih.gov/automating-downloads.html for technical details on NLM authentication.     python download_umls.py \       --apikey <KEY> \       --url https://download.nlm.nih.gov/umls/kss/2020AB/umls-2020AB-metathesaurus.zip  B. InstallationCurrently there are 2 ways to initalize the UMLS:- From the source zip file (e.g., `umls-2020AB-metathesaurus.zip`)- From the Rich Release Format (RRF) files generated by [MetamorphoSys](https://www.ncbi.nlm.nih.gov/books/NBK9683/) - **TBD** An existing database instanceDepending on your machine, this should take 2-5 minutes.   Option 1: Install from NLM Zip File<jupyter_code>%%time
from trove.labelers.umls import UMLS

# setup defaults
UMLS.config(
    cache_root = "~/.trove/umls2020AB",
    backend = 'pandas'
)
 
NLM_ZIPFILE_PATH = "umls-2020AB-metathesaurus.zip"
if not UMLS.is_initalized():
    print("Initializing the UMLS from zip file...")
    UMLS.init_from_nlm_zip(NLM_ZIPFILE_PATH)<jupyter_output>Initializing the UMLS from zip file...
CPU times: user 2min 1s, sys: 1min, total: 3min 2s
Wall time: 3min 47s<jupyter_text>Option 2: Install from RRF FilesIf you have installed the UMLS before using [MetamorphoSys](https://www.ncbi.nlm.nih.gov/books/NBK9683/) to create custom vocabulary subsets you can directly use the generated RRF files.<jupyter_code>%%time

RRF_FILE_PATH = ""
if not UMLS.is_initalized():
    print("Initializing the UMLS from RRFs...")
    UMLS.init_from_rrfs(RRF_FILE_PATH)<jupyter_output>CPU times: user 131 µs, sys: 311 µs, total: 442 µs
Wall time: 384 µs<jupyter_text>Option 3: Install from an Existing Database Instance**TBD**: If you have a live UMLS database instance running, you can initialize Trove as follows.<jupyter_code># if not UMLS.is_initalized():
#     UMLS.init_from_dbconn(engine='mysql', dbname='UMLS2020AB')<jupyter_output><empty_output><jupyter_text>3. Test the InstallationHere we apply some common term transformations. This should run in 2-4 minutes.<jupyter_code>%%time
from trove.labelers.umls import UMLS
from trove.transforms import SmartLowercase

# english stopwords
stopwords = set(open('data/stopwords.txt','r').read().splitlines())
stopwords = stopwords.union(set([t[0].upper() + t[1:] for t in stopwords]))

# options for filtering terms
config = {
    "type_mapping"  : "TUI",  # TUI = semantic types, CUI = concept ids
    'min_char_len'  : 2,
    'max_tok_len'   : 8,
    'min_dict_size' : 500,
    'stopwords'     : stopwords,
    'transforms'    : [SmartLowercase()],
    'languages'     : {"ENG"},
    'filter_sabs'   : {"SNOMEDCT_VET"},
    'filter_rgx'    : r'''^[-+]*[0-9]+([.][0-9]+)*$'''  # filter numbers
}

umls = UMLS(**config)<jupyter_output>CPU times: user 1min 25s, sys: 6.59 s, total: 1min 32s
Wall time: 1min 28s<jupyter_text>Look at semantic type assignments for an example term `acetaminophen` from the Medical Subject Headings (MeSH®) terminology.<jupyter_code>from trove.labelers.umls import SemanticGroups

semgroups = SemanticGroups()
stys = umls.terminologies['MSH']['acetaminophen']
print(stys)
print([semgroups.types[sty] for sty in stys])<jupyter_output>{'T109', 'T121'}
['Organic Chemical', 'Pharmacologic Substance']
{'fraction_non_alphanumeric': 0.11743428984808295, 'fraction_numerical': 0.02556064625030142, 'mean_word_length': 4.959770114942529, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '483960', 'license': 'Apache-2.0', 'n_tokens_mistral': 1525, 'n_tokens_neox': 1392, 'n_words': 401, 'path': 'tutorials/1_Installing_the_UMLS.ipynb', 'repo_name': 'raymondzmc/trove', 'size': '4178'}