<jupyter_start><jupyter_text>Acronym resolution (`acres`) demo This is a demo of the `acres` project.  Configuration  Working directory All `acres` code should run under the root folder.<jupyter_code>import os
cwd = os.getcwd()
if "notebooks" not in os.listdir(cwd):
    #os.chdir("..")
    print("Wrong working directory: " + cwd)
    
os.getcwd()<jupyter_output><empty_output><jupyter_text>Clear output Let's disable gensim warnings (this seems to be a problem with gensim 3.6.0 only).<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Let's also disable `matplotlib` debug messages.<jupyter_code>import logging 
mpl_logger = logging.getLogger('matplotlib') 
mpl_logger.setLevel(logging.WARNING)<jupyter_output><empty_output><jupyter_text>Using word2vec We trained a word2vec model using [gensim](https://radimrehurek.com/gensim/models/word2vec.html) on a cardiology corpus written in German. We obtained best results with a continuous bag-of-words (CBOW) model with a window size of 2.  2D representation We can use PCA to visualize the vector space.<jupyter_code>from acres.preprocess import resource_factory

model = resource_factory.get_nn_model()

words = ["gutem", "gebessertem",
         "CMP", "Cardiomyopathie", "Kardiomyopathie", "Kardiomoypathie",
         #"SR", "Sinusrhythmus",
         "HF", "Herzfrequenz",
         #"GGT", "y_GT",
         "RR", "Blutdruck",
         "EF", "Auswurffraktion",
         "AP", "Angina_pectoris"]

# or 50 random words...
#words = list(model.wv.vocab)[0:50]

X = model[words]
from matplotlib import pyplot

def display_words(data, words):
    f = pyplot.figure()
    pyplot.scatter(data[:, 0], data[:, 1])

    for i, word in enumerate(words):
        pyplot.annotate(word, xy=(data[i, 0], data[i, 1]))
    pyplot.show()
    
    f.savefig("plot.pdf", bbox_inches="tight")
from sklearn.decomposition import PCA

pca = PCA(n_components=2)
data = pca.fit_transform(X)
display_words(data, words)<jupyter_output><empty_output><jupyter_text>If we're feeling fancier, we can also try tSNE:<jupyter_code>from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=0)
data = tsne.fit_transform(X)
display_words(data, words)<jupyter_output><empty_output><jupyter_text>Raw results from word2vec<jupyter_code>from acres.nn import test

def word2vec_raw(acronym):
    return test.find_candidates(acronym)<jupyter_output><empty_output><jupyter_text>The model allows us to get words that occur in a similar context.<jupyter_code>word2vec_raw("gutem")<jupyter_output><empty_output><jupyter_text>It can also be used for spell checking, as typos happen in similar contexts than the correct spelling.<jupyter_code>word2vec_raw("Cardiomyopathie")<jupyter_output><empty_output><jupyter_text>It turns out that acronym expansions also happen in similar contexts than the acronym itself:<jupyter_code>word2vec_raw("SR")
word2vec_raw("HF")
word2vec_raw("GGT")<jupyter_output><empty_output><jupyter_text>As this is a semantic model, note that it also works for non-trivial expansions, like `RR => Blutdruck`.<jupyter_code>word2vec_raw("RR")<jupyter_output><empty_output><jupyter_text>(RR is a common acronym for designating "blood pressure" in German. It stems from the inventors of the cuff-based sphygmomanometer, the italians Riva and Rocci 🙂) It also works for terms borrowed from other languages (e.g. English), such as `EF => Auswurffraktion` (ejection fraction).<jupyter_code>word2vec_raw("EF")<jupyter_output><empty_output><jupyter_text>We also enriched the model with common collocations provided by the `Phraser` module so that it also works for multiwords acronyms:<jupyter_code>word2vec_raw("AP")<jupyter_output><empty_output><jupyter_text>It does not work, however, for acronyms that are never expanded in the collection:<jupyter_code>word2vec_raw("EKG")<jupyter_output><empty_output><jupyter_text>Filtered results For proper acronym expansion, we further filter results using hand-crafted rules.<jupyter_code>from acres.evaluation import evaluation

def word2vec_filtered(acronym):
    return evaluation.cached_resolve(acronym, "", "", evaluation.Strategy.WORD2VEC)
word2vec_filtered("SR")
word2vec_filtered("HF")
word2vec_filtered("AP")<jupyter_output><empty_output><jupyter_text>Unfortunately, the current rules are too restrictive and do not allow semantic expansions...<jupyter_code>word2vec_filtered("RR")
word2vec_filtered("EF")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1086276262852034, 'fraction_numerical': 0.008493518104604381, 'mean_word_length': 6.597623089983022, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '317285', 'license': 'Apache-2.0', 'n_tokens_mistral': 1585, 'n_tokens_neox': 1510, 'n_words': 374, 'path': 'notebooks/demo.ipynb', 'repo_name': 'bst-mug/acres', 'size': '4474'}