<jupyter_start><jupyter_text><jupyter_code># Run this cell to mount your Google Drive.
from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>While embedding vectors are a part of the model, BPEMB is required to use the subword tokenizer of appropriate granularity and encode words to subword-IDs.<jupyter_code>!pip install bpemb==0.3.3<jupyter_output>Collecting bpemb==0.3.3
  Downloading https://files.pythonhosted.org/packages/f2/6f/9191b85109772636a8f8accb122900c34db26c091d2793218aa94954524c/bpemb-0.3.3-py3-none-any.whl
Requirement already satisfied: gensim in /usr/local/lib/python3.7/dist-packages (from bpemb==0.3.3) (3.6.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from bpemb==0.3.3) (2.23.0)
Collecting sentencepiece
[?25l  Downloading https://files.pythonhosted.org/packages/ac/aa/1437691b0c7c83086ebb79ce2da16e00bef024f24fec2a5161c35476f499/sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2MB)
[K     |████████████████████████████████| 1.2MB 6.0MB/s 
[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from bpemb==0.3.3) (1.19.5)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from bpemb==0.3.3) (4.41.1)
Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.7/dist-packages (f[...]<jupyter_text>To de-serialize the model, we use load_model method from TensorFlow 2:<jupyter_code>import tensorflow as tf

model = tf.keras.models.load_model( '/content/drive/My Drive/combo/model_final/1model_final/')
model.compile(optimizer='Adamax')
from bpemb import BPEmb

params = {  'embedding_size': 100, 'vocabulary_size': 50000 }

bpemb_pl = BPEmb(lang="pl", dim=params['embedding_size'], vs=params['vocabulary_size'])
tokenids = bpemb_pl.encode_ids('Ala ma kota, pięć psów i mieszka na wsi.')
tokenids
sentence_size = len(tokenids)<jupyter_output><empty_output><jupyter_text>We need a method to pad the subword IDs to the max length of 31:<jupyter_code>def pad_input(tokenids):
  return tf.keras.preprocessing.sequence.pad_sequences(tokenids, maxlen=31, padding='post')

pad_input([tokenids])<jupyter_output><empty_output><jupyter_text>The easiest way of using Combo-NER is via TensorFlow's predict API. For each input sentence, the model returns four lists of predictions:*   part-of-speech*   dependency heads*   dependency relation labels*   named entities<jupyter_code>y_pos, y_heads, y_deprels, y_namedents = model.predict( pad_input([tokenids]) )<jupyter_output><empty_output><jupyter_text>For each subword token and each class, the model returns its probability. To obtain the label ID, we have to apply argmax:<jupyter_code>y_pos = tf.argmax(y_pos, -1)<jupyter_output><empty_output><jupyter_text>Now, each subword has it's own class label as an integer:<jupyter_code>y_pos<jupyter_output><empty_output><jupyter_text>To obtain string labels, we need to load sklearn's label encoders, available from http://mozart.ipipan.waw.pl/~axw/Combo-NER/<jupyter_code>import pickle

le_ner = pickle.load(open('/content/drive/My Drive/combo/le_ner.pkl', 'rb'))
le_upostag = pickle.load(open('/content/drive/My Drive/combo/le_upostag.pkl', 'rb'))
le_deprel = pickle.load(open('/content/drive/My Drive/combo/le_deprel.pkl', 'rb'))<jupyter_output><empty_output><jupyter_text>Now let's print POS, remembering to pad to sentence size:<jupyter_code>list(le_upostag.inverse_transform(y_pos[0][:sentence_size-1]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13048368953880765, 'fraction_numerical': 0.05258717660292463, 'mean_word_length': 6.649462365591398, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '475184', 'license': 'Apache-2.0', 'n_tokens_mistral': 1447, 'n_tokens_neox': 1328, 'n_words': 264, 'path': 'Combo_NER_usage.ipynb', 'repo_name': 'alexwz/ComboNER', 'size': '3556'}