<jupyter_start><jupyter_text>---Author:                 **`<NAME>`**Topic:                  **`"Named Entity Recognition" (NER)`** Main:                   **`Natural Language Processing NLP`**Library:                **`TensorFlow (2.x)`**Programing Language:    **`Python`**Date:                   **`2021-09-20`**---  Named Entity RecognitionIn this series of notebooks we are going to go thought what is called `NER` (Named Entity Recoginition) using tensorflow 2. We are going to use the [conll2003](https://www.clips.uantwerpen.be/conll2003/ner/) the english version in this notebook as our dataset. I've downloaded the data and uploaded it on my google drive so that it can be loaded here on google colab easily. Mounting the google drive<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Imports<jupyter_code>import os, time, io

import tensorflow as tf
import numpy as np
from tensorflow import keras

from collections import Counter<jupyter_output><empty_output><jupyter_text>Path to files.<jupyter_code>root = '/content/drive/My Drive/NLP Data/ner-CoNLL-2003'
os.path.exists(root)<jupyter_output><empty_output><jupyter_text>File structuresWe have three files in the `ner-CoNLL-2003` folder which are:1. train.txt2. valid.txt3. test.txtEaxh file contains data of the following nature:```txt-DOCSTART- -X- -X- OCRICKET NNP B-NP O- : O OLEICESTERSHIRE NNP B-NP B-ORGTAKE NNP I-NP OOVER IN B-PP OAT NNP B-NP OTOP NNP I-NP OAFTER NNP I-NP OINNINGS NNP I-NP OVICTORY NN I-NP O. . O O``` Data preprocessingWe are going to extract the words with their named entities into an array for example:```[  ['EU', 'BB-ORG'], ['TOP', 'O']...]```The following code helps us to create a single function that we can reuse for all our sets.<jupyter_code>def split_text_label(filename):
  f = open(os.path.join(root, filename))
  split_labeled_text = []
  sentence = []
  for line in f:
    if len(line)==0 or line.startswith('-DOCSTART') or line[0]=="\n":
       if len(sentence) > 0:
         split_labeled_text.append(sentence)
         sentence = []
       continue
    splits = line.split(' ')
    sentence.append([splits[0],splits[-1].rstrip("\n")])
  if len(sentence) > 0:
    split_labeled_text.append(sentence)
    sentence = []
  return split_labeled_text
train = split_text_label( "train.txt")
valid = split_text_label( "valid.txt")
test = split_text_label( "test.txt")
valid[:5]<jupyter_output><empty_output><jupyter_text>Building the vocabularyNext we are going to build the vocabulary for all unique words and labels.<jupyter_code>labelSet = set()
wordSet = set()
# words and labels
for data in [train, valid, test]:
  for labeled_text in data:
    for word, label in labeled_text:
      labelSet.add(label)
      wordSet.add(word.lower())<jupyter_output><empty_output><jupyter_text>We are going to create a word index maping. We are going to start with our entity labels.<jupyter_code>label2Idx = {}
for label in sorted(list(labelSet), key=len):
  label2Idx[label] = len(label2Idx)
idx2Label = {v: k for k, v in label2Idx.items()}
len(idx2Label)<jupyter_output><empty_output><jupyter_text>Word index mapping for the words<jupyter_code>word2Idx = {}
word2Idx["<pad>"] = 0 # padding token
word2Idx["<unk>"] = 1 # unknown token 

for word in wordSet:
  word2Idx[word] = len(word2Idx)

idx2Word = {v:k for k, v in word2Idx.items()}
print(len(idx2Word))
def createMatrices(data, word2Idx, label2Idx):
  sentences = []
  labels = []
  for split_labeled_text in data:
     wordIndices = []
     labelIndices = []
     for word, label in split_labeled_text:
       if word in word2Idx:
          wordIdx = word2Idx[word]
       elif word.lower() in word2Idx:
          wordIdx = word2Idx[word.lower()]
       else:
          wordIdx = word2Idx['<unk>']
       wordIndices.append(wordIdx)
       labelIndices.append(label2Idx[label])
     sentences.append(wordIndices)
     labels.append(labelIndices)
  return sentences, labels

train_sents, train_labels = createMatrices(
    train, word2Idx, label2Idx
)
valid_sents, valid_labels = createMatrices(
    valid, word2Idx, label2Idx
)
test_sents, test_labels = createMatrices(
    test, word2Idx, label2Idx
)<jupyter_output><empty_output><jupyter_text>Padding the sequencesWe are going to pad our sequences to have the same length of 100.<jupyter_code>def padding(sents, labels, max_len=100):
  padded_sentences = keras.preprocessing.sequence.pad_sequences(sents, max_len,       
  padding='post', truncating="post")
  padded_labels = keras.preprocessing.sequence.pad_sequences(labels, max_len,       
  padding='post', truncating="post")
  return padded_sentences, padded_labels

train_features, train_labels = padding(train_sents, train_labels)
valid_features, valid_labels = padding(valid_sents, valid_labels)
test_features, test_labels = padding(test_sents, test_labels)<jupyter_output><empty_output><jupyter_text>Word embeddingsWe are going to load our word embedding, from a local file. Basically we are going to use the `glove.6B.100d` word vectors.<jupyter_code>embedding_path = "/content/drive/MyDrive/NLP Data/glove.6B/glove.6B.100d.txt"
os.path.exists(embedding_path)
embedding_dict = dict()
with open(embedding_path, encoding="utf8") as glove:
  for line in glove:
    records = line.split();
    word = records[0]
    vectors = np.asarray(records[1: ], dtype=np.float32)
    embedding_dict[word] = vectors

print(len(embedding_dict))
embedding_dict["what"].shape<jupyter_output>400000<jupyter_text>Embedding matrixWe are then going to create an embedding matrix thate suits our data.<jupyter_code>vocab_size = len(word2Idx)
embedding_matrix = np.zeros((vocab_size, 100))

for word, index in word2Idx.items():
  vector = embedding_dict.get(word)
  if vector is not None:
    embedding_matrix[index] = vector<jupyter_output><empty_output><jupyter_text>Input PipelineWe are going to make use of the `tf.data.Dataset.from_tensor_slices` to create a dataset from tensor slices so that we will be able to batch and shuffle it.<jupyter_code>BATCH_SIZE = 64
BUFFER_SIZE = train_features.shape[0]

train_dataset = tf.data.Dataset.from_tensor_slices(
    (train_features, train_labels)
).shuffle(BUFFER_SIZE, reshuffle_each_iteration=True).batch(BATCH_SIZE, 
                                                             drop_remainder=True)

valid_dataset = tf.data.Dataset.from_tensor_slices(
    (valid_features, valid_labels)
).batch(BATCH_SIZE, drop_remainder=True)

test_dataset = tf.data.Dataset.from_tensor_slices(
    (test_features, test_labels)
).batch(BATCH_SIZE, drop_remainder=True)<jupyter_output><empty_output><jupyter_text>ModelWe are going to build a Bi-Directional Long-Short Term Memory, aka BiLSTM model. We are going to use the keras subclassing API, feel free to use the sequential API it will work as well.<jupyter_code>class NER(keras.Model):
  def __init__(self, max_seq_len,
               embedding_dim,
               output_dim, weights):
    super(NER, self).__init__()

    self.embedding = keras.layers.Embedding(
        embedding_dim, 100,
        weights = [weights],
        trainable=True,
        input_length=max_seq_len
    )
    self.bilstm = keras.layers.Bidirectional(
        keras.layers.LSTM(128, 
                          dropout=.5,
                          return_sequences=True
        )
    )
    self.dropout = keras.layers.Dropout(rate=.5)
    self.out = keras.layers.Dense(output_dim)

  def call(self, x):
    x = self.dropout(self.embedding(x))
    x = self.bilstm(x)
    return self.dropout(self.out(x))
model = NER(
    max_seq_len= 100,
    embedding_dim=len(word2Idx),
    weights = embedding_matrix,
    output_dim= len(idx2Label)
)<jupyter_output><empty_output><jupyter_text>Now the model can be trained by calling the `.fit()` as follows:<jupyter_code>model.compile(
    optimizer = 'adam',
    loss = keras.losses.SparseCategoricalCrossentropy(
    from_logits=True
    ),
     metrics = ["acc"]
)

model.fit(
    train_dataset, epochs = 10, validation_data = valid_dataset
)<jupyter_output>Epoch 1/10
219/219 [==============================] - 18s 66ms/step - loss: 0.8984 - acc: 0.9705 - val_loss: 0.0676 - val_acc: 0.9810
Epoch 2/10
219/219 [==============================] - 11s 52ms/step - loss: 0.8256 - acc: 0.9792 - val_loss: 0.0472 - val_acc: 0.9876
Epoch 3/10
219/219 [==============================] - 13s 60ms/step - loss: 0.8182 - acc: 0.9807 - val_loss: 0.0399 - val_acc: 0.9887
Epoch 4/10
219/219 [==============================] - 11s 52ms/step - loss: 0.8122 - acc: 0.9820 - val_loss: 0.0337 - val_acc: 0.9912
Epoch 5/10
219/219 [==============================] - 11s 52ms/step - loss: 0.8092 - acc: 0.9828 - val_loss: 0.0295 - val_acc: 0.9925
Epoch 6/10
219/219 [==============================] - 11s 52ms/step - loss: 0.8074 - acc: 0.9835 - val_loss: 0.0277 - val_acc: 0.9926
Epoch 7/10
219/219 [==============================] - 12s 55ms/step - loss: 0.8030 - acc: 0.9839 - val_loss: 0.0245 - val_acc: 0.9936
Epoch 8/10
219/219 [==============================] - 13s 60ms[...]<jupyter_text>Evaluating the model<jupyter_code>model.evaluate(test_dataset, verbose=1)<jupyter_output>53/53 [==============================] - 1s 13ms/step - loss: 0.0205 - acc: 0.9942<jupyter_text>Model inference / Making predictionsNow our model is ready to make predictions. we are going to create a function called `predict_entities` which will make predictions for each word in a sentence and returns it's entity.<jupyter_code>sent = [w for w in [t for t, i in train[0]]]
labels = [w for w in [i for t, i in train[0]]]
sent, labels
def pedict_entities(sent):
  tokenized = sent.lower().split(" ")
  tokens = [word2Idx[token] for token in tokenized]
  tokens_padded = keras.preprocessing.sequence.pad_sequences([tokens], 100,       
  padding='post', truncating="post")

  predictions = model(tokens_padded)
  predictions= tf.squeeze(tf.argmax(predictions, axis=-1))[:len(tokens)].numpy()
  predicted_labels = [idx2Label[i] for i in predictions]
  return tokenized, predicted_labels

pedict_entities(" ".join(sent))
sent = [w for w in [t for t, i in train[0]]]
labels = [w for w in [i for t, i in train[0]]]
pedict_entities(" ".join(sent))<jupyter_output><empty_output><jupyter_text>Making more predictions<jupyter_code>for i in range(10):
  print("*"*50)
  sent = [w for w in [t for t, i in test[i]]]
  labels = [w for w in [i for t, i in test[i]]] 
  tokenized, preds = pedict_entities(" ".join(sent))
  print("sentence: ", sent)
  print("actual labels: ", labels)
  print("predicted labels: ", preds)
  print("*"*50)
  print()<jupyter_output>**************************************************
sentence:  ['\t-DOCSTART-']
actual labels:  ['O']
predicted labels:  ['O']
**************************************************

**************************************************
sentence:  ['SOCCER', '-', 'JAPAN', 'GET', 'LUCKY', 'WIN', ',', 'CHINA', 'IN', 'SURPRISE', 'DEFEAT', '.']
actual labels:  ['O', 'O', 'B-LOC', 'O', 'O', 'O', 'O', 'B-PER', 'O', 'O', 'O', 'O']
predicted labels:  ['O', 'O', 'B-LOC', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'O']
**************************************************

**************************************************
sentence:  ['Nadim', 'Ladki']
actual labels:  ['B-PER', 'I-PER']
predicted labels:  ['B-PER', 'O']
**************************************************

**************************************************
sentence:  ['AL-AIN', ',', 'United', 'Arab', 'Emirates', '1996-12-06']
actual labels:  ['B-LOC', 'O', 'B-LOC', 'I-LOC', 'I-LOC', 'O']
predicted labels:  ['B-LOC', 'O', 'B-LOC', 'I-LOC', '[...]<jupyter_text><jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17752732819252606, 'fraction_numerical': 0.03398017117193458, 'mean_word_length': 4.35480943738657, 'pattern_counts': {'":': 1, '<': 68, '<?xml version=': 0, '>': 70, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '345697', 'license': 'MIT', 'n_tokens_mistral': 4302, 'n_tokens_neox': 3970, 'n_words': 1020, 'path': '16_NLP_tuts/02_ner/01_NER_BiLSTM.ipynb', 'repo_name': 'CrispenGari/keras-api', 'size': '11807'}