<jupyter_start><jupyter_text>Neural Network for Spanish Named Entity Recognition   Jupyter Notebook based on: **<NAME>** NER with Bidirectional LSTM-CNNs implementation available on Github. https://github.com/kamalkraj/Named-Entity-Recognition-with-Bidirectional-LSTM-CNNs.**Versión: -v_1.2-**Notas de version:            - Se implementa las word embeddings en español:         GloVe embeddings for SWBC; dimensions=300, vectors=855380.    - Se modifico y mejoró el preprocesamiento de los datos de entrada para predicción. Ahora puede predecir las etiquetas I-(PER/LOC/ORG/MISC)    Para 50 epoch:    -Tiene un accuracy :~80     -No se implementa nada para el español, se encontró que era perjudicial con las embeddings.    -Tiempo: 38 min aprox.      Para 100 epoch:     -Tiene un accuracy :~84 (    -No se implementa nada para el español, se encontró que era perjudicial con las embeddings.    -Tiempo: 1 hr 20 min aprox.      Entrenamiento realizado en:    DESKTOP-0UQLV13    Processor: Intel Core i7-6700HQ CPU 2.6GHz     RAM: 16GB    OS: Windows 10 Home Single x64    Tipo de memoria: SSD    Requiere:    unidecode    numpy (pip install --upgrade numpy)    nltk (pip install --upgrade nltk)    * Descargar nltk punkt y nltk stopwords:    * >> import nltk     * >> nltk.download('stopwords')    * >> nltk.download('punkt')    * Para más información: https://www.nltk.org/data.html     random    tensorflow 1.13.1 (pip install --upgrade tensorflow) *Actualmente (10/abril/19) no funciona con python 3.7.    * Para más información: https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class01_intro_python.ipynb     keras (pip install --upgrade keras) NER task can be formulated as: _Given a sequence of tokens (words, and may be punctuation symbols) provide a tag from predefined set of tags for each token in the sequence._For NER task there are some common types of entities which essentially are tags:- Persons- Locations- Organizations- Expressions of time- Quantities- Monetary values Furthermore, to distinguish consequent entities with the same tags BIO tagging scheme is used. "B" stands for beginning, "I" stands for the continuation of an entity and "O" means the absence of entity. Example with dropped punctuation:    Bernhard        B-PER    Riemann         I-PER    Carl            B-PER    Friedrich       I-PER    Gauss           I-PER    and             O    Leonhard        B-PER    Euler           I-PERIn the example above PER means person tag, and "B-" and "I-" are prefixes identifying beginnings and continuations of the entities. Without such prefixes, it is impossible to separate <NAME> from <NAME>.<jupyter_code># Np for math 
# Keras for models, layers 4 NN layers 
import numpy as np
from keras.models import Model
from keras.layers import TimeDistributed,Conv1D,Dense,Embedding,Input,Dropout,LSTM,Bidirectional,MaxPooling1D,Flatten,concatenate
from keras.utils import Progbar
from keras.preprocessing.sequence import pad_sequences
from keras.initializers import RandomUniform
import unidecode
import string 

# Read file (txt) and divide the sentences into character bins (word, tag).
def readfile(filename):
    '''
    read file
    return format :
    [ ['EU', 'B-ORG'], ['rejects', 'O'], ['German', 'B-MISC'], ['call', 'O'], ['to', 'O'], ['boycott', 'O'], ['British', 'B-MISC'], ['lamb', 'O'], ['.', 'O'] ]
    '''
    f = open(filename, encoding='utf-8-sig') # open the file. Update to fix 'ï»¿'
    sentences = []
    sentence = []
    for line in f:
        if len(line)==0 or line.startswith('-DOCSTART') or line[0]=="\n":
            if len(sentence) > 0:     
                sentences.append(sentence)
                sentence = []
            continue
        splits = line.split(' ')
        #splits[0] = unidecode.unidecode(splits[0]) # Remove special characters from spanish
        #splits[0] = splits[0].lower() # Lowercase the words 
        #splits[0] = splits[0].translate(str.maketrans('', '', string.punctuation)) # remove puntuation 
        splits[-1] = splits[-1].replace('\n', '').replace('\r', '') #Remove all line breaks from a long string of text
        if splits[0] != '':
            sentence.append([splits[0],splits[-1]])

    if len(sentence) >0: 
        sentences.append(sentence)
        sentence = []
    return sentences
# Read the 3 sets ************************************************* PATH ************************************************
# Dataset CoNLL 2002 for Spanish, wich is divided into train, test, valid (dev) sets. Each row contains a word and it's tag
# https://github.com/teropa/nlp/tree/master/resources/corpora/conll2002 
trainSentences = readfile("tidy_data/train.txt")
devSentences = readfile("tidy_data/valid.txt")
testSentences = readfile("tidy_data/test.txt")
print(len(trainSentences))
trainSentences[0]
devSentences[0]
testSentences[0]
# Create new attribute in the character bins for padding
def addCharInformatioin(Sentences):
    for i,sentence in enumerate(Sentences):
        for j,data in enumerate(sentence):
            chars = [c for c in data[0]]
            Sentences[i][j] = [data[0],chars,data[1]]
    return Sentences
trainSentences = addCharInformatioin(trainSentences)
devSentences = addCharInformatioin(devSentences)
testSentences = addCharInformatioin(testSentences)
trainSentences[0]
devSentences[0]
testSentences[0]
# 1.Creates the label set ( tag's set)
# 2.Creates a set with the lowercased words contained in the train,dev,test sets 
labelSet = set()
words = {}

for dataset in [trainSentences, devSentences, testSentences]:
    for sentence in dataset:
        for token,char,label in sentence:
            labelSet.add(label)
            words[token.lower()] = True
labelSet
words
# Gives the labels a numerical id.
# :: Create a mapping for the labels ::
label2Idx = {}
for label in labelSet:
    label2Idx[label] = len(label2Idx)
label2Idx
# Look up table
# :: Hard coded case lookup ::
case2Idx = {'numeric': 0, 'allLower':1, 'allUpper':2, 'initialUpper':3, 'other':4, 'mainly_numeric':5, 'contains_digit': 6, 'PADDING_TOKEN':7}
caseEmbeddings = np.identity(len(case2Idx), dtype='float32')
case2Idx
caseEmbeddings
# :: Read in word embeddings ::
word2Idx = {}
wordEmbeddings = []
# *********************************************************************************************** PATH *************************************************
# GloVe embeddings from SBWC
# https://github.com/uchile-nlp/spanish-word-embeddings
#* Hace los wordEmbedings en base a la lista de embedings + revisa si la palabra en embeddings esta contenido en la lista 
# de palabras ** Nota: Remember that the words are seen as vectors.
with open("word_embeddings/SBW-vectors-300-min5.txt", encoding="utf-8") as fEmbeddings:  ## change to skip first line (headings)
    next(fEmbeddings)
    for line in fEmbeddings:
        split = line.strip().split(' ')
        word = split[0]

        if len(word2Idx) == 0: #Add padding+unknown
            word2Idx["PADDING_TOKEN"] = len(word2Idx)
            vector = np.zeros(len(split)-1) #Zero vector vor 'PADDING' word
            wordEmbeddings.append(vector)

            word2Idx["UNKNOWN_TOKEN"] = len(word2Idx)
            vector = np.random.uniform(-0.25, 0.25, len(split)-1)
            wordEmbeddings.append(vector)

        if split[0].lower() in words:
            vector = np.array([float(num) for num in split[1:]])
            wordEmbeddings.append(vector)
            word2Idx[split[0]] = len(word2Idx)

    wordEmbeddings = np.array(wordEmbeddings)
wordEmbeddings
wordEmbeddings.shape[0]
wordEmbeddings.shape[1]
char2Idx = {"PADDING":0, "UNKNOWN":1}
for c in " 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZáéíóúñäëïöüÁÉÍÓÚÄËÏÖÜÃÂñÑàèìòùÀÈÌÒÙ.,-_()[]{}¡!¿?:;#'\"/\\%$`&=*+@^~|‘´·»³©\xadº±¼\xa0":
    char2Idx[c] = len(char2Idx)
# characters and position (val)
char2Idx 
# words and possition (val)
word2Idx
### Padding the sentences 
def padding(Sentences):
    maxlen = 52
    for sentence in Sentences:
        char = sentence[2]
        for x in char:
            maxlen = max(maxlen,len(x))
    for i,sentence in enumerate(Sentences):
        Sentences[i][2] = pad_sequences(Sentences[i][2],52,padding='post')
    return Sentences
# classify the word according to the caseLookup ( numeric, mainly_numeric, allLower, allUpper, initialUpper, contains_digit) 
def getCasing(word, caseLookup):   
    casing = 'other'
    #Get number of digits in word
    numDigits = 0
    for char in word:
        if char.isdigit():
            numDigits += 1
            
    digitFraction = numDigits / float(len(word))
    
    if word.isdigit(): #Is a digit
        casing = 'numeric'
    elif digitFraction > 0.5:
        casing = 'mainly_numeric'
    elif word.islower(): #All lower case
        casing = 'allLower'
    elif word.isupper(): #All upper case
        casing = 'allUpper'
    elif word[0].isupper(): #is a title, initial char upper, then all lower
        casing = 'initialUpper'
    elif numDigits > 0:
        casing = 'contains_digit'
    
   
    return caseLookup[casing]
# Create words embeding matrices to padding
def createMatrices(sentences, word2Idx, label2Idx, case2Idx,char2Idx):
    unknownIdx = word2Idx['UNKNOWN_TOKEN']
    paddingIdx = word2Idx['PADDING_TOKEN']    
        
    dataset = []
    
    wordCount = 0
    unknownWordCount = 0
    
    for sentence in sentences:
        wordIndices = []    
        caseIndices = []
        charIndices = []
        labelIndices = []
        
        for word,char,label in sentence:  
            wordCount += 1
            # if the word is in the list of words to index, then index it (verify with the lower cased word)
            if word in word2Idx:
                wordIdx = word2Idx[word]
            elif word.lower() in word2Idx:
                wordIdx = word2Idx[word.lower()]                 
            else: # else tag it as unknown
                wordIdx = unknownIdx
                unknownWordCount += 1
            charIdx = []
            for x in char:
                charIdx.append(char2Idx[x])
            #Get the label and map to int            
            wordIndices.append(wordIdx)
            caseIndices.append(getCasing(word, case2Idx)) #Call getCasing
            charIndices.append(charIdx)
            labelIndices.append(label2Idx[label])
           
        dataset.append([wordIndices, caseIndices, charIndices, labelIndices]) 
        
    return dataset
# Padding the train/dev/test set and convert them to embedings

train_set = padding(createMatrices(trainSentences,word2Idx,  label2Idx, case2Idx,char2Idx))
dev_set = padding(createMatrices(devSentences,word2Idx, label2Idx, case2Idx,char2Idx))
test_set = padding(createMatrices(testSentences, word2Idx, label2Idx, case2Idx,char2Idx))

trainSentences[0]
train_set[0]
# save the words and labels to index as dict types
idx2Label = {v: k for k, v in label2Idx.items()}
#***************************************PATH***********************
np.save("model_data/idx2Label.npy",idx2Label)
np.save("model_data/word2Idx.npy",word2Idx)
# Create a batch for each set (later we will create mini-batch)
def createBatches(data):
    l = []
    for i in data:
        l.append(len(i[0]))
    l = set(l)
    batches = []
    batch_len = []
    z = 0
    for i in l:
        for batch in data:
            if len(batch[0]) == i:
                batches.append(batch)
                z += 1
        batch_len.append(z)
    return batches,batch_len
train_batch,train_batch_len = createBatches(train_set)
dev_batch,dev_batch_len = createBatches(dev_set)
test_batch,test_batch_len = createBatches(test_set)
#train_batch_len<jupyter_output><empty_output><jupyter_text>Start with Tensorflow. Remember that tf first construct a graph, and then run it. tf automatically determines the best contruction taking into consideration each node requirements.<jupyter_code># Create a tensor for the inputs
words_input = Input(shape=(None,),dtype='int32',name='words_input')
# Create a tensor of the embeddings using the words embeddings and feeding with the words_input tensor
words = Embedding(input_dim=wordEmbeddings.shape[0], output_dim=wordEmbeddings.shape[1],  weights=[wordEmbeddings], trainable=False)(words_input)
# Create a tensor of casing input
casing_input = Input(shape=(None,), dtype='int32', name='casing_input')
#Create a tensor of the casing using the words embeddings and feeding with the casing_input tensor
casing = Embedding(output_dim=caseEmbeddings.shape[1], input_dim=caseEmbeddings.shape[0], weights=[caseEmbeddings], trainable=False)(casing_input)<jupyter_output><empty_output><jupyter_text>More tensors for the model....<jupyter_code>character_input=Input(shape=(None,52,),name='char_input')
embed_char_out=TimeDistributed(Embedding(len(char2Idx),30,embeddings_initializer=RandomUniform(minval=-0.5, maxval=0.5)), name='char_embedding')(character_input)
# Establish the dropout (neurons?)
dropout= Dropout(0.5)(embed_char_out)
conv1d_out= TimeDistributed(Conv1D(kernel_size=3, filters=30, padding='same',activation='tanh', strides=1))(dropout)
# max pool of the convolutional
maxpool_out=TimeDistributed(MaxPooling1D(52))(conv1d_out)
# Flattern  layer for the CNN, it is requered to be flattern for the CNN
char = TimeDistributed(Flatten())(maxpool_out)
char = Dropout(0.5)(char)
output = concatenate([words, casing,char])
output = Bidirectional(LSTM(200, return_sequences=True, dropout=0.50, recurrent_dropout=0.25))(output)
output = TimeDistributed(Dense(len(label2Idx), activation='softmax'))(output)<jupyter_output><empty_output><jupyter_text>Model. Inlcudes a Summary of the Model.<jupyter_code>model = Model(inputs=[words_input, casing_input,character_input], outputs=[output])
model.compile(loss='sparse_categorical_crossentropy', optimizer='nadam')
model.summary()
#Number of epochs
epochs = 150
# Minibatches
def iterate_minibatches(dataset,batch_len): 
    start = 0
    for i in batch_len:
        tokens = []
        caseing = []
        char = []
        labels = []
        data = dataset[start:i]
        start = i
        for dt in data:
            t,c,ch,l = dt
            l = np.expand_dims(l,-1)
            tokens.append(t)
            caseing.append(c)
            char.append(ch)
            labels.append(l)
        yield np.asarray(labels),np.asarray(tokens),np.asarray(caseing),np.asarray(char)
#Training of n epochs 
for epoch in range(epochs):    
    print("Epoch %d/%d"%(epoch,epochs))
    a = Progbar(len(train_batch_len))
    for i,batch in enumerate(iterate_minibatches(train_batch,train_batch_len)):
        labels, tokens, casing,char = batch       
        model.train_on_batch([tokens, casing,char], labels)
        a.update(i)
    a.update(i+1)
    print(' ')
# Saving the model
model.save("model_data/model.h5")<jupyter_output><empty_output><jupyter_text>Evaluating model accurracy. Using F1, precision  and recal for Dev and Test sets.<jupyter_code>def tag_dataset(dataset):
    correctLabels = []
    predLabels = []
    b = Progbar(len(dataset))
    for i,data in enumerate(dataset):    
        tokens, casing,char, labels = data
        tokens = np.asarray([tokens])     
        casing = np.asarray([casing])
        char = np.asarray([char])
        pred = model.predict([tokens, casing,char], verbose=False)[0]   
        pred = pred.argmax(axis=-1) #Predict the classes            
        correctLabels.append(labels)
        predLabels.append(pred)
        b.update(i)
    b.update(i+1)
    return predLabels, correctLabels
#Method to compute the accruarcy. Call predict_labels to get the labels for the dataset
def compute_f1(predictions, correct, idx2Label): 
    label_pred = []    
    for sentence in predictions:
        label_pred.append([idx2Label[element] for element in sentence])
        
    label_correct = []    
    for sentence in correct:
        label_correct.append([idx2Label[element] for element in sentence])
            
    
    #print label_pred
    #print label_correct
    
    prec = compute_precision(label_pred, label_correct)
    rec = compute_precision(label_correct, label_pred)
    
    f1 = 0
    if (rec+prec) > 0:
        f1 = 2.0 * prec * rec / (prec + rec);
        
    return prec, rec, f1
def compute_precision(guessed_sentences, correct_sentences):
    assert(len(guessed_sentences) == len(correct_sentences))
    correctCount = 0
    count = 0
    
    
    for sentenceIdx in range(len(guessed_sentences)):
        guessed = guessed_sentences[sentenceIdx]
        correct = correct_sentences[sentenceIdx]
        assert(len(guessed) == len(correct))
        idx = 0
        while idx < len(guessed):
            if guessed[idx][0] == 'B': #A new chunk starts
                count += 1
                
                if guessed[idx] == correct[idx]:
                    idx += 1
                    correctlyFound = True
                    
                    while idx < len(guessed) and guessed[idx][0] == 'I': #Scan until it no longer starts with I
                        if guessed[idx] != correct[idx]:
                            correctlyFound = False
                        
                        idx += 1
                    
                    if idx < len(guessed):
                        if correct[idx][0] == 'I': #The chunk in correct was longer
                            correctlyFound = False
                        
                    
                    if correctlyFound:
                        correctCount += 1
                else:
                    idx += 1
            else:  
                idx += 1
    
    precision = 0
    if count > 0:    
        precision = float(correctCount) / count
        
    return precision
#   Performance on dev dataset        
predLabels, correctLabels = tag_dataset(dev_batch)        
pre_dev, rec_dev, f1_dev = compute_f1(predLabels, correctLabels, idx2Label)
print("Dev-Data: Prec: %.3f, Rec: %.3f, F1: %.3f" % (pre_dev, rec_dev, f1_dev))
#   Performance on test dataset       
predLabels, correctLabels = tag_dataset(test_batch)        
pre_test, rec_test, f1_test= compute_f1(predLabels, correctLabels, idx2Label)
print("Test-Data: Prec: %.3f, Rec: %.3f, F1: %.3f" % (pre_test, rec_test, f1_test))<jupyter_output>1517/1517 [==============================] - 8s 5ms/step
Test-Data: Prec: 0.849, Rec: 0.853, F1: 0.851<jupyter_text>Test with data<jupyter_code>from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords<jupyter_output><empty_output><jupyter_text>Defining class for testing.<jupyter_code>import numpy as np
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
from keras.models import load_model
from keras.preprocessing.sequence import pad_sequences
from nltk import word_tokenize

class Parser:

    def __init__(self):
        # ::Hard coded char lookup ::
        self.char2Idx = {"PADDING":0, "UNKNOWN":1}
        for c in " 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ.,-_()[]{}!?:;#'\"/\\%$`&=*+@^~|":
            self.char2Idx[c] = len(self.char2Idx)
        # :: Hard coded case lookup ::
        self.case2Idx = {'numeric': 0, 'allLower':1, 'allUpper':2, 'initialUpper':3, 'other':4, 'mainly_numeric':5, 'contains_digit': 6, 'PADDING_TOKEN':7}

    def load_models(self, loc=None):
        if not loc:
            loc = os.path.join(os.path.expanduser('~'), '.ner_model')
        self.model = load_model(os.path.join(loc,"model.h5"))
        # loading word2Idx
        self.word2Idx = np.load(os.path.join(loc,"word2Idx.npy")).item()
        # loading idx2Label
        self.idx2Label = np.load(os.path.join(loc,"idx2Label.npy")).item()

    def getCasing(self,word, caseLookup):   
        casing = 'other'
        
        numDigits = 0
        for char in word:
            if char.isdigit():
                numDigits += 1
                
        digitFraction = numDigits / float(len(word))
        
        if word.isdigit(): #Is a digit
            casing = 'numeric'
        elif digitFraction > 0.5:
            casing = 'mainly_numeric'
        elif word.islower(): #All lower case
            casing = 'allLower'
        elif word.isupper(): #All upper case
            casing = 'allUpper'
        elif word[0].isupper(): #is a title, initial char upper, then all lower
            casing = 'initialUpper'
        elif numDigits > 0:
            casing = 'contains_digit'  
        return caseLookup[casing]

    def createTensor(self,sentence, word2Idx,case2Idx,char2Idx):
        unknownIdx = word2Idx['UNKNOWN_TOKEN']
    
        wordIndices = []    
        caseIndices = []
        charIndices = []
            
        for word,char in sentence:  
            word = str(word)
            if word in word2Idx:
                wordIdx = word2Idx[word]
            elif word.lower() in word2Idx:
                wordIdx = word2Idx[word.lower()]                 
            else:
                wordIdx = unknownIdx
            charIdx = []
            for x in char:
                if x in char2Idx.keys():
                    charIdx.append(char2Idx[x])
                else:
                    charIdx.append(char2Idx['UNKNOWN'])   
            wordIndices.append(wordIdx)
            caseIndices.append(self.getCasing(word, case2Idx))
            charIndices.append(charIdx)
            
        return [wordIndices, caseIndices, charIndices]

    def addCharInformation(self, sentence):
        return [[word, list(str(word))] for word in sentence]

    def padding(self,Sentence):
        Sentence[2] = pad_sequences(Sentence[2],52,padding='post')
        return Sentence

    def predict(self,Sentence):
        Sentence = words =  word_tokenize(Sentence)
        Sentence = self.addCharInformation(Sentence)
        Sentence = self.padding(self.createTensor(Sentence,self.word2Idx,self.case2Idx,self.char2Idx))
        tokens, casing,char = Sentence
        tokens = np.asarray([tokens])     
        casing = np.asarray([casing])
        char = np.asarray([char])
        pred = self.model.predict([tokens, casing,char], verbose=False)[0]   
        pred = pred.argmax(axis=-1)
        pred = [self.idx2Label[x].strip() for x in pred]

        return  list(zip(words,pred))
       
p = Parser()
p.load_models("model_data/")
from nltk import sent_tokenize
text_file = open("Input_sample.txt").read()
token_sent = sent_tokenize(text_file)
print(token_sent)<jupyter_output>['MÉXICO.—<NAME> va a renunciar a la Secretaría de Relaciones Exteriores, según versiones periodísticas que hoy fueron desmentidas por la SRE.', '“En el ámbito de la fantasía”\n\n<NAME>, vocero de la Cancillería, aseguró a través de Twitter que es totalmente falsa la versión que circuló sobre el tema.', 'El evento referido, señaló, sólo ocurrió en el ámbito de la fantasía, según lo publicado por El Universal.', 'Primero <NAME>\n\nAyer lunes también circuló la versión sobre una supuesta renuncia del Fiscal General de la República, <NAME>, por cuestiones de salud.', 'También fue desmentido por la dependencia.', 'Hoy en importante firma\n\nEsta mañana, en Palacio Nacional, con el presidente <NAME> como testigo de honor, <NAME>, alta comisionada de Naciones Unidas para los Derechos Humanos, y el canciller <NAME> firmaron el Acuerdo para la formación en materia de derechos humanos y operación de acuerdo a[...]<jupyter_text>Input:<jupyter_code>print(text_file)
outlist =[]
for t in token_sent:
    t= unidecode.unidecode(t)
    outlist.append(p.predict(t))

to_out=[]
for s in outlist:
    for w in s:
        if ('O') not in w:
            print(w)
            to_out.append(w)
            
with open('Output_sample.txt', 'w') as f:
    for item in to_out:
        f.write("\n")
        for x in item:
            f.write("%s " %x)<jupyter_output>('MEXICO.', 'B-ORG')
('Marcelo', 'B-PER')
('Ebrard', 'I-PER')
('Secretaria', 'B-ORG')
('de', 'I-ORG')
('Relaciones', 'I-MISC')
('Exteriores', 'I-MISC')
('SRE', 'B-MISC')
('Roberto', 'B-PER')
('Velasco', 'I-PER')
('Alvarez', 'I-PER')
('Cancilleria', 'B-PER')
('Twitter', 'B-LOC')
('senalo', 'I-MISC')
('El', 'B-ORG')
('Universal', 'I-ORG')
('Gertz', 'B-PER')
('Moreno', 'I-PER')
('Ayer', 'I-PER')
('Fiscal', 'B-MISC')
('General', 'I-MISC')
('de', 'I-ORG')
('la', 'I-ORG')
('Republica', 'I-ORG')
('Alejandro', 'B-PER')
('Gertz', 'I-PER')
('Manero', 'I-PER')
('Tambien', 'B-PER')
('Esta', 'B-MISC')
('Palacio', 'B-LOC')
('Nacional', 'I-LOC')
('Andres', 'B-PER')
('Manuel', 'I-PER')
('Lopez', 'I-PER')
('Obrador', 'I-PER')
('Michelle', 'I-PER')
('Bachelet', 'I-PER')
('Naciones', 'B-MISC')
('Unidas', 'I-MISC')
('para', 'I-ORG')
('los', 'I-MISC')
('Derechos', 'I-MISC')
('Humanos', 'I-MISC')
('Marcelo', 'B-PER')
('Ebrard', 'I-PER')
('Acuerdo', 'B-MISC')
('Guardia', 'B-ORG')
('Nacional', 'I-ORG')
('Ebra[...]
{'fraction_non_alphanumeric': 0.13146621403168446, 'fraction_numerical': 0.015963465890720983, 'mean_word_length': 3.0111849570432807, 'pattern_counts': {'":': 7, '<': 43, '<?xml version=': 0, '>': 54, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '378793', 'license': 'Apache-2.0', 'n_tokens_mistral': 8182, 'n_tokens_neox': 7838, 'n_words': 2054, 'path': 'code/final_code_model/Jupyter_example/NER_v2.1_SampleNotebook.ipynb', 'repo_name': 'agrro/NER--Spanish-', 'size': '24858'}