<jupyter_start><jupyter_text>esBERTus: evaluation of the models resultsIn this notebook, an evaluation of the results obtained by the two models will be performed. The idea here is not as much to measure a benchmarking metric on the models but to understand the qualitative difference of the models.In order to do so  Keyword extractionIn order to understand what are the "hot topics" of the corpuses that are being used to train the models, a keyword extraction is performed.Although the possibility to extract keywords based in a word embeddings approach has been considered, TF-IDF has been chosen over any other approach to model the discussion topic over the different corpuses due to it's interpretability  Cleaning the textsFor this, a Spacy pipeline is used to speed up the cleaning process<jupyter_code>from spacy.language import Language
import re
@Language.component("clean_lemmatize")
def clean_lemmatize(doc):
    text = doc.text
    text = re.sub(r'\w*\d\w*', r'', text) # remove words containing digits    
    text = re.sub(r'[^a-z\s]', '', text) # remove anything that is not a letter or a space
    return nlp.make_doc(text)

print('Done!')
import spacy
# Instantiate the pipeline, disable ner component for perfomance reasons
nlp = spacy.load("en_core_web_sm", disable=['ner'])

# Add custom text cleaning function
nlp.add_pipe('clean_lemmatize', before="tok2vec")

# Apply to EU data
with open('../data/02_preprocessed/full_eu_text.txt') as f:
    eu_texts = f.readlines()
    
nlp.max_length = max([len(text)+1 for text in eu_texts])
eu_texts = [' '.join([token.lemma_ for token in doc]) for doc in nlp.pipe(eu_texts, n_process=10)] # Get lemmas
with open('../data/04_evaluation/full_eu_text_for_tfidf.txt', 'w+') as f:
    for text in eu_texts:
        f.write(text)
        f.write('\n')

print('Done EU!')

# Apply to US data
with open('../data/02_preprocessed/full_us_text.txt') as f:
    us_texts = f.readlines()
    
nlp.max_length = max([len(text)+1 for text in us_texts])
us_texts = [' '.join([token.lemma_ for token in doc]) for doc in nlp.pipe(us_texts, n_process=10)] # Get lemmas
with open('../data/04_evaluation/full_us_text_for_tfidf.txt', 'w+') as f:
    for text in us_texts:
        f.write(text)
        f.write('\n')

print('Done US!')

print('Done!')<jupyter_output>Done EU!
Done US!
Done!<jupyter_text>Keyword extractionDue to the differences in legths and number of texts, it's not possible to use a standard approach to keywords extraction. TF-IDF has been considered, but it takes away most of the very interesting keywords such as "pandemic" or "covid". This is the reason why a hybrid approach between both of the European and US corpuses has been chosen.The approach takes the top n words from one of the corpus that intersects with the top n words from the other corpus. In order to find the most relevant words, a simple count vector is used, that counts the frequency of the words. This takes only the words that are really relevant in both cases, even if you using a relatively naive approach.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

# Read the processed data
with open('../data/04_evaluation/full_eu_text_for_tfidf.txt') as f:
    eu_texts = f.readlines()
with open('../data/04_evaluation/full_us_text_for_tfidf.txt') as f:
    us_texts = f.readlines()
    
# Join the texts together
from nltk.corpus import stopwords
stopwords = set(stopwords.words('english'))
max_df = 0.9
max_features = 1000
cv_eu=CountVectorizer(max_df=max_df, stop_words=stopwords , max_features=max_features)
word_count_vector=cv_eu.fit_transform(eu_texts)
cv_us=CountVectorizer(max_df=max_df, stop_words=stopwords , max_features=max_features)
word_count_vector=cv_us.fit_transform(us_texts)
n_words = 200

keywords = [word for word in list(cv_eu.vocabulary_.keys())[:n_words] if word in list(cv_us.vocabulary_.keys())[:n_words]]
keywords<jupyter_output><empty_output><jupyter_text>Measure the models performance on masked tokens  Extract sentences where the keywords appear<jupyter_code>keywords = ['coronavirus', 'covid', 'covid-19', 'virus', 'influenza', 'flu', 
            'pandemic', 'epidemic', 'outbreak', 'crisis', 'emergency',
            'vaccine', 'vaccinated', 'mask',
           'quarantine', 'symptoms', 'antibody', 'inmunity', 'distance', 'isolation',             
            'test', 'positive', 'negative',
            'nurse', 'doctor', 'health', 'healthcare',]
import spacy
from spacy.matcher import PhraseMatcher

with open('../data/02_preprocessed/full_eu_text.txt') as f:
    eu_texts = f.readlines()

with open('../data/02_preprocessed/full_us_text.txt') as f:
    us_texts = f.readlines()
    
nlp = spacy.load("en_core_web_sm", disable=['ner'])
texts = [item for sublist in [eu_texts, us_texts] for item in sublist]

nlp.max_length = max([len(text) for text in texts])
phrase_matcher = PhraseMatcher(nlp.vocab)
patterns = [nlp(text) for text in keywords]
phrase_matcher.add('KEYWORDS', None, *patterns)

docs = nlp.pipe(texts, n_process=12)

sentences = []
block_size = 350
# Parse the docs for sentences
open('../data/04_evaluation/sentences.txt', 'wb').close()

print('Starting keyword extraction')
for doc in docs:
    for sent in doc.sents:
        # Check if the token is in the big sentence
        for match_id, start, end in phrase_matcher(nlp(sent.text)):
            if nlp.vocab.strings[match_id] in ["KEYWORDS"]:    
                # Create sentences of length of no more than block size
                tokens = sent.text.split(' ')
                if len(tokens) <= block_size:
                    sentence = sent.text
                else:
                    sentence = " ".join(tokens[:block_size])
                with open('../data/04_evaluation/sentences.txt', 'ab') as f:
                        f.write(f'{sentence}\n'.encode('UTF-8'))
                
print(f"There are {len(open('../data/04_evaluation/sentences.txt', 'rb').readlines())} sentences containing keywords")<jupyter_output>There are 68086 sentences containing keywords<jupyter_text>Measure the probability of outputing the real token in the sentence<jupyter_code># Define a custom function that feeds the three models an example and returns the perplexity
def get_masked_token_probaility(sentence:str, keywords:list, models_pipelines:list):
    # Find the word in the sentence to mask
    sentence = sentence.lower()
    keywords = [keyword.lower() for keyword in keywords]
    target = None
    for keyword in keywords:
        # Substitute only the first matched keyword
        if keyword in sentence:
            target = keyword
            masked_sentence = sentence.replace(keyword, '{}', 1)
            break
    
    if target:
        model_pipeline_results = []
        for model_pipeline in models_pipelines:
            masked_sentence = masked_sentence.format(model_pipeline.tokenizer.mask_token)
            try:
                result = model_pipeline(masked_sentence, targets=target)
                model_pipeline_results.append(result[0]['score'])
            except Exception as e:
                model_pipeline_results.append(0)
        
        return keyword, model_pipeline_results
from transformers import pipeline, AutoModelForMaskedLM, DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained('../data/03_models/tokenizer/')

# The best found European model
model=AutoModelForMaskedLM.from_pretrained("../data/03_models/eu_bert_model")
eu_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)

# The best found US model
model=AutoModelForMaskedLM.from_pretrained("../data/03_models/us_bert_model")
us_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)

model_checkpoint = 'distilbert-base-uncased'
# The baseline model from which the trainin
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)
base_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=model_checkpoint
)
results = []

print(f"There are {len(open('../data/04_evaluation/sentences.txt').readlines())} sentences to be evaluated")

for sequence in open('../data/04_evaluation/sentences.txt').readlines():
    results.append(get_masked_token_probaility(sequence, keywords, [eu_model_pipeline, us_model_pipeline, base_model_pipeline]))
    
import pickle
pickle.dump(results, open('../data/04_evaluation/sentence_token_prediction.pickle', 'wb'))<jupyter_output><empty_output><jupyter_text>Evaluate the results<jupyter_code>import pickle
results = pickle.load(open('../data/04_evaluation/sentence_token_prediction.pickle', 'rb'))
results[0:5]<jupyter_output><empty_output><jupyter_text>Frequences of masked words in the pipeline<jupyter_code>from collections import Counter
import numpy as np
import matplotlib.pyplot as plt
words = Counter([result[0] for result in results if result!=None]).most_common(len(keywords)) # most_common also sorts them

labels = [word[0] for word in words]
values = [word[1] for word in words]

indexes = np.arange(len(labels))
fix, ax = plt.subplots(figsize=(10,5))
ax.set_xticks(range(len(words)))
plt.bar(indexes, values, width=.8, align="center",alpha=.8)
plt.xticks(indexes, labels, rotation=45)

plt.title('Frequences of masked words in the pipeline')
plt.show()<jupyter_output><empty_output><jupyter_text>Average probability of all the masked keywords by model<jupyter_code>n_results = len([result for result in results if result!=None])
eu_results = sum([(result[1][0]) for result in results if result!=None]) / n_results
us_results = sum([(result[1][1]) for result in results if result!=None]) / n_results
base_results = sum([(result[1][2]) for result in results if result!=None]) / n_results

labels = ['EU model', 'US model', 'Base model']
values = [eu_results, us_results, base_results]

indexes = np.arange(len(labels))
fix, ax = plt.subplots(figsize=(10,5))
ax.set_xticks(range(len(words)))
plt.bar(indexes, values, width=.6, align="center",alpha=.8)
plt.xticks(indexes, labels, rotation=45)

plt.title('Average probability of all the masked keywords by model')
plt.show()<jupyter_output><empty_output><jupyter_text>Get the first predicted token in each sentence, masking<jupyter_code>def get_first_predicted_masked_token(sentence:str, eu_pipeline, us_pipeline, base_pipeline):
    sentence = sentence.lower()
    model_pipeline_results = []
    eu_model_pipeline_results = eu_pipeline(sentence.format(eu_pipeline.tokenizer.mask_token), top_k=1)
    us_model_pipeline_results = us_pipeline(sentence.format(us_pipeline.tokenizer.mask_token), top_k=1)
    base_model_pipeline_results = base_pipeline(sentence.format(base_pipeline.tokenizer.mask_token), top_k=1)
    
    return (eu_model_pipeline_results[0]['token_str'].replace(' ', ''), 
            us_model_pipeline_results[0]['token_str'].replace(' ', ''), 
            base_model_pipeline_results[0]['token_str'].replace(' ', '')
           )
# Create a function that identifies the first keyword in the sentences, masks it and feeds the it to the prediction function
results = []
for sequence in open('../data/04_evaluation/sentences.txt').readlines():
    target = None
    for keyword in keywords:
        if keyword in sequence:
            target = keyword
            break
    if target:
        masked_sentence = sequence.replace(target, '{}', 1)
        try:
            predictions = get_first_predicted_masked_token(masked_sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
            results.append({'masked_token': target, 
                           'eu_prediction': predictions[0],
                           'us_prediction': predictions[1],
                           'base_prediction': predictions[2]})
        except:
            pass

import pickle
pickle.dump(results, open('../data/04_evaluation/sentence_first_predicted_tokens.pickle', 'wb'))<jupyter_output>Token indices sequence length is longer than the specified maximum sequence length for this model (594 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (514 > 512). Running this sequence through the model will result in indexing errors<jupyter_text>Evaluate the results<jupyter_code>import pickle
results = pickle.load(open('../data/04_evaluation/sentence_first_predicted_tokens.pickle', 'rb'))
print(len(results))
# Group the results by masked token
from itertools import groupby
from operator import itemgetter
from collections import Counter
import numpy as np
import matplotlib.pyplot as plt
n_words = 10
results = sorted(results, key=itemgetter('masked_token'))

for keyword, v in groupby(results, key=lambda x: x['masked_token']):
    token_results = list(v)
    
    fig, ax = plt.subplots(1,3, figsize=(25,5))
    for idx, (key, name) in enumerate(zip(['eu_prediction', 'us_prediction', 'base_prediction'], ['EU', 'US', 'Base'])):
        words = Counter([item[key] for item in token_results]).most_common(n_words)
        labels, values = zip(*words)

        ax[idx].barh(labels, values, align="center",alpha=.8)
        ax[idx].set_title(f'Predicted tokens by {name} model for {keyword}')

    plt.show()<jupyter_output><empty_output><jupyter_text>Qualitative evaluation of masked token predictionThe objective of this section is not to compare the score obtained by all the models that are being used, but to compare what are the qualitative outputs of these models. This means that the comparison is going to be done manually, by inputing phrases that contain words related to the COVID-19 pandemic, and comparing the outputs of the models among them, enabling the possibility of discussion of these results.  Feeding selected phrases belonging to the European and United States institutions websites<jupyter_code>def get_masked_token(sentence:str, eu_pipeline, us_pipeline, base_pipeline, n_results=1):
    sentence = sentence.lower()
    model_pipeline_results = []
    eu_prediction = eu_pipeline(sentence.format(eu_pipeline.tokenizer.mask_token), top_k =n_results)[0]
    us_prediction = us_pipeline(sentence.format(us_pipeline.tokenizer.mask_token), top_k =n_results)[0]
    base_prediction = base_pipeline(sentence.format(base_pipeline.tokenizer.mask_token), top_k =n_results)[0]
     
    token = eu_prediction['token_str'].replace(' ', '')
    print(f"EUROPEAN MODEL      -------> {token}\n\t{eu_prediction['sequence'].replace(token, token.upper())}")
    token = us_prediction['token_str'].replace(' ', '')
    print(f"UNITED STATES MODEL -------> {token}\n\t{us_prediction['sequence'].replace(token, token.upper())}")
    token = base_prediction['token_str'].replace(' ', '')
    print(f"BASE MODEL          -------> {token}\n\t{base_prediction['sequence'].replace(token, token.upper())}")
from transformers import pipeline, AutoModelForMaskedLM, DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained('../data/03_models/tokenizer/')

# The best found European model
model=AutoModelForMaskedLM.from_pretrained("../data/03_models/eu_bert_model")
eu_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)

# The best found US model
model=AutoModelForMaskedLM.from_pretrained("../data/03_models/us_bert_model")
us_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)

model_checkpoint = 'distilbert-base-uncased'
# The baseline model from which the trainin
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)
base_model_pipeline = pipeline(
    "fill-mask",
    model=model,
    tokenizer=model_checkpoint
)<jupyter_output><empty_output><jupyter_text>European institutions sentences<jupyter_code># Source https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response_en
# Masked token: coronavirus
sentence = """The European Commission is coordinating a common European response  to the {} outbreak. We are taking resolute action to reinforce our public health sectors and mitigate the socio-economic impact in the European Union. We are mobilising all means at our disposal to help our Member States coordinate their national responses and are providing objective information about the spread of the virus and effective efforts to contain it."""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response_en
# Masked token: vaccine
sentence = """A safe and effective {} is our best chance to beat coronavirus and return to our normal lives"""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response_en
# Masked token: medicines
sentence = """The European Commission is complementing the EU Vaccines Strategy with a strategy on COVID-19 therapeutics to support the development and availability of {}"""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://ec.europa.eu/info/strategy/recovery-plan-europe_en
# Masked token: recovery
sentence = """The EU’s long-term budget, coupled with NextGenerationEU, the temporary instrument designed to boost the {}, will be the largest stimulus package ever financed in Europe. A total of €1.8 trillion will help rebuild a post-COVID-19 Europe. It will be a greener, more digital and more resilient Europe."""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)<jupyter_output>EUROPEAN MODEL      -------> economy
	the eu ’ s long - term budget, coupled with nextgenerationeu, the temporary instrument designed to boost the ECONOMY, will be the largest stimulus package ever financed in europe. a total of €1. 8 trillion will help rebuild a post - covid - 19 europe. it will be a greener, more digital and more resilient europe.
UNITED STATES MODEL -------> economy
	the eu ’ s long - term budget, coupled with nextgenerationeu, the temporary instrument designed to boost the ECONOMY, will be the largest stimulus package ever financed in europe. a total of €1. 8 trillion will help rebuild a post - covid - 19 europe. it will be a greener, more digital and more resilient europe.
BASE MODEL          -------> economy
	the eu ’ s long - term budget, coupled with nextgenerationeu, the temporary instrument designed to boost the ECONOMY, will be the largest stimulus package ever financed in europe. a total of €1. 8 trillion will help rebuild a post - covid - 19 europe. it wil[...]<jupyter_text>US Government sentences<jupyter_code># Source https://www.usa.gov/covid-unemployment-benefits
# Masked token: provide
sentence = 'The federal government has allowed states to change their laws to {} COVID-19 unemployment benefits for people whose jobs have been affected by the coronavirus pandemic.'
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://www.usa.gov/covid-passports-and-travel
# Masked token: mask-wearing
sentence = """Many museums, aquariums, and zoos have restricted access or are closed during the pandemic. And many recreational areas including National Parks have COVID-19 restrictions and {} rules. Check with your destination for the latest information."""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://www.usa.gov/covid-stimulus-checks
# Masked token: people
sentence = """The American Rescue Plan Act of 2021 provides $1,400 Economic Impact Payments for {} who are eligible. You do not need to do anything to receive your payment. It will arrive by direct deposit to your bank account, or by mail in the form of a paper check or debit card."""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://www.usa.gov/covid-scams
# Masked token: scammers
sentence = """During the COVID-19 pandemic, {} may try to take advantage of you. They might get in touch by phone, email, postal mail, text, or social media. Protect your money and your identity. Don't share personal information like your bank account number, Social Security number, or date of birth. Learn how to recognize and report a COVID vaccine scam and other types of coronavirus scams. """
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)
# Source https://www.acf.hhs.gov/coronavirus
# Masked token: situation
sentence = """With the COVID-19 {} continuing to evolve, we continue to provide relevant resources to help our grantees, partners, and stakeholders support children, families, and communities in need during this challenging time."""
get_masked_token(sentence, eu_model_pipeline, us_model_pipeline, base_model_pipeline)<jupyter_output>EUROPEAN MODEL      -------> crisis
	with the covid - 19 CRISIS continuing to evolve, we continue to provide relevant resources to help our grantees, partners, and stakeholders support children, families, and communities in need during this challenging time.
UNITED STATES MODEL -------> pandemic
	with the covid - 19 PANDEMIC continuing to evolve, we continue to provide relevant resources to help our grantees, partners, and stakeholders support children, families, and communities in need during this challenging time.
BASE MODEL          -------> program
	with the covid - 19 PROGRAM continuing to evolve, we continue to provide relevant resources to help our grantees, partners, and stakeholders support children, families, and communities in need during this challenging time.
{'fraction_non_alphanumeric': 0.09841387626379743, 'fraction_numerical': 0.008394397551247565, 'mean_word_length': 4.584822584822585, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 55, 'https://': 9, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '2503', 'license': 'Apache-2.0', 'n_tokens_mistral': 6521, 'n_tokens_neox': 6264, 'n_words': 2031, 'path': '04_Evaluation/04_evaluation.ipynb', 'repo_name': 'danieldiezmallo/euBERTus', 'size': '21562'}