<jupyter_start><jupyter_text>--- Post-procesamiento básico del texto[<NAME>](https://github.com/dmlls) \GPL-3.0 License*Última actualización: 9 de noviembre de 2020*---  IntroducciónEste notebook se centra en el proceso de post-procesado del texto de salida de los modelos `BART`y `T5`, una vez llevado a cabo el proceso de resumen. Los textos empleados como ejemplo están en inglés, dado que estos modelos están optimizados para este idioma. ---------  Requerimientos Para poder ejecutar este notebook, se debe tener instalada la última versión de los siguientes paquetes:- `NLTK`- `SpaCy` con `en_core_web_sm`- [`truecase`](https://github.com/daltonfury42/truecase) ---------  Post-procesamiento del texto El post-procesamiento del texto que vamos a llevar a cabo va a consistir en:- Formatear correctamente el texto. Esta tarea consiste fundamentalmente en eliminar espacios innecesarios (o añadirlos, p. ej., al inicio de una frase intermedia).- Añadir mayúsculas allá donde sea necesario, principalmente:  - Al comienzo de cada frase.  - En el caso de nombres de personas, países, organizaciones, etc. Dicho de forma más precisa, en el caso de aquellas entidades reconocidas que requieran mayúsculas. --- Lo primero que haremos será dividir el texto en frases para poder añadir un espacio al inicio de cada frase intermedia.Para ello, nos vamos a ayudar de la función que desarrollamos en el notebook [Pre-procesamiento del texto](https://github.com/dmlls/jizt/blob/main/notebooks/Preprocesamiento%20del%20texto.ipynb). Allí se encuentra una explicación detallada de cómo se llegó a ella.<jupyter_code>from nltk.tokenize import RegexpTokenizer
from nltk import sent_tokenize
from typing import List, Optional

def sentence_tokenize(text: str, tokenizer: Optional[RegexpTokenizer] = None) -> List[str]:
    """Divides the text into sentences.
    The steps followed are:
        - Remove characters such as '\n', '\t', etc.
        - Splits the text into sentences, taking into account Named Entities and
        special cases such as:
        + "I was born in 02.26.1980 in New York", "As we can see in Figure 1.1.
            the model will not fail.": despite the periods in the date and the
            Figure number, these texts will not be split into different sentences.
        + "Mr. Elster looked worried.", "London, capital of U.K., is famous
            for its red telephone boxes": the preprocessor applies Named Entity
            Recognition and does not split the previous sentences.
        + "Hello.Goodbye.", "Seriously??!That can't be true.": these sentences
            are split into: ['Hello.', 'Goodbye.'] and ['Seriously??!', 'That can't
            be true.'], respectively.
    
    Args:
        text:
            Text to be split in sentences.
        tokenizer:
            Regular expression to carry out a preliminar split (the text will be
            afterwards split once again by the NLTK `sent_tokenize` method).
    """

    # punctuation that shouldn't be preceeded by a whitespace
    PUNCT_NO_PREV_WHITESPACE = ".,;:!?"

    if tokenizer is None:
        # if next letter after period is lowercase, consider it part of the same sentence
        # ex: "As we can see in Figure 1.1. the sentence will not be split."
        tokenizer = RegexpTokenizer(r'[^.!?]+[.!?]+[^A-Z]*')

    # if there's no final period, add it (this makes the assumption that the last
    # sentence is not interrogative or exclamative, i.e., ends with '?' or '!')
    if text[-1] != '.' and text[-1] != '?' and text[-1] != '!':
        text += '.'

    text = ' '.join(text.split()) # remove '\n', '\t', etc.
    
    # split sentences with the regexp and ensure there's 1 whitespace at most
    sentences = ' '.join(tokenizer.tokenize(text))
    
    # remove whitespaces before PUNCT_WITHOUT_PREV_WHITESPACE
    for punct in PUNCT_NO_PREV_WHITESPACE:
        sentences = sentences.replace(' ' + punct, punct)

    sentences = sent_tokenize(sentences)

    final_sentences = [sentences[0]]

    for sent in sentences[1:]:
        # if the previous sentence doesn't end with a '.', '!' or '?'
        # we concatenate the current sentence to it
        if final_sentences[-1][-1] != '.' and \
            final_sentences[-1][-1] != '!' and \
            final_sentences[-1][-1] != '?':
            final_sentences[-1] += (' ' + sent)
        # if the next sentence doesn't start with a letter or a number,
        # we concatenate it to the previous
        elif not sent[0].isalpha() and not sent[0].isdigit():
            final_sentences[-1] += sent
        else:
            final_sentences.append(sent)
    
    return final_sentences<jupyter_output><empty_output><jupyter_text>La ventaja de esta función, aparte de identificar las frases en un texto, es que también nos elimina elementos como espacios o tabuladores innecesarios, lo cual nos interesa para el correcto formateo del texto final.Como primer paso para formatear correctamente un texto, uniremos las frases que nos devuleve el anterior método, dejando un espacio entre cada una de ellas:<jupyter_code>text = "This is a sentence.Ups!Someone forgot to add spaces after each sentence."

' '.join(sentence_tokenize(text))<jupyter_output><empty_output><jupyter_text>--- A continuación, llevaremos a cabo un proceso de Reconocimiento de Entidades Nombradas (NER, por sus siglas en inglés), con el fin de añadir mayúsculas a aquellas entidades que lo requieran.A la hora de llevar a cabo NER, dos de las opciones más ampliamente utilizadas se basan en el uso de las librerías [NLTK](https://www.nltk.org/), [SpaCy](https://spacy.io/) o [CoreNLP](https://stanfordnlp.github.io/CoreNLP/). Si atendemos al [estudio comparativo realizado por Analytics Vidhya](https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%E2%80%8Bin-python/), la librería que mejores resultados ofrece en NER es CoreNLP. No obstante, también es la más lenta.En nuestra opinión, parece que la librería SpaCy ofrece un buen compromiso entre precisión y velocidad: es más precisa que NLTK (aunque más lenta), y más rápida que CoreNLP (aunque menos precisa).<jupyter_code>import spacy
import en_core_web_sm

nlp = spacy.load('en_core_web_sm')

text = """President <NAME> announced on Twitter Monday that he has fired Secretary
          of Defense Mark Esper, and that <NAME>, who serves as director of
          the National Counterterrorism Center, will become acting secretary "effective
          immediately.""" # source: https://lite.cnn.com/en/article/h_9a38ab64c73389fb61475e431c0893e3

doc = nlp(text)

for ent in doc.ents:
    print(f" {ent.text + ' ':.<40} {ent.label_}")<jupyter_output><NAME> ........................... PERSON
 Twitter ................................ ORG
 Monday ................................. DATE
 Defense Mark Esper ..................... ORG
 <NAME> ..................... PERSON
 the National Counterterrorism Center ... ORG<jupyter_text>El modelo parece estar trabajando perfectamente en el caso anterior. Ha reconocido todas las entidades nombradas. Sin embargo, esto cambia bastante cuando pasamos todo el texto a minúsculas:<jupyter_code>text = text.lower() # pasar el texto a minúsculas

doc = nlp(text)

for ent in doc.ents:
    print(f" {ent.text + ' ':.<40} {ent.label_}")<jupyter_output>donald trump ........................... PERSON
 monday ................................. DATE
 <NAME> ..................... PERSON<jupyter_text>En este caso, solo ha identificado correctamente un 50% de las entidades nombradas. --- Echemos un vistazo a cómo la librería NLTK se comporta en este caso:<jupyter_code>import nltk
from nltk import ne_chunk
from nltk.tag import pos_tag
from nltk.tokenize import word_tokenize
from nltk.tree import Tree

text = """President <NAME> announced on Twitter Monday that he has fired Secretary
          of Defense Mark Esper, and that <NAME>, who serves as director of
          the National Counterterrorism Center, will become acting secretary "effective
          immediately.""" # source: https://lite.cnn.com/en/article/h_9a38ab64c73389fb61475e431c0893e3

ne_chunks = nltk.ne_chunk(nltk.pos_tag(word_tokenize(text)))

# print with a nice format
for chunk in ne_chunks:
    if type(chunk) == Tree: # get only NEs
        ent_text = " ".join([token for token, _ in chunk.leaves()])
        ent_label = chunk.label()
        print(f" {ent_text + ' ':.<40} {ent_label}")<jupyter_output><NAME> ........................... PERSON
 Twitter ................................ PERSON
 Defense Mark Esper ..................... ORGANIZATION
 <NAME> ..................... PERSON
 National Counterterrorism Center ....... ORGANIZATION<jupyter_text>Vemos que ha identificado correctamente todas las entidades nombradas (NLTK, a diferencia de SpaCy, no incluye las fechas como entidades nombradas). Pero, de nuevo, el problema persiste. Al eliminar las mayúsculas de las entidades nombradas, NLTK no es capaz de reconocer ninguna:<jupyter_code>text = text.lower() # pasar el texto a minúsculas

ne_chunks = nltk.ne_chunk(nltk.pos_tag(word_tokenize(text)))

# print with a nice format
for chunk in ne_chunks:
    if type(chunk) == Tree: # get only NEs
        ent_text = " ".join([token for token, _ in chunk.leaves()])
        ent_label = chunk.label()
        print(f" {ent_text + ' ':.<40} {ent_label}") # nothing to print<jupyter_output><empty_output><jupyter_text>--- Otra posibilidad podría ser poner en mayúsculas la primera letra de cada palabra y posteriormente realizar el NER:<jupyter_code>text = """President <NAME> announced on Twitter Monday that he has fired Secretary
          of Defense Mark Esper, and that <NAME>, who serves as director of
          the National Counterterrorism Center, will become acting secretary "effective
          immediately.""" # source: https://lite.cnn.com/en/article/h_9a38ab64c73389fb61475e431c0893e3

text = text.lower()

text = " ".join(word.capitalize() for word in text.split())

doc = nlp(text)

for ent in doc.ents:
    print(f" {ent.text + ' ':.<40} {ent.label_}")<jupyter_output><NAME> ........................... PERSON
 On Twitter ............................. PERSON
 Monday ................................. DATE
 <NAME> ..................... PERSON
 The National Counterterrorism Center ... ORG
 Will Become ............................ PERSON<jupyter_text>Esto ha mejorado la predicción, pero también ha introducido valores incorrectos (i.e., `On Twitter`, `Will Become`). --- Resulta que nos encontramos ante un problema mucho más complejo de lo que parecía en un primer momento. Al proceso de corregir el uso de mayúsculas en un texto se lo conoce como "truecasing". Existe un [conocido artículo](https://www.cs.cmu.edu/~llita/papers/lita.truecasing-acl2003.pdf) ofreciendo una posible solución estadística al problema. Tras la publicación de este artículo, aparecieron varias implementaciones del mismo en Python. Una de esas implementaciones nos la proporciona la librería [`truecase`](https://github.com/daltonfury42/truecase):<jupyter_code>from truecase import get_true_case

text = """President <NAME> announced on Twitter Monday that he has fired Secretary
          of Defense Mark Esper, and that <NAME>, who serves as director of
          the National Counterterrorism Center, will become acting secretary "effective
          immediately.""" # source: https://lite.cnn.com/en/article/h_9a38ab64c73389fb61475e431c0893e3

text = text.lower() # not necessary, truecase already does this

get_true_case(text)<jupyter_output><empty_output><jupyter_text>El resultado no es perfecto (i.e. `Secretary of defense mark Esper`, `National Counterterrorism center`), pero es el mejor obtenido hasta ahora. Además, el modelo que emplea por defecto `truecase` está entrenado sobre el corpus de inglés de NLTK. Entrenar un nuevo modelo es relativamente sencillo, así que en un futuro se podría probar a entrenar el modelo con un corpus más grande (p. ej., un volcado reciente de Wikipedia). --------- Poniendo todo junto, ya tenemos una primera y sencilla versión de nuestro post-procesador:<jupyter_code>from truecase import get_true_case

def post_process(text):
    txt = ' '.join(sentence_tokenize(text))
    return get_true_case(txt)
texts = ["covid is a pandemic that's been rife in the u.s. for the past 9 " +
            "months. it's not affected their daily life any more than fewer " +
            "tourists.It's as if i have arrived with my mask from a foreign planet " +
            "and they were all curious to hear what my hell has looked like " +
            "for the recent 9 months",
         "ne<NAME> was incarcerated on july 2, 2019 because of a " +
            "probation violation for a 17 year old marijuana case. the u.s. " +
            "criminal legal system is death. in 2003, neko was searched for " +
            "being a black passenger in a speeding car.",
         "the history of mutual funds in india can be broadly divided into " +
            "four distinct phases: first phase (1964-1987) in 1963, " +
            "government of india and reserve bank of india came together " +
            "to form uti (unit trust of india) and it is the first "
            "mutual fund in india."]

for txt in texts:
    print(post_process(txt), '\n')<jupyter_output>Covid is a pandemic that's been rife in the U. S. for the past 9 months. It's not affected their daily life any more than fewer tourists. It's as if I have arrived with my mask from a foreign planet and they were all curious to hear what my hell has looked like for the recent 9 months. 

<NAME> was incarcerated on July 2, 2019 because of a probation violation for a 17 year old marijuana case. The U. S. criminal legal system is death. In 2003, Neko was searched for being a black passenger in a speeding car. 

The history of mutual funds in India can be broadly divided into four distinct phases: first phase( 1964-1987) in 1963, government of India and Reserve Bank of India came together to form Uti( unit trust of India) and it is the first mutual fund in India.
{'fraction_non_alphanumeric': 0.11604604889141558, 'fraction_numerical': 0.015065378055713473, 'mean_word_length': 3.114912280701754, 'pattern_counts': {'":': 3, '<': 55, '<?xml version=': 0, '>': 51, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '338383', 'license': 'Apache-2.0', 'n_tokens_mistral': 4369, 'n_tokens_neox': 4147, 'n_words': 1637, 'path': 'notebooks/Post-procesamiento del texto.ipynb', 'repo_name': 'dmlls/jizt', 'size': '14231'}