<jupyter_start><jupyter_text>Text ReordererThis notebook provides a method to reorder the sentences in a text, using BERT For Next Sentence Prediction.It is an important NLP task, especially when you have some unordered text coming from different sources such as in Summarization o Multi-Summarization tasks.    Required dependencies and set-upThe `transformers` library from HuggingFace to import BertTokenizer, BertForNextSentencePrediction is used. The tokenizer is used to tokenize the senteces used in the Next-Sentence-Prediction Model. The `bert-base-multilingual-cased` as pretrained weights are used to support cased text and several languages.The `tensorflow` library is used to build the tensors for the model.Finally, `tensorflow.keras` is used later to compute the softmax from the logits in the Next-Sentence-Prediction task.<jupyter_code>!pip install transformers nltk tensorflow
from transformers import BertTokenizer, TFBertForNextSentencePrediction
import tensorflow as tf
import tensorflow.keras

from preprocess_test import preprocess
# pretrained_weights = 'bert-base-multilingual-cased'
pretrained_weights = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)
nsp_model = TFBertForNextSentencePrediction.from_pretrained(pretrained_weights)<jupyter_output><empty_output><jupyter_text>FunctionsIn this section, the function needed for the text reordering are explained.  predict_next_sentence_probThe `predict_next_sentence_prob` function aims to return the probability that the second sentence provided in input is the continuation of the first one.<jupyter_code>def predict_next_sentence_prob(sent1, sent2):
    # encode the two sequences. Particularly, make clear that they must be 
    # encoded as "one" input to the model by using 'seq_B' as the 'text_pair'
    # NOTE how the token_type_ids are 0 for all tokens in seq_A and 1 for seq_B, 
    # this way the model knows which token belongs to which sequence
    encoded = tokenizer.encode_plus(sent1, text_pair=sent2)
    encoded["input_ids"] = tf.constant(encoded["input_ids"])[None, :]
    encoded["token_type_ids"] = tf.constant(encoded["token_type_ids"])[None, :]
    encoded["attention_mask"] = tf.constant(encoded["attention_mask"])[None, :]

    # a model's output is a tuple, we only need the output tensor containing
    # the relationships which is the first item in the tuple
    outputs = nsp_model(encoded)
    seq_relationship_scores = outputs[0]

    # we need softmax to convert the logits into probabilities
    # index 0: sequence B is a continuation of sequence A
    # index 1: sequence B is a random sequence
    probs = tf.keras.activations.softmax(seq_relationship_scores, axis=-1)
    return probs.numpy()[0][0]<jupyter_output><empty_output><jupyter_text>In the following snippet the Next-Sentence-Prediction probabilities over two sentences are computed. You can see from the examples the results.<jupyter_code>all_sentences = prerprocess("dev_nsp.tsv")  # 2차원 리스트  

for sentences in all_sentences:
    print("Probability that sent2 is the next sentence with respect to sent1:",
      predict_next_sentence_prob(sentences[0], sentences[1]))
    print("Probability that sent1 is the next sentence with respect to sent2:",
      predict_next_sentence_prob(sentences[1], sentences[0])) # better
    print("Probability that a sentence is the next sentence with respect to itself:",
      predict_next_sentence_prob(sentences[1], sentences[1])) # very low
    
    
'''  
sentences = [
    "So, the old man decided to go home to make his dog feel better.",
    "In the park, there were also a mouse and the dog was frightened.",
    "The dog felt better because the man gave 3 biscuits to him.",
    "One day, an old man went to the park with his dog.",
]

print("Probability that sent2 is the next sentence with respect to sent1:",
      predict_next_sentence_prob(sentences[0], sentences[1]))
print("Probability that sent1 is the next sentence with respect to sent2:",
      predict_next_sentence_prob(sentences[1], sentences[0])) # better
print("Probability that a sentence is the next sentence with respect to itself:",
      predict_next_sentence_prob(sentences[1], sentences[1])) # very low
'''<jupyter_output>Probability that sent2 is the next sentence with respect to sent1: 0.9987509
Probability that sent1 is the next sentence with respect to sent2: 0.99957067
Probability that a sentence is the next sentence with respect to itself: 0.008009296<jupyter_text>create_correlation_matrixThe `create_correlation_matrix` takes a string array-like and create a correlation matrix which takes care of the all probabilities computed over the sentences given in input.<jupyter_code>def create_correlation_matrix(sentences: list):
    num_sentences = len(sentences) 
    correlation_matrix = np.empty(shape=(num_sentences, num_sentences), dtype=float, order='C')
    for i, s1 in enumerate(sentences):
        for j, s2 in enumerate(sentences):
            correlation_matrix[i][j] = predict_next_sentence_prob(s1, s2)
    return correlation_matrix
for sentences in all_sentences:
    correlation_matrix = create_correlation_matrix(sentences)
    correlation_matrix

'''    
correlation_matrix = create_correlation_matrix(sentences)
correlation_matrix
'''<jupyter_output><empty_output><jupyter_text>reorder_sentencesThe `reorder_sentences` takes a string array-like. It creates a correlation matrix with the previous function and iteratively finds the best association in the matrix. It deletes the sentences selected in this way and continues until no association can be done. The association is expressed as the tuple (X, Y), where X is the first sentence and the Y is the most probable next sentence. The latter is used as the new first sentence from which search the best next one.<jupyter_code>def reorder_sentences(sentences: list):
    ordering = []
    correlation_matrix = create_correlation_matrix(sentences)
    hint = None
    while correlation_matrix.any():
        if hint == None:
            ind = np.unravel_index(np.argmax(correlation_matrix, axis=None), correlation_matrix.shape)
        else:
            ind = np.unravel_index(np.argmax(correlation_matrix[hint,:], axis=None), correlation_matrix[hint,:].shape)
            ind = (hint, ind[0])
        hint = ind[1]    
        correlation_matrix[ind[0], :] = 0
        correlation_matrix[:, ind[0]] = 0
        ordering.append(ind[0])
    return ordering
for sentences in all_sentences:
    ordering = reorder_sentences(sentences)
    reordered_sentences = [sentences[idx] for idx in ordering]
    ordering, reordered_sentences

'''
ordering = reorder_sentences(sentences)
reordered_sentences = [sentences[idx] for idx in ordering]
ordering, reordered_sentences
'''<jupyter_output><empty_output><jupyter_text>ExampleThis example shows how the text-reodered works over a medium long non-ordered sentences. Correct order is [2, 0, 1, 3, 5, 4], but it found another that fits well [2, 0, 1, 5, 4, 3]. Let's read the text to get a proof.<jupyter_code># 찐 test
for sentences in all_sentences:
    ordering = reorder_sentences(sentences)
    reordered_sentences = [sentences[idx] for idx in ordering]
    ordering, reordered_sentences

'''
sentences =  [
    "famous thanks to the victories obtained during the first campaign of Italy, after the coup d'etat of the 18th brumaire (9 November 1799) he assumed power in france: he was first consul from November of that year to 18 May 1804, and emperor of the French, with the name of <NAME> (<NAME>) from 2 December 1804 to 14 April 1814 and again from 20 March to 22 June 1815 .",
    "thanks to his system of alliances and a series of brilliant victories against European powers, he conquered and governed a large part of continental Europe, exporting the revolutionary ideals of social renewal and managing to control numerous kingdoms through people loyal to him (giuseppe bonaparte in spain , joachim murat in the kingdom of naples, girolamo bonaparte in westphalia, jean-baptiste jules bernadotte in the kingdom of sweden and luigi bonaparte in the kingdom of holland) .",
    "<NAME> (ajaccio, 15 august 1769 - longwood, sant'elena island, 5 may 1821) was a french politician and general, founder of the first french empire and protagonist of the first phase of european contemporary history called napoleonic age .",
    "great man of war, protagonist of over twenty years of campaigns in europe, napoleon was considered the greatest strategist in history by the military historian basil liddell hart, while the historian evgenij tàrle does not hesitate to define him `` the incomparable master of art of the war `` is `` the greatest of the great '' .",
    "in march 1815, stealthily abandoned the island, he landed in Golfe Juan, near Antibes and returned to Paris without encountering opposition, regaining power for the so-called << one hundred days >> period, until he was definitively defeated by the seventh coalition in the battle of Waterloo, 18 June 1815; he spent the last years of his life in exile on the island of Saint Helena, under the control of the British .",
    "napoleon was defeated in the battle of Leipzig by the European allies in October 1813 and he abdicated on April 4, 1814, and was exiled to the island of Elba : it marked the decline of his dominion over Europe .",
]
ordering = reorder_sentences(sentences)
reordered_sentences = [sentences[idx] for idx in ordering]
ordering, reordered_sentences
'''
from tensorflow.python.saved_model import tag_constants

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
    # Restore from checkpoint
    loader = tf.train.import_meta_graph('./albert/model_ckpt/albert_base.meta')
    loader.restore(sess, tf.train.latest_checkpoint('./albert/model_ckpt/'))

    # Export checkpoint to SavedModel
    builder = tf.saved_model.builder.SavedModelBuilder('./albert/saved_model')
    builder.add_meta_graph_and_variables(sess,
                                         [],
                                         strip_default_attrs=True)
    builder.save()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06884272997032641, 'fraction_numerical': 0.01483679525222552, 'mean_word_length': 4.636008918617614, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '242279', 'license': 'MIT', 'n_tokens_mistral': 2921, 'n_tokens_neox': 2755, 'n_words': 1122, 'path': 'textReorderer/text-reorderer.ipynb', 'repo_name': 'hyejinhyun/NLPG_bert', 'size': '10132'}