<jupyter_start><jupyter_text>Word probabilities Following https://arxiv.org/pdf/1906.00363.pdf, we try to compute the predictability score of a sentence based on the probabilities of its component tokens. We use a language model trained on a large corpora of literary text and want to use the predictability score as a proxy for literary "inventivity": i.e. if the model can predict the various tokens in the sentence with high probability, then the sentence must be hackeneyed or cliché. The formula given in the paper is:$(p_{1}*p_{2}...p_{n})^{-1/n} =(\prod_{i=1}^{n}(p_{i} | sentence))^{-1/n} $  Initializing First we'll define our preprocessing function (applied prior to the model tokenizer for better handling of korean morphosyntactical features), then we'll load our model pre-trained on North Korean data<jupyter_code>from konlpy.tag import Komoran
komoran = Komoran()
def preproc(sentence):
    return ' '.join([_ for _ in komoran.morphs(sentence)]).replace('MASK', '[MASK]')
from transformers import AutoModelWithLMHead, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("./jobert")
model = AutoModelWithLMHead.from_pretrained("./jobert")
import logging
logging.getLogger("transformers.tokenization_utils").setLevel(logging.ERROR)<jupyter_output><empty_output><jupyter_text>Function to compute the probability of a single word with BERT's masked language modelling<jupyter_code>def compute_word_proba(sequence, word):
    global model, tokenizer
    input_ids = tokenizer.encode(sequence, return_tensors="pt")
    mask_token_index = torch.where(input_ids == tokenizer.mask_token_id)[1]
    token_logits = model(input_ids)[0]
    mask_token_logits = token_logits[0, mask_token_index, :]
    mask_token_logits = torch.softmax(mask_token_logits, dim=1)
    sought_after_token = word
    sought_after_token_id = tokenizer.encode(sought_after_token, add_special_tokens=False, add_prefix_space=True)[0]
    token_score = mask_token_logits[:, sought_after_token_id]
    return token_score.detach().numpy()[0]<jupyter_output><empty_output><jupyter_text>Extended to compute the probability of each word in a sequence<jupyter_code>def compute_word_by_word_proba(sequence):
    global tokenizer
    preprocessed_sequence = preproc(sequence)
    word_dict = {}
    for token in preprocessed_sequence.split(' '):
        masked_sequence = preprocessed_sequence.replace(token, tokenizer.mask_token)
        word_dict[token] = compute_word_proba(masked_sequence, token)
    return word_dict
compute_word_by_word_proba('아무것도  가늠할수  없도록  몸과  마음이  굳어져버 린속에  한가지  생각이  끌날처럼  머리속으로  비껴  들었다')<jupyter_output><empty_output><jupyter_text>And averaging out the individual probabilities for a sentence score:<jupyter_code>import numpy as np

def geometric_mean(series):
    return np.array(series).prod()**(1.0/len(series))

def compute_sentence_score(sentence):
    return geometric_mean(list(compute_word_by_word_proba(sentence).values()))
    
compute_sentence_score('아무것도  가늠할수  없도록  몸과  마음이  굳어져버 린속에  한가지  생각이  끌날처럼  머리속으로  비껴  들었다')<jupyter_output><empty_output><jupyter_text>Running a test with a sentence full of cliches and one with a creative metaphor:<jupyter_code>cliche = '어둠속에  빛나는  그윽한  눈동자 에  웬  불빛이  어리여  불꽃처럼  아름답게  반짝이 고있었다.'
original = '단풍잎같은 손이 물을 연신 퍼서 뿌려댄다.'
print('Cliche sentence score: ', compute_sentence_score(cliche))
print('Original sentence score: ', compute_sentence_score(original))<jupyter_output>Cliche sentence score:  0.03292289714058116
Original sentence score:  0.015907948372126945
{'fraction_non_alphanumeric': 0.08923512747875353, 'fraction_numerical': 0.01671388101983003, 'mean_word_length': 5.896484375, 'pattern_counts': {'":': 1, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '114003', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1305, 'n_tokens_neox': 1334, 'n_words': 302, 'path': 'language_model/Word & Sentence Probabilities.ipynb', 'repo_name': 'benber86/nkbert', 'size': '3530'}