<jupyter_start><jupyter_text>Machine Learning Nanodegree Capstone Chat Room Recommendation<jupyter_code>import os
import collections
import random
import numpy as np
from gensim.models import Doc2Vec<jupyter_output>Using TensorFlow backend.<jupyter_text>Load Data<jupyter_code># path will need to be changed pending on where the repo is cloned to
os.chdir(os.path.expanduser('~/PycharmProjects/chat-room-recommendation/'))
lines = open('cornell-movie-dialogs-corpus/movie_lines.txt','r').read().split('\n')
conv_lines = open('cornell-movie-dialogs-corpus/movie_conversations.txt','r').read().split('\n')
character_metadata = open('cornell-movie-dialogs-corpus/movie_characters_metadata.txt','r').read().split('\n')
movie_metadata = open('cornell-movie-dialogs-corpus/movie_titles_metadata.txt','r').read().split('\n')
lines[:10]
conv_lines[:10]<jupyter_output><empty_output><jupyter_text>Use gensim simple pre-processing tool to create a dictionary with keys = movie_id and value = tokenized text of all the lines in the movie.<jupyter_code>from gensim import utils
movieLines = {}
for line in lines:
    _line = line.split(' +++$+++ ')
    if len(_line) == 5:
        if _line[2] in movieLines: 
            movieLines[_line[2]] = movieLines.get(_line[2]) + utils.simple_preprocess(_line[4])
        else:
            movieLines[_line[2]] = utils.simple_preprocess(_line[4])
# sanity check that there are the appropriate number of movies in the movieLines dict
print "Correct # of movies" if len(movieLines) == 617 else "something went wrong with movieLines dict"
# print movieLines.get("m616")
# find average length of tokenized text for the movies.
totalLines = 0
for key, value in movieLines.iteritems():
    totalLines = totalLines + len(value)
    
totalLines/len(movieLines)<jupyter_output><empty_output><jupyter_text>Create a dictionary to map each line's id with it's text<jupyter_code>id2line = {}
for line in lines:
    _line = line.split(' +++$+++ ')
    if len(_line) == 5:
        id2line[_line[0]] = _line[4]
# sanity check for id2line dict
print "Correct # of lines" if len(id2line) == 304713 else "something went wrong with id2line dict"<jupyter_output>Correct # of lines<jupyter_text>Use gensim simple pre-processing tool to create a dictionary with keys = character_id and value = tokenized text of all conversations for that character.<jupyter_code>characterConversations = {}
for line in conv_lines:
    _line = line.split(' +++$+++ ')
    if len(_line) == 4:
        _line[3] = _line[3].strip("[]")
        for conv in _line[3].split(","):
            conv = conv.replace("'","").replace(" ", "")
            if _line[0] in characterConversations:
                characterConversations[_line[0]] = characterConversations.get(_line[0]) + \
                                                   utils.simple_preprocess(id2line.get(str(conv)))
            else:
                characterConversations[_line[0]] = utils.simple_preprocess(id2line.get(str(conv)))

            if _line[1] in characterConversations:
                characterConversations[_line[1]] = characterConversations.get(_line[1]) + \
                                                   utils.simple_preprocess(id2line.get(str(conv)))
            else:
                characterConversations[_line[1]] = utils.simple_preprocess(id2line.get(str(conv)))
# sanity check that there are the appropriate # of characters in the characterConversation dict
print "Correct # of characters" if len(characterConversations) == 9035 else "something went wrong with character dict"
#print characterConversations.get("u0")
# find average length of tokenized text for each character.
totalLines = 0
for key, value in characterConversations.iteritems():
    totalLines = totalLines + len(value)
    
totalLines/len(characterConversations)
# function that creates corpus (list of TaggedDocmuments) from dictionaries
from gensim.models.doc2vec import TaggedDocument


def create_corpus(dictname):
    corpus_list =[]
    for key, value in dictname.iteritems():
        corpus_list.append(TaggedDocument(value, [int(key[1:])]))
    return corpus_list
train_corpus = create_corpus(movieLines)
test_corpus = create_corpus(characterConversations)
# sanity check of length of corpus
print "Correct # of movies in train_corpus" if len(train_corpus) == 617 else "something went wrong with train_corpus"
print "Correct # of characters in test_corpus" if len(test_corpus) == 9035 else "something went wrong with test_corpus"
# print train_corpus[0].tags<jupyter_output>Correct # of movies in train_corpus
Correct # of characters in test_corpus<jupyter_text>Instantiate a Doc2Vec Object<jupyter_code>from gensim.models import Doc2Vec

model = Doc2Vec(size=50, iter=20, min_count=2)
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
# load model if saved during a previous session
from gensim.models import Doc2Vec

model = Doc2Vec.load('/tmp/movie_model.doc2vec')<jupyter_output><empty_output><jupyter_text>Assessing ModelTo assess the doc2vec model, inferring new vectors for each document of the training corpus, compare the inferred vectors with the training corpus, and then returning the rank of the document based on self-similarity. This approach is pretending as if the training corpus is some new unseen data and then seeing how they compare with the trained model. The expectation is that model will be overfit and so finding similar documents should be very easily. The second ranks will also be tracked for comparison of less similar docutents. (https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-lee.ipynb)<jupyter_code>ranks = []
first_ranks = []
second_ranks = []
for doc_id in range(len(train_corpus)):

    inferred_vector = model.infer_vector(train_corpus[doc_id].words)
    sims = model.docvecs.most_similar([inferred_vector], topn=len(model.docvecs))
    
    rank = [docid for docid, sim in sims].index(train_corpus[doc_id].tags[0])
        
    ranks.append(rank) 
    
    first_ranks.append(sims[0])
    second_ranks.append(sims[1])
collections.Counter(ranks)<jupyter_output><empty_output><jupyter_text>Utility functions<jupyter_code># get character metadata by id
def get_character_metadata(id):
    for character in character_metadata:
        _character = character.split(' +++$+++ ')
        if len(_character) == 6:
            if _character[0] == 'u'+str(id):
                return _character
        
           
def get_movie_title(id):
    for movie in movie_metadata:
        _movie = movie.split(' +++$+++ ')
        if len(_movie) == 6:
            if _movie[0] == 'm'+str(id):
                return _movie[1]
            
            
# create def to look up tag doc in corpus by tag id
def get_corpus_index(corpus, tag):
    for tag_index in range(len(corpus)):
        if corpus[tag_index].tags[0] == tag:
            return tag_index
     
     
def get_movie_genres(id):
    for movie in movie_metadata:
        _movie = movie.split(' +++$+++ ')
        if len(_movie) == 6:
            if _movie[0] == 'm'+str(id):
                return _movie[5]<jupyter_output><empty_output><jupyter_text>Testing the modelMean Average Precision (MAP) will be used to test the model.<jupyter_code># find similarity of movie character is from and index 
def get_source_similarity(sim_list, movie_id):
    for sim_index in range(len(sim_list)):
        if int(movie_id[1:]) == sim_list[sim_index][0]:
            return get_movie_title(sim_list[sim_index][0]), sim_index, sim_list[sim_index][1]
        
        
# print top 5 recommended movies
def get_recommended_movies(sim_list):
    print('\nTop 5\n {}\n {}\n {}\n {}\n {}\n'.format(get_movie_title(sim_list[0][0]), get_movie_title(sim_list[1][0]), 
                                                get_movie_title(sim_list[2][0]), get_movie_title(sim_list[3][0]),
                                                get_movie_title(sim_list[4][0])))
    
    
def display_character_similarity(char_id, show_words):
    inferred_vector = model.infer_vector(test_corpus[char_id].words)
    test_sims = model.docvecs.most_similar([inferred_vector], topn=len(model.docvecs))
    character_metadata = get_character_metadata(char_id)
    print('Character : {}\n ID : {}\n Movie : {}'.format(character_metadata[1], character_metadata[2], 
                                                         character_metadata[3]))
    if show_words :
        print('Test Document ({}): «{}»\n'.format(char_id, ' '.join(test_corpus[get_corpus_index(test_corpus, char_id)].words)))
    print test_sims[:10]
    get_recommended_movies(test_sims)
    print get_source_similarity(test_sims, character_metadata[2]), '\n'
    
    
def get_doc2vec_similarity(char_id, test_model):
    inferred_vector = test_model.infer_vector(test_corpus[char_id].words)
    test_sims = test_model.docvecs.most_similar([inferred_vector], topn=len(test_model.docvecs))

    return test_sims[:20]


def precision_at_k(r, k):    
    assert k >= 1
    r = np.asarray(r)[:k] != 0
    if r.size != k:
        raise ValueError('Relevance score length < k')
    return np.mean(r)


def calc_average_precision(recommendations, movie_id):
    metadata = get_character_metadata(movie_id)
    genres_str = get_movie_genres(metadata[2][1:])
    # print genres_str
    
    genres_str = genres_str.strip("[]").replace("'","").replace(" ", "")
    genre_list = genres_str.split(",")  
    predicted_list = []
    for item in recommendations:
        # print get_movie_genres(item[0])
        is_relevant = False
        for gen in genre_list:
            if gen in get_movie_genres(item[0]):
                if is_relevant is False:
                    # print "+"
                    is_relevant = True
        if is_relevant:
            predicted_list.append(1)
        else:
            predicted_list.append(0)
            
    # print predicted_list
    
    predicted_list = np.asarray(predicted_list) != 0
    out = [precision_at_k(predicted_list, k + 1) for k in range(predicted_list.size) if predicted_list[k]]
    if not out:
        return 0.
    return np.mean(out)


def calc_mean_average_precision_k(test_list, test_model):
    av_list = []
    for char in test_list:
        d2v_result = get_doc2vec_similarity(char, test_model)
        av_list.append(calc_average_precision(d2v_result, char))
    
    return np.mean(av_list)
# Generate random test data to be used - Of the 9035 character 500 will be randomly added to a list for testing
test_character_list = []
for _ in range(500):
    rand_id = random.randint(0, len(test_corpus))
    test_character_list.append(rand_id)
calc_mean_average_precision_k(test_character_list[:200], model)
import matplotlib.pyplot as plt

test_sizes_list = [10, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500]
mapk_results = []
for test_size in test_sizes_list:
    mapk_results.append(calc_mean_average_precision_k(test_character_list[:test_size], model))
    

plt.plot(test_sizes_list, mapk_results)
plt.axis([0, 500, 0, 1])
plt.show()
# final 4
model = Doc2Vec(alpha=0.25, min_alpha=0.025) 
model.build_vocab(train_corpus)
# %time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
for epoch in range(10):
    model.train(train_corpus, total_examples=model.corpus_count, epochs=epoch)
    model.alpha -= 0.002  # decrease the learning rate
    model.min_alpha = model.alpha  # fix the learning rate, no decay
    
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=10) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=20) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=300, window=8, min_count=19, iter=20) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=20, min_count=19, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
# final 4
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=20, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=10, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=100, alpha=0.25, min_alpha=0.01) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
# final 4
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=150) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
print model
# final 4
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=200) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=300) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(test_character_list[:200], model)
# try to run against entire character test
character_list = []
for id in range(len(test_corpus)):
    character_list.append(id)

model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) 
model.build_vocab(train_corpus)
%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
print calc_mean_average_precision_k(character_list, model)
# save model to be loaded later if needed
model.save('/tmp/movie_model.doc2vec')<jupyter_output><empty_output><jupyter_text>Test ResultsRound 1Doc2Vec(size=50, iter=20, min_count=2) : 0.64590552578393612Doc2Vec(alpha=0.025, min_alpha=0.025)/decrease learning rate : 0.652129159832Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=10) : 0.643467036911Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=20) : 0.641963598382Doc2Vec(dm=1, dm_mean=1, size=300, window=8, min_count=19, iter=20) : 0.649278035546Doc2Vec(dm=1, dm_mean=1, size=200, window=8, min_count=19, iter=100) : 0.635627557614Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.664204807617Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=20, iter=100) : 0.639810561736Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=10, iter=100) : 0.641714744505Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=100) : 0.645888DD061985Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=100, alpha=0.25, min_alpha=0.01) : 0.64739936922Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=150) : 0.6501950632Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=200) : 0.653454281673Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=300) : 0.644189477281Round 2 - new random test dataDoc2Vec(alpha=0.025, min_alpha=0.025)/decrease learning rate : 0.616014300118Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.644970657008Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=150) : 0.634117843088Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=200) : 0.638209043472Round 3 - new random test dataDoc2Vec(alpha=0.025, min_alpha=0.025)/decrease learning rate : 0.607888950913Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.614642972259Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=150) : 0.619167139798Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=200) : 0.619998446705AverageDoc2Vec(alpha=0.025, min_alpha=0.025)/decrease learning rate : (0.652129159832 + 0.616014300118 + 0.607888950913)/3 = 0.625344137Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : (0.664204807617 + 0.644970657008 + 0.614642972259)/3 = 0.641272812Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=150) : (0.6501950632 + 0.634117843088 + 0.619167139798)/3 = 0.634493349Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=15, iter=200) : (0.653454281673+ 0.638209043472 + 0.619998446705)/3 = 0.637220591Round 4 - new random test data min_count 19 vs min_count 20Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.654732378791Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=20, iter=100) : 0.649219429736Round 5 - same random test data min_count 19 vs min_count 20Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.647298604327 Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=20, iter=100) : 0.647055708097 MAP run with full test corpusDoc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100) : 0.64917898177  Sensitivity analysisTest Doc2Vec model with random lines removed from the training_corpus. The model will be assessed the same way the original training_corpus was assessed.<jupyter_code># altlines = open('cornell-movie-dialogs-corpus/movie_lines_mod.txt','r').read().split('\n')
from random import randint
from gensim import utils

altMovieLines = {}
altTotalLines = 0
for line in lines:
    if randint(0, 9) != 7:
        altTotalLines = altTotalLines+1
        _line = line.split(' +++$+++ ')
        if len(_line) == 5:
            if _line[2] in altMovieLines: 
                altMovieLines[_line[2]] = altMovieLines.get(_line[2]) + utils.simple_preprocess(_line[4])
            else:
                altMovieLines[_line[2]] = utils.simple_preprocess(_line[4])
print "Correct # of movies" if len(altMovieLines) == 617 else "something went wrong with movieLines dict"
print "# of lines were decreased" if altTotalLines < 304713 else "something went wrong with id2line dict"
print altTotalLines
alt_train_corpus = create_corpus(altMovieLines)
alt_model = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100)
alt_model.build_vocab(alt_train_corpus)
%time alt_model.train(alt_train_corpus, total_examples=alt_model.corpus_count, epochs=alt_model.iter)
print calc_mean_average_precision_k(test_character_list[:200], alt_model)
alt_model.save('/tmp/alt_model.doc2vec')
from gensim.models import Doc2Vec

alt_model = Doc2Vec.load('/tmp/alt_model.doc2vec')<jupyter_output><empty_output><jupyter_text>Test Doc2Vec model with test_corpus (individual character converations) which consist of 9035 different documents.  The model will be assessed the same way the training_corpus was assessed.This is essentually recommending users to other users.<jupyter_code># alt MAP functions to use test_corpus to train.  
def calc_average_precision2(recommendations, movie_id):
    metadata = get_character_metadata(movie_id)
    genres_str = get_movie_genres(metadata[2][1:])
    # print genres_str
    
    genres_str = genres_str.strip("[]").replace("'","").replace(" ", "")
    genre_list = genres_str.split(",")  
    predicted_list = []
    for item in recommendations:
        # print get_movie_genres(item[0])
        is_relevant = False
        for gen in genre_list:
            if gen in get_movie_genres(get_character_metadata(item[0])[2][1:]):
                if is_relevant is False:
                    # print "+"
                    is_relevant = True
        if is_relevant:
            predicted_list.append(1)
        else:
            predicted_list.append(0)
            
    # print predicted_list
    
    predicted_list = np.asarray(predicted_list) != 0
    out = [precision_at_k(predicted_list, k + 1) for k in range(predicted_list.size) if predicted_list[k]]
    if not out:
        return 0.
    return np.mean(out)


def calc_mean_average_precision_k2(test_list, test_model):
    av_list = []
    for char in test_list:
        d2v_result = get_doc2vec_similarity(char, test_model)
        av_list.append(calc_average_precision2(d2v_result, char))
    
    return np.mean(av_list)
model2 = Doc2Vec(dm=1, dm_mean=1, size=200, window=4, min_count=19, iter=100)
model2.build_vocab(test_corpus)
%time model2.train(test_corpus, total_examples=model2.corpus_count, epochs=model2.iter)
print calc_mean_average_precision_k2(test_character_list[:200], model2)
model2.save('/tmp/character_model.doc2vec')
from gensim.models import Doc2Vec

model2 = Doc2Vec.load('/tmp/character_model.doc2vec')<jupyter_output><empty_output><jupyter_text>Simple test of the first movie in the training corpus to ensure that the movie was returned as the most similar document by the model.  The similarity score is also displayed. A random document from the training coupus is aslo selected along and compared with the similarity score of the second most similar document.<jupyter_code>print('Movie title : {}\nMovie id : {}\n'.format(get_movie_title(train_corpus[doc_id].tags[0]), train_corpus[doc_id].tags[0]))
print(u'MODEL %s:\n' % model)

print get_movie_title(train_corpus[get_corpus_index(train_corpus, sims[0][0])].tags[0])
print sims[0], '\n\n----------------------\n'

# Pick a random document from the train corpus and infer a vector from the model
rand_train_id = random.randint(0, len(train_corpus))

# Inspect the score of the second ranked movie. 
# The score for the second ranked movie should be much lower then for the top document.
print('Train Document ({}): «{}»\n'.format(rand_train_id, get_movie_title(rand_train_id)))
top_id = first_ranks[get_corpus_index(train_corpus, rand_train_id)]
print('Top Document {}: «{}»\n'.format(top_id, get_movie_title(top_id[0])))
sim_id = second_ranks[get_corpus_index(train_corpus, rand_train_id)]
print('Similar Document {}: «{}»\n'.format(sim_id, get_movie_title(sim_id[0])))
# Two characters from the same movie 
display_character_similarity(1220, False)
print("AP@20 for character 1220 : {}".format(calc_average_precision(get_doc2vec_similarity(1220, model), 1220)))
display_character_similarity(1223, False)
print("AP@20 for character 1223 : {}".format(calc_average_precision(get_doc2vec_similarity(1223, model), 1223)))
# compare results from three characters that have conversations in a movie and a random character
# Star wars : Hans, Luke and Vader
# u7821 +++$+++ HAN +++$+++ m529
display_character_similarity(7821, False)

# u7824 +++$+++ LUKE +++$+++ m529
display_character_similarity(7824, False)

# u7827 +++$+++ VADER
display_character_similarity(7827, True)

# random character from test corpus
display_character_similarity(random.randint(0, len(test_corpus)), False)
# two characters that only have conversations with each other so they have identical words in their document
display_character_similarity(5519, True)
display_character_similarity(5508, True)
# two characters that only have conversations with each other so they have identical words in their document
display_character_similarity(259, True)
display_character_similarity(265, True)<jupyter_output>Character : PAPAGENO
 ID : m16
 Movie : amadeus
Test Document (259): «here am my angel what who the devil are you ve taken pity on you my angel heard your wish oh well thank you how wonderful some people get all the luck now you ve got to promise me faithfully you ll remain true to me forever then you ll see how tenderly your little birdie will love you can wait well promise then what do you mean now of course now right away before get any older well don know mean you re delicious delightful delectable little bird but don you think you might be just little tough oh tender enough for you my boy tender enough for you»

[(369, 0.48146891593933105), (203, 0.3504234254360199), (494, 0.3287377655506134), (424, 0.32663020491600037), (272, 0.3177914023399353), (387, 0.31425905227661133), (613, 0.3100992441177368), (436, 0.3068705201148987), (309, 0.2976357638835907), (581, 0.2970328629016876)]

Top 5
 the godfather: part ii
 the godfather
 ronin
 lord of illusions
 blade

('amadeus', 83, 0.244[...]
{'fraction_non_alphanumeric': 0.11521976309226932, 'fraction_numerical': 0.06748753117206982, 'mean_word_length': 4.929990757855823, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '136167', 'license': 'MIT', 'n_tokens_mistral': 9955, 'n_tokens_neox': 8753, 'n_words': 2002, 'path': 'chatroom_recommendation_capstone.ipynb', 'repo_name': 'tylershumaker/chat-room-recommendation', 'size': '25664'}