<jupyter_start><jupyter_text>WikiConv Create ConversationsThis notebook provides different forms of conversations from  the Wikiconv data set. In particular, it showcases the final version of selected conversations and how that conversation developed over time. It also provides a framework to print out rnadomly selected final conversations and the corresponding wikipedia page.<jupyter_code>#import relevant modules
from datetime import datetime, timedelta
from convokit import Corpus, User, Utterance, Conversation
import re
import random
#Final Corpus path  is where the corpus data is stored
corpus_path = "/Users/adityajha/Desktop/Cornell-Conversational-Analysis-Toolkit-master/second_set/conversation_corpus_year_2015"
wikiconv_corpus = Corpus(filename=corpus_path)<jupyter_output><empty_output><jupyter_text>Some basic facts about this subset of the corpus: 10,824 conversations and 22,007 utterances<jupyter_code>len(list(wikiconv_corpus.iter_conversations()))
len(list(wikiconv_corpus.iter_utterances()))<jupyter_output><empty_output><jupyter_text>From the corpus, we will randomly select conversations to print out based on a few metrics:1. number_of_conversations - how many conversations we want to print out2. conversation_min_length -  the minimum number of utterances we want in the conversation<jupyter_code>#Randomly chooses the set number of conversations to print from the entire conversaton set
def print_random_conversations(conversation_list, number_of_conversations, conversation_min_length,  conversation_corpus): 
    randomly_generated_conversation_list = []
    while (len(randomly_generated_conversation_list) != number_of_conversations):
        new_conversation = random.randint(0, (len(conversation_list)-1))
        new_conversation_id = conversation_list[new_conversation]
        conversation_ids_list = new_conversation_id.get_utterance_ids()
        if (new_conversation not in randomly_generated_conversation_list 
            and (len(conversation_ids_list) >= conversation_min_length)):
            randomly_generated_conversation_list.append(new_conversation_id)
        
    return randomly_generated_conversation_list<jupyter_output><empty_output><jupyter_text>Here we'll get a set of random conversatinos from the corpus based on our specifications (print out 3, conversations, with a minimum length of 2) and the output will be a set of serialized conversations.<jupyter_code>conversation_list = list(wikiconv_corpus.iter_conversations())
number_of_conversations_to_print = 3
conversation_min_length = 2

random_conversations = print_random_conversations(conversation_list, number_of_conversations_to_print,
                                                     conversation_min_length, wikiconv_corpus)
print (random_conversations)<jupyter_output>[<convokit.model.Conversation object at 0x145baca20>, <convokit.model.Conversation object at 0x145e19550>, <convokit.model.Conversation object at 0x145d0a5c0>]<jupyter_text>Next, stored in the conversation meta data is the wikipedia information from the page that this conversation is from.We will find that information and print out the link to the associated wikipedia page for each conversation.<jupyter_code>def wikipedia_link_info(conversation):
    page_title = conversation.meta['page_title']
    page_title = re.sub('\s+', '_', page_title)
    page_type = conversation.meta['page_type']
    link_value = "https://en.wikipedia.org/w/index.php?title="+page_type+":"+page_title
    
    return link_value

for conversation in random_conversations:
    print(wikipedia_link_info(conversation))
    conversation_ids_list = conversation.get_utterance_ids()<jupyter_output>https://en.wikipedia.org/w/index.php?title=user_talk:172.16.31.10
https://en.wikipedia.org/w/index.php?title=user_talk:172.16.17.32
https://en.wikipedia.org/w/index.php?title=user_talk:Starke~enwiki<jupyter_text>Now that we have the conversation and the actual wikipedia page where they exist, we will want to print out the conversation's final form from the utterance data. But to do this, first we will need to compute the correct order of the comments. The corpus functionality does not guarantee the comments are in the right order, so we will compute this flow now.<jupyter_code>#For any comments that do not have matching reply to ids, sort these comments in order of recency 
def sort_by_timestamp(conversation_ids_list, conversation_corpus):
    list_of_utterances = []
    for id_val in conversation_ids_list:
        utterance_value = conversation_corpus.get_utterance(id_val)
        timestamp_val = utterance_value.timestamp
        tuple_val = (id_val, timestamp_val)
        list_of_utterances.append(tuple_val)

    sorted_utterance_list = sorted(list_of_utterances, key = lambda x:x[1])
    sorted_utterance_list.reverse()
    id_list = [i[0] for i in sorted_utterance_list]
    return (id_list)
#Find cases in which an utterance's reply to is to a comment in the chain that has been modified, deleted or restored
def check_lists_for_match(x, conversation_ids_list, utterance, next_utterance_value, conversation_corpus):
    modification_list = utterance.meta['modification']
    deletion_list = utterance.meta['deletion']
    restoration_list = utterance.meta['restoration']
    if (len(modification_list)>0):
        for utterance_val in modification_list:
            if (utterance_val.id == next_utterance_value.reply_to):
                conversation_ids_list.insert(x+1, next_utterance_value.id)
    if (len(deletion_list)>0):
        for utterance_val in deletion_list:
            if (utterance_val.id == next_utterance_value.reply_to):
                conversation_ids_list.insert(x+1, next_utterance_value.id)
    if (len(restoration_list)>0):
        for utterance_val in restoration_list:
            if (utterance_val.id == next_utterance_value.reply_to):
                conversation_ids_list.insert(x+1, next_utterance_value.id)
# Build the conversation flow correctly and add utterances if the reply-to id matches the current utterance in the list
def add_utterance(conversation_ids_list, next_utterance_value, conversation_corpus):
    if next_utterance_value.id in conversation_ids_list:
        return conversation_ids_list
    elif (next_utterance_value.reply_to is None):
        conversation_ids_list.append(next_utterance_value.id)
    else:
        for x in range(0,len(conversation_ids_list)):
            utterance_id = conversation_ids_list[x]
            if (utterance_id == next_utterance_value.reply_to):
                conversation_ids_list.insert(x+1, next_utterance_value.id)
            else:
                check_lists_for_match(x, conversation_ids_list, conversation_corpus.get_utterance(utterance_id), next_utterance_value, conversation_corpus)

    return conversation_ids_list
#The order of the returned conversation ids is not guaranteed; compute the correct ordering 
def find_correct_order(conversation_ids_list, conversation_corpus):
    correct_list_order = []
    #if the conversation has only one comment, return the conversation list
    if (len(conversation_ids_list) == 1 ):
        return conversation_ids_list

    #When the conversation has more than one comment, find the correct order of the comments
    if (len(conversation_ids_list) >1):
        #Implement a fail safe to efficiently sort 
        number_of_iterations = 0
        while (number_of_iterations <20 and len(correct_list_order) != len(conversation_ids_list)):
            for utterance_id in conversation_ids_list:
                correct_list_order = add_utterance(correct_list_order, conversation_corpus.get_utterance(utterance_id), conversation_corpus)
            number_of_iterations+=1

        #In some of the conversations, new utterances will be added that don't reply directly to the current conversation
        #Instead, these new utterances are part of the topic at hand (under the same conversation root) and are sorted by recency
        if (len(correct_list_order) != len(conversation_ids_list)):
            difference_in_sets = set(conversation_ids_list).difference(correct_list_order)
            timestamp_sorted_difference = sort_by_timestamp(list(difference_in_sets), conversation_corpus)
            correct_list_order.extend(timestamp_sorted_difference)
    return correct_list_order<jupyter_output><empty_output><jupyter_text>And so, we can compute the correct order of utterances in each randomly selected conversation.<jupyter_code>for conversation in random_conversations:
    conversation_ids_list = conversation.get_utterance_ids()
    print ('Original Order of IDs:' + str(conversation_ids_list))
    print('Correct Order of IDs:' + str(find_correct_order(conversation_ids_list, wikiconv_corpus)))
    print ('\n')<jupyter_output>Original Order of IDs:['693119069.2.10986', '688099454.12316.12316', '693119069.2.10967', '693119069.2.11819']
Correct Order of IDs:['693119069.2.10986', '693119069.2.10967', '693119069.2.11819', '688099454.12316.12316']


Original Order of IDs:['693119688.20.0', '693156840.1278.1278', '693119688.0.0']
Correct Order of IDs:['693119688.0.0', '693156840.1278.1278', '693119688.20.0']


Original Order of IDs:['652619734.0.0', '652619734.35.0']
Correct Order of IDs:['652619734.0.0', '652619734.35.0']<jupyter_text>Print out the final form of the conversations<jupyter_code>#Print the conversation text from the list of conversation ids
def print_final_conversation(random_conversations, conversation_corpus):
    for conversation in random_conversations:
        print(wikipedia_link_info(conversation))
        conversation_ids_list = conversation.get_utterance_ids()
        #First correctly reorder the comments
        ordered_list = find_correct_order(conversation_ids_list, conversation_corpus)
        #For each utterance, print the text present if the utterance has not been deleted
        for utterance_id in ordered_list:
            utterance_value = conversation_corpus.get_utterance(utterance_id)
            if (utterance_value.text != " "):
                print (utterance_value.text)
                date_time_val = datetime.fromtimestamp(utterance_value.timestamp).strftime('%H:%M %d-%m-%Y')
                formatted_user_name = "--" + str(utterance_value.user.name) + "  " + str(date_time_val)
                print (formatted_user_name)
        print ('\n\n')
print_final_conversation(random_conversations,  wikiconv_corpus)<jupyter_output>https://en.wikipedia.org/w/index.php?title=user_talk:172.16.31.10
 Please refrain from making unconstructive edits to Wikipedia, as you did at North Stafford High School. Your edits appear to constitute vandalism and have been reverted or removed. If you would like to experiment, please use the sandbox. Repeated vandalism can result in the loss of editing privileges. Thank you. ''►  ''' & ''' ◄''
''If this is a shared IP address, and you did not make the edits, consider creating an account for yourself so you can avoid further irrelevant notices.''
--Bgpaulus  13:03 29-10-2015



https://en.wikipedia.org/w/index.php?title=user_talk:172.16.17.32
 November 2015 
--ClueBot NG  10:21 30-11-2015
 Please refrain from making unconstructive edits to Wikipedia, as you did at Arranged marriage in the Indian subcontinent. Your edits continue to appear to constitute vandalism and have been '''automatically''' reverted.
 If you would like to experiment, please use the sandbox. Note that human [...]<jupyter_text>Let's create a compact method to change the default values easily<jupyter_code>def change_defaults_print_final(conversation_list, number_of_conversations, conversation_min_length,  
                                conversation_corpus):
    random_conversations = print_random_conversations(conversation_list, number_of_conversations_to_print,
                                                     conversation_min_length, wikiconv_corpus)
    print_final_conversation(random_conversations, conversation_corpus)
conversation_list = list(wikiconv_corpus.iter_conversations())
number_of_conversations_to_print = 1
conversation_min_length = 2
#Refresher on where the wikiconv_corpus  is defined
# corpus_path = "/Users/adityajha/Desktop/Cornell-Conversational-Analysis-Toolkit-master/second_set/conversation_corpus_year_2015"
# wikiconv_corpus = Corpus(filename=corpus_path)

change_defaults_print_final(conversation_list, number_of_conversations_to_print, conversation_min_length,
                            wikiconv_corpus)<jupyter_output>https://en.wikipedia.org/w/index.php?title=user_talk:GiantSnowman<jupyter_text>Finally, we will create a method to print out the final comment and the intermediate steps in the conversation<jupyter_code>def sort_changes_by_timestamp(modification_list, deletion_list, restoration_list,  original_utterance):
    text_time_tuple_list = []
    if (original_utterance is not None):
        text_time_original  = (original_utterance.text,original_utterance.timestamp,
                           original_utterance.user.name, 'original')
        text_time_tuple_list.append(text_time_original)
        

    for utterance in modification_list:
        text_time= (utterance.text, utterance.timestamp,
                    utterance.user.name, 'modification')
        text_time_tuple_list.append(text_time)
    
    for utterance in deletion_list:
        text_time= ('', utterance.timestamp,
                    utterance.user.name, 'deletion')
        text_time_tuple_list.append(text_time)
        
    for utterance in restoration_list:
        text_time= (utterance.text, utterance.timestamp,
                    utterance.user.name, 'restoration')
        text_time_tuple_list.append(text_time)
            
    text_time_tuple_list.sort(key=lambda x: x[1])
    #text_time_tuple_list.reverse()
    
    
    
    return text_time_tuple_list
        
    
def print_intermediate_conversation(random_conversations, conversation_corpus):
    for conversation in random_conversations:
        conversation_ids_list = conversation.get_utterance_ids()
        #First correctly reorder the comments
        ordered_list = find_correct_order(conversation_ids_list, conversation_corpus)
        #For each utterance, print the text present if the utterance has not been deleted
        for utterance_id in ordered_list:
            utterance_value = conversation_corpus.get_utterance(utterance_id)
            if (utterance_value.text != " "):
                final_comment =  utterance_value.text
                date_time_val = datetime.fromtimestamp(utterance_value.timestamp).strftime('%H:%M %d-%m-%Y')
                formatted_user_name = "--" + str(utterance_value.user.name) + "  " + str(date_time_val)
                
        
                final_timestamp = utterance_value.timestamp
                modification_list = utterance_value.meta['modification']
                deletion_list = utterance_value.meta['deletion']
                restoration_list = utterance_value.meta['restoration']
                
                sorted_timestamps = sort_changes_by_timestamp(modification_list, deletion_list, restoration_list,
                                                             utterance_value.meta['original'])
                
                if (len(sorted_timestamps)>0):
                    print(wikipedia_link_info(conversation))
                    print ('Final Comment')
                    print (final_comment)
                    print (formatted_user_name)
                    
                    for value in sorted_timestamps:
                        print ('\n')
                        print (value[3])
                        print (value[0])
                        formatted_user_name = "--" + str(value[2]) + "  " + str(datetime.fromtimestamp(float(value[1])).strftime('%H:%M %d-%m-%Y'))
                        #str(datetime.fromtimestamp(value[1]).strftime('%H:%M %d-%m-%Y'))
                        print (formatted_user_name)<jupyter_output><empty_output><jupyter_text>Our method to quikcly print out intermediate conversations defined below (only conversations with modification, deletion and restoration conversations  are shown)<jupyter_code>def change_defaults_print_intermediate(conversation_list, number_of_conversations, conversation_min_length,  
                                conversation_corpus):
    random_conversations = print_random_conversations(conversation_list, number_of_conversations_to_print,
                                                     conversation_min_length, wikiconv_corpus)
    print_intermediate_conversation(random_conversations, conversation_corpus)<jupyter_output><empty_output><jupyter_text>Here, the flow of different conversations  is shown with the final comment first displayed and the corresponding actions that have occurred from earliest to latest actions<jupyter_code>conversation_list = list(wikiconv_corpus.iter_conversations())
number_of_conversations_to_print = 10
conversation_min_length = 3
#Refresher on where the wikiconv_corpus  is defined
# corpus_path = "/Users/adityajha/Desktop/Cornell-Conversational-Analysis-Toolkit-master/second_set/conversation_corpus_year_2015"
# wikiconv_corpus = Corpus(filename=corpus_path)

change_defaults_print_intermediate(conversation_list, number_of_conversations_to_print, conversation_min_length,
                            wikiconv_corpus)<jupyter_output>https://en.wikipedia.org/w/index.php?title=user_talk:172.16.58.3
Final Comment
 Please stop adding inappropriate external links to Wikipedia. It is considered spamming and Wikipedia is not a vehicle for advertising or promotion. Because Wikipedia uses nofollow tags, additions of links to Wikipedia will not alter search engine rankings. If you continue spamming, you may be blocked from editing Wikipedia.   ''''  
--Deli nk  12:05 03-05-2015


original
 Please stop your disruptive editing. If you continue to vandalize Wikipedia, you may be blocked from editing. ''SPAM: ''  
''If this is a shared IP address, and you did not make the edits, consider creating an account for yourself so you can avoid further irrelevant notices.''
--Deli nk  12:04 03-05-2015


modification
 Please stop adding inappropriate external links to Wikipedia. It is considered spamming and Wikipedia is not a vehicle for advertising or promotion. Because Wikipedia uses nofollow tags, additions of links to Wikipedia [...]
{'fraction_non_alphanumeric': 0.07202743902439024, 'fraction_numerical': 0.023410278745644598, 'mean_word_length': 3.9929328621908127, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 52, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 5, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '376638', 'license': 'MIT', 'n_tokens_mistral': 5520, 'n_tokens_neox': 5171, 'n_words': 1322, 'path': 'examples/dataset-examples/wikiconv/Create_Conversations_Script.ipynb', 'repo_name': 'sophieball/Cornell-Conversational-Analysis-Toolkit', 'size': '18381'}