<jupyter_start><jupyter_text>Apply length trackerThis demonstrates the use of a length tracker that, for each conversation in a dataset, attaches a marker to the metadata that indicates which  out of a conversation an utterance is, as well as how far through the conversation this utterance is. This is indexed as a list in the field utterance.meta['length_tracker']. The list consists of [position_of_utterance_in_conversation, percentage_of_conversation']For example, let's imagine a conversation of 100 utterances that begins with Person 1 saying "Hello, how are you?" and Person 2  responding with "Good, thanks. How are you?"Here is what the metadata would look like:* "Hello, how are you?"    * Metadata: [0, 0.0] - This tells us the utterance above is the first utterance in a conversation, and that exactly 0% of the conversation has occurred by the time this utterance is said.* "Good thanks. How are you?"  * Metadata: [1, 0.01] - This tells us the utterance above is the 2nd utterance in a conversation, and the exactly 1% of the conversation has occurred at the time this utterance is said.      This transformer allows anyone to easily map information or trends across time.<jupyter_code># import required modules and set up environment
import os

# replace file path below with your own local convokit
os.chdir('/Users/marianneaubin/Documents/Classes/CS6742/Cornell-Conversational-Analysis-Toolkit')
import convokit

from convokit import Corpus, Parser, LengthTracker, Transformer
import nltk
# open corpus
iq2 = convokit.Corpus(filename='datasets/iq2_corpus/iq2_corpus')
# print basic info about the corpus
iq2.print_summary_stats()
lt = LengthTracker();
iq2_length = lt.transform(iq2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06242774566473988, 'fraction_numerical': 0.014450867052023121, 'mean_word_length': 5.294545454545455, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '267859', 'license': 'MIT', 'n_tokens_mistral': 486, 'n_tokens_neox': 459, 'n_words': 225, 'path': 'datasets/iq2_corpus/apply_length_tracker.ipynb', 'repo_name': 'maubinle/Cornell-Conversational-Analysis-Toolkit', 'size': '1730'}