<jupyter_start><jupyter_text>Importing necessary libraries<jupyter_code>import pandas as pd
import numpy as np
from nltk.tokenize import RegexpTokenizer
from nltk.tokenize.treebank import TreebankWordDetokenizer
import pickle
import codecs<jupyter_output><empty_output><jupyter_text>Setting up tokenizer<jupyter_code># creating tokenizer that will produce token(specifically 'words' and thats what we desire to have) 
# which would comprise of words. 
# r"[\w'’-]+" is a regex that will extract words (it would even extract words having apostrophe and hyphen, which
# otherwise would not have happened, had the regex been r'\w+')
tokenizer = RegexpTokenizer(r"[\w'’-]+")<jupyter_output><empty_output><jupyter_text>Reading dataset.csv and storing it into the dataframe named 'df'<jupyter_code># setting up the column names for dataframe that will be created after
# reading 'dataset.csv'
colnames=['Snippet', 'Target'] 

# creating dataframe named 'df'
df = pd.read_csv('dataset.csv', names = colnames, header = None)<jupyter_output><empty_output><jupyter_text>Path to embedding file  The embedding file will be used for the creation of word embeddings (transformation of words to vector)<jupyter_code>embedding_file = './crawl-300d-2M.vec'<jupyter_output><empty_output><jupyter_text>A look at the content of 'crawl-300d-2M.vec'   Following function creates embedding index using the embedding file. The embedding index will be later used for transforming words to vector<jupyter_code>def loadEmbedding():
    
    print('Loading Word Embeddings...')
    
    # 'embeddings_index' will maintain word embedding dictionary
    # the format of the dictionary will be word : word-embedding
    embeddings_index = {}
    
    # opening 'embedding_file' in reading mode
    # When you need to open a file that has a certain encoding, you would use 
    # the codecs module, codecs.open
    f = codecs.open(embedding_file, encoding='utf-8')
    
    # for every line in the file that the file pointer 'f' is pointing to,
    # iterate through the body of the for loop
    for line in f:
        
        # for the current 'line' in  'f', split it into tokens by using ' '(space) as
        # the delimiter.
        # also, trim the whitespaces from the produced tokens
        values = line.rstrip().rsplit(' ')
        
        # index 0 of the list 'values' contains the word
        word = values[0]
        
        # index 1 uptil the last index of the list 'values', we have the corresponding embedding
        coefs = np.asarray(values[1:], dtype='float64')
        
        # put the <word : word-embedding> pair in the dictionary named 'embeddings_index'
        embeddings_index[word] = coefs
    
    # close the file
    f.close()
    
    # the number of word vectors present in the embedding file
    print('Found %s word vectors' % len(embeddings_index))
    
    # returning the created 'embeddings_index'
    return embeddings_index<jupyter_output><empty_output><jupyter_text>Invoking loadEmbedding function to create the emedding-index<jupyter_code># Loading pre-trained embeddings by invoking loadEmbedding function
model = loadEmbedding()<jupyter_output>Loading Word Embeddings...
Found 1999996 word vectors<jupyter_text>Lets have a look how some random word(if present in the embedding index) will be represented using word-embedding<jupyter_code># lets say, our random word is 'love'
# following expression will give the word vector assoiated with word 'love'
model['love']<jupyter_output><empty_output><jupyter_text>Number Of Snippets in our 'dataset.csv'<jupyter_code>len(df['Snippet'])<jupyter_output><empty_output><jupyter_text>Extract all the words from our 'dataset.csv'<jupyter_code># the list 'all_words' will maintain all the words present in 'dataset.csv' 
all_words=[]

# iterate over all the snippets, tokenize them and store the tokens(i.e. words)
for i in range(len(df['Snippet'])):
    
    # tokenizer.tokenize(df['Snippet'][i]) generates the list of tokens(words) contained 
    # in the i'th snippet.
    
    # list of tokens that are generated for i'th snippet, will be appended to the 'all_word' list
    all_words.extend(tokenizer.tokenize(df['Snippet'][i]))<jupyter_output><empty_output><jupyter_text>A look at 'all_word' list<jupyter_code>all_words<jupyter_output><empty_output><jupyter_text>What is the length of 'all_word' list? Note: 'all_word' will even be maintaining the presence of duplicate words<jupyter_code>len(all_words)<jupyter_output><empty_output><jupyter_text>Lets remove the presence of duplicate words, if any<jupyter_code>all_words=list(dict.fromkeys(all_words))<jupyter_output><empty_output><jupyter_text>A look at 'all_word' list Note: 'all_word' list no longer peserves the duplicacy of words<jupyter_code>all_words<jupyter_output><empty_output><jupyter_text>What is the length of 'all_word' list after removing duplicates?<jupyter_code>len(all_words)<jupyter_output><empty_output><jupyter_text>Converting every word in the 'all_word' list to lowercase<jupyter_code>all_words=[x.lower() for x in all_words]<jupyter_output><empty_output><jupyter_text>A look at 'all_word' list after every word has been converted to lowercase<jupyter_code>all_words<jupyter_output><empty_output><jupyter_text>Now, we'll be creating the word vector corresponding to every word present in the 'all_word' list<jupyter_code># 'embeddings' dictionary will maintain the word from the 'all_word' list and the 
# corresponding word vector.
# if 'embedding_index' / 'model' represents the universe of the word and the 
# corresponding word embedding, then 'embeddings' represents the subset of that 
# universe of word embeddings
embeddings = {}

# iterate through all the words present in the 'all_word' list
for word in all_words:
    
    # if the current 'word' is not in the 'model.keys()', it means such 
    # word has no word vector representation in the universe of word embedding i.e
    # 'model' / 'embedding_index'.
    
    # and, if the such word has no vector in the in the universe of word embedding, its
    # always better to reprsent them using the word vector of 'unk'('unk' is anlogous to unknown).
    
    # you can think of 'unk' as the category of words whose vector represntation is not
    # present in the 'model' / 'embedding_index'
    if word not in model.keys():
        
        # assigning the vector representation of 'unk' to the current 'word'
        # i.e creating <word, vector representation of 'unk'> pair for the current 'word' 
        embeddings[word] = model['unk']
        
    else:
        
        # if the current 'word' is in the 'model.keys()', it means such 
        # word has a word vector representation in the universe of word embedding i.e
        # 'model' / 'embedding_index'.
        
        # creating <'word', vector representation of 'word'> pair for the current 'word'
        embeddings[word] = model[word]<jupyter_output><empty_output><jupyter_text>Reading pre_processed_dataset.csv and storing it into the dataframe named 'ppd'<jupyter_code>ppd=pd.read_csv('pre_processed_dataset.csv',encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Finding out the length(in terms of words) of left context associated with the candidate word present in 'pre_processed_dataset.csv'<jupyter_code># list 'length_l' will maintain the length(in terms of words) of the left context 
# associated with the probable candidate word  
length_l=[]

# iterate through the all the left contexts associated with the i'th snippet in the
# 'pre_processed_dataset.csv'
for i in range(len(ppd['left_context'])):
    # tokenizer.tokenize(ppd['left_context'][i]) tokenizes the left context of the i'th
    # snippet.
    
    # len(tokenizer.tokenize(ppd['left_context'][i])) calculates the length of the left 
    # context of the i'th snippet in 'pre_processed_dataset.csv'.
    
    # and the calculated length is further appended to the list 'length_l'
    length_l.append(len(tokenizer.tokenize(ppd['left_context'][i])))<jupyter_output><empty_output><jupyter_text>Finding out the length(in terms of words) of right context associated with the candidate word present in 'pre_processed_dataset.csv'<jupyter_code># list 'length_r' will maintain the length(in terms of words) of the right context 
# associated with the probable candidate word  
length_r=[]

# iterate through the all the right contexts associated with the i'th snippet in the
# 'pre_processed_dataset.csv'
for i in range(len(ppd['right_context'])):
    # tokenizer.tokenize(ppd['right_context'][i]) tokenizes the right context of the i'th
    # snippet.
    
    # len(tokenizer.tokenize(ppd['right_context'][i])) calculates the length of the right 
    # context of the i'th snippet in 'pre_processed_dataset.csv'.
    
    # and the calculated length is further appended to the list 'length_r'
    length_r.append(len(tokenizer.tokenize(ppd['right_context'][i])))<jupyter_output><empty_output><jupyter_text>Embedding associated with padding<jupyter_code>embeddings['<pad>']= [0]*300<jupyter_output><empty_output><jupyter_text>The peculiar behaviour of 'model' / 'embedding_index'<jupyter_code># if the following expression is executed then it displays the
# vector representation associated with 'how.'
model['how.']
# but, if I run the following expression, the model generates key error.
# why ?
# probably beacuse the word 'how?' is not present in the 'model' and 
# hence, no vector representation
model['how?']<jupyter_output><empty_output><jupyter_text>But, what makes me point at that peculiarity?<jupyter_code># to be continued<jupyter_output><empty_output><jupyter_text>Why use tokenize() and not split()?  1. split()<jupyter_code># for sentence "hey! isn't it good!!!?? ....", lets see what split() does
tokens_from_split = ("hey! isn't it good!!!?? ....").split()

# Number of tokens generated using split()
number_of_tokens = len(tokens_from_split)

print("Tokens Obtained After Using split() : ", tokens_from_split)
print("Number of tokens generated using split() : ", number_of_tokens)<jupyter_output>Tokens Obtained After Using split() :  ['hey!', "isn't", 'it', 'good!!!??', '....']
Number of tokens generated using split() :  5<jupyter_text>2. tokenize()<jupyter_code># for sentence "hey! isn't it good!!!?? ....", lets see what tokenize() does
tokens_from_tokenize = tokenizer.tokenize("hey! isn't ,,, it good!!!?? ....")

# Number of tokens generated using split()
number_of_tokens = len(tokens_from_tokenize)

print("Tokens Obtained After Using tokenize() : ", tokens_from_tokenize)
print("Number of tokens generated using tokenize() : ", number_of_tokens)<jupyter_output>Tokens Obtained After Using tokenize() :  ['hey', "isn't", 'it', 'good']
Number of tokens generated using tokenize() :  4<jupyter_text>Note that the tokens generated from tokenize using previously defined regex expression makes more sense than simply using split() whose default delimiter is ' '(a whitespace). Why they make more sense? Words without punctuation are highly likely to be present in the 'model' than the words with several punctuations.  Creation of the embedding for left context<jupyter_code># following list will maintain the embedding of left context associated with candidate 
# word present in the i'th snippet in 'pre_processed_dataset.csv' at the i'th index
keras_left_context = []

# iterate over all the left contexts of the snippets present in the
# dataframe 'ppd'('ppd' was created from 'pre_processed_dataset.csv')
for i in range(len(ppd['left_context'])):
    
    # one_vector will maintain the embedding for the left context associated with 
    # i'th snippet
    one_vector = []
    
    # temp stores the list of tokens generated after tokenizing the left context 
    # of the i'th snippet
    temp = tokenizer.tokenize(ppd['left_context'][i])
    
    # as given in the paper, the text will be appended with <start> tag in the beginning
    # to mark the starting of the text
    
    # therfore, lets append the vector representation of <start> to the list 'one_vector',
    # which maintains the embedding of the left context
    one_vector.append(model['start'])
    
    # for every token(except the one present at 0'th index) present in the 'temp' list,
    # iterate
    for m in temp[1:]:
        try:
            
            # extract the embedding associated with the token represented by 'm' in 'temp' list
            # and, store it in 'a'
            a = embeddings[m.lower()]
        
        except KeyError:
            
            # if the token is not present in the 'embeddings' dictionary,
            # then give that token a vector representation of unknown i.e 'unk'
            a = model['unk']
        
        # append the embeddings associated with token 'm' in 'one_vector' list
        one_vector.append(a)
     
    # for different snippets the size of left context(i.e the number of tokens present in the left context) 
    # will be different.
    
    # and, if you recollect the description of the model from the paper, you must understand that for the snippet,
    # [left context, candidate word, right context] is what is going to be passed to the RNN's.
    # therefore, we need to agree on some bare minimum size(78 is considered to be the are minimum size here) of the 
    # vector representation of left context
    
    # if the number of words in the left context is less than 78, then the representation of left context will 
    # padded with the 78 - length_l[i] vector representation of '<pad>'
    one_vector.extend([embeddings['<pad>'] for x in range(78-length_l[i])])
    
    # append the embedding of the left context associated with the i'th snippet to 
    # the list 'keras_left_context'
    keras_left_context.append(one_vector)<jupyter_output><empty_output><jupyter_text>How many left context embeddings are present in the list 'keras_left_context' ?<jupyter_code>len(keras_left_context)<jupyter_output><empty_output><jupyter_text>And, you should not be surprised to see 5913, coz the number of snippets in 'pre_processed_dataset.csv' is 5913. And, associated with every snippet, we've the left context. Hence, 5913.  Lets investigate the embedding of the left context of the snippet present at 0'th index in the data frame 'ppd<jupyter_code>keras_left_context[0]<jupyter_output><empty_output><jupyter_text>If you go through the above produced output carefully, you'll understand that there are 77 embeddings of 'pad' and one embedding of 'start'. Looking at the embedding of the left context associated with 0'th snippet in the data frame 'ppd', one can say that each left context embedding is the collection of 78 vectors and each vector in turn is of size 300.  Creation of the embedding for left context<jupyter_code># following list will maintain the embedding of right context associated with candidate 
# word present in the i'th snippet in 'pre_processed_dataset.csv' at the i'th index
keras_right_context=[]

# iterate over all the right contexts of the snippets present in the
# dataframe 'ppd'('ppd' was created from 'pre_processed_dataset.csv')
for i in range(len(ppd['right_context'])):
    
    # one_vector will maintain the embedding for the right context associated with 
    # i'th snippet
    one_vector=[]
    
    # temp stores the list of tokens generated after tokenizing the right context 
    # of the i'th snippet
    temp=tokenizer.tokenize(ppd['right_context'][i])
    
    # for every token(except the last present at -1'th index i.e. last index) present in the 'temp' list,
    # iterate
    for m in temp[:-1]:
        
        try:
            
            # extract the embedding associated with the token represented by 'm' in 'temp' list
            # and, store it in 'a'
            a = embeddings[m.lower()]
        
        except keyError:
            
            # if the token is not present in the 'embeddings' dictionary,
            # then give that token a vector representation of unknown i.e 'unk'
            a = model['unk']
        
        # append the embeddings associated with token 'm' in 'one_vector' list
        one_vector.append(a)
    
    # as given in the paper, the text will be appended with <end> tag in the end
    # to mark the ending of the text
    
    # therfore, lets append the vector representation of <end> to the list 'one_vector',
    # which maintains the embedding of the right context
    one_vector.append(model['end'])
    
    # if the number of words in the right context is less than 78, then the representation of right context will 
    # padded with the (78 - length_r[i]) vector representation of '<pad>'
    one_vector.extend([embeddings['<pad>'] for x in range(78-length_r[i])])
    
    # append the embedding of the right context associated with the i'th snippet to 
    # the list 'keras_right_context'
    keras_right_context.append(one_vector)<jupyter_output><empty_output><jupyter_text>How many right context embeddings are present in the list 'keras_right_context' ?<jupyter_code>len(keras_right_context)<jupyter_output><empty_output><jupyter_text>And, you should not be surprised to see 5913, coz the number of snippets in 'pre_processed_dataset.csv' is 5913. And, associated with every snippet, we've the right context. Hence, 5913.  Lets investigate the embedding of the right context of the snippet present at 0'th index in the data frame 'ppd'<jupyter_code>keras_right_context[0]<jupyter_output><empty_output><jupyter_text>Looking at the embedding of the right context associated with 0'th snippet in the data frame 'ppd', one can say that each right context embedding is the collection of 78 vectors and each vector in turn is of size 300.  Creation of the Embedding of the candidate word<jupyter_code># keras_middle will hold the embedding associated with the candidate word of every snippet 
# present in the data frame 'ppd' 
keras_middle=[]

# for every candidate word associated with the i'th snippet present in the data frame 'ppd',
# iterate
for i in range(len(ppd['Candidate_words'])):
    
    # ppd['Candidate_words'][i] - represents candidate word of the i'th snippet contained in the 
    # data frame named 'ppd'
    
    # ppd['Candidate_words'][i].lower() - candidate word is changed into its lowercase representation
    
    # embeddings[ppd['Candidate_words'][i].lower()] - generates the vector representation of the 
    # candidate word associated with i'th snippet of the dataframe 'ppd'
    
    # once the embedding of the candidate word has been created , append it to the list 'keras_middle'
    keras_middle.append(embeddings[ppd['Candidate_words'][i].lower()])<jupyter_output><empty_output><jupyter_text>The question is why the 'KeyError' was thrown. That means in the original data set 'him' along with '.' was made the candidate word for whatsoever snippet it belongs to. And as mentioned before, the model is highly likely to produce the vector representation for word without any punctuations than the word with punctuation.  The above reasoning pushes us to use tokenize() function, so that we just get word from the candidate word and we need not worry about punctuations. And thus, it becomes highly likely that the word and the corresponding embedding will be found in the 'embeddings' list.  Lets tokenize the candidate word and see what happens<jupyter_code># keras_middle will hold the embedding associated with the candidate word of every snippet 
# present in the data frame 'ppd' 
keras_middle=[]

# for every candidate word associated with the i'th snippet present in the data frame 'ppd',
# iterate
for i in range(len(ppd['Candidate_words'])):
    
    # ppd['Candidate_words'][i] - represents candidate word of the i'th snippet contained in the 
    # data frame named 'ppd'
    
    # ppd['Candidate_words'][i].lower() - candidate word is changed into its lowercase representation
    
    # tokenizer.tokenize(ppd['Candidate_words'][i].lower())- generates the candidate word that wont be
    # having any punctuations attached to it
    
    # candidate word without any punctuations
    candidate = tokenizer.tokenize(ppd['Candidate_words'][i].lower())
    
    # printing candidates word
    print(candidate)<jupyter_output>['i']
['frowned']
['at']
['him']
["isn't"]
['sarcasm']
['the']
['opiate']
['of']
['the']
['masses']
["you're"]
['thinking']
['of']
['religion']
['he']
['replied']
['sarcasm']
['is']
['the']
['xanax']
['of']
['the']
['morally']
['bereft']
['fine']
['it’s']
['decided']
['vain']
['resumed']
['his']
['stride']
['i’ll']
['do']
['all']
['the']
['killing']
['and']
['you']
['do']
['all']
['the']
['stuff']
['that']
['an']
['avun-riah']
['does']
['whatever']
['the']
['hell']
['that']
['is']
['what']
['time']
['is']
['it']
['one']
["o'clock"]
['i']
['nearly']
['spit']
['out']
['some']
['soda']
['in']
['the']
['afternoon']
['no']
['in']
['the']
['morning']
["don't"]
['let']
['that']
['damn']
['sunlight']
['fool']
['you']
['it']
['lies']
['i']
['understand']
['that']
['you']
['don’t']
['want']
['to']
['marry']
['me']
['i']
['said']
['i']
['mean']
['i']
['don’t']
['know']
['why']
['since']
['i’m']
['simply']
['delightful']
['to']
['be']
['around']
['but']
['to']
['each']
['his']
['own']
['taste']
['[...]<jupyter_text>Now that we have candidates word without any punctuations,we are good to go to get the associated embedding<jupyter_code># keras_middle will hold the embedding associated with the candidate word of every snippet 
# present in the data frame 'ppd' 
keras_middle=[]


# for every candidate word associated with the i'th snippet present in the data frame 'ppd',
# iterate
for i in range(len(ppd['Candidate_words'])):
    
    # ppd['Candidate_words'][i] - represents candidate word of the i'th snippet contained in the 
    # data frame named 'ppd'
    
    # ppd['Candidate_words'][i].lower() - candidate word is changed into its lowercase representation
    
    # tokenizer.tokenize(ppd['Candidate_words'][i].lower())- generates the candidate word that wont be
    # having any punctuations attached to it
    
    # candidate word without any punctuations
    candidate = tokenizer.tokenize(ppd['Candidate_words'][i].lower())
    
    # lets have a look at the embedding of the candidate word
    print(embeddings[candidate])<jupyter_output><empty_output><jupyter_text>TypeError: unhashable type: 'list'. Why? Note:  TypeError: unhashable type: 'list' usually means that you are trying to use a list as an hash argument. Did you notice square brackets(representing that its a list) while we printed the candidate words after tokeninzing them?\ The dictionary 'embedding' is expecting a string as a key and not some list  Let us see how many candidate words are there whose vector representation can be fetched from the 'embedding' dictionary<jupyter_code># keras_middle will hold the embedding associated with the candidate word of every snippet 
# present in the data frame 'ppd'
keras_middle=[]

# 'c' represents the number of candidates word whose vector representation was found in the 
# 'embeddings' dictionary
c = 0

# for every candidate word associated with the i'th snippet present in the data frame 'ppd',
# iterate
for i in range(len(ppd['Candidate_words'])):
    
    # ppd['Candidate_words'][i] - represents candidate word of the i'th snippet contained in the 
    # data frame named 'ppd'
    
    # ppd['Candidate_words'][i].lower() - candidate word is changed into its lowercase representation
    
    # tokenizer.tokenize(ppd['Candidate_words'][i].lower())- generates the candidate word that wont be
    # having any punctuations attached to it
    
    # but the candidate word that was produced without any punctuations is actually a list
    # and we've seen that before
    
    # converting candidate word from list to string using join() function
    candidate = " ".join(tokenizer.tokenize(ppd['Candidate_words'][i].lower()))
    
    # check if the candidate word is in the dictionary or not
    if candidate in embeddings:
        
        # if candidate word is in the 'embeddings' dictionary, increment the count of 'c'
        c = c + 1

print("Number of words for which embedding was found : ", c)<jupyter_output>Number of words for which embedding was found :  5862<jupyter_text>But there are 5913 snippets in the dataframe 'ppd'. So, logically, we should be having 5913 candidate words and 5913 corresponding vector representations. But our 'embeddings' dictionary has knowledge about just 5862 words. Remaining 51 words, its unaware of. So, those 51 words will be given the vector represenation of unknown which has been previously represented as 'unk'.  A final attempt to produce the embeddings for the candidate words after we have faced different errors<jupyter_code># keras_middle will hold the embedding associated with the candidate word of every snippet 
# present in the data frame 'ppd'
keras_middle=[]

# for every candidate word associated with the i'th snippet present in the data frame 'ppd',
# iterate
for i in range(len(ppd['Candidate_words'])):
    
    # ppd['Candidate_words'][i] - represents candidate word of the i'th snippet contained in the 
    # data frame named 'ppd'
    
    # ppd['Candidate_words'][i].lower() - candidate word is changed into its lowercase representation
    
    # tokenizer.tokenize(ppd['Candidate_words'][i].lower())- generates the candidate word that wont be
    # having any punctuations attached to it
    
    # but the candidate word that was produced without any punctuations is actually a list
    # and we've seen that before
    
    # converting candidate word from list to string using join() function
    candidate = " ".join(tokenizer.tokenize(ppd['Candidate_words'][i].lower()))
    

    if candidate in embeddings:
        
        # if candidate word is in the 'embeddings' dictionary, then append
        # the corresponding vector representation to the 'keras_middle' list
        keras_middle.append(embeddings[candidate])
    
    else:
        
        # if candidate word is nt in the 'embeddings' dictionary, then append
        # the vector representation of unknown i.e 'unk' to the 'keras_middle' list
        keras_middle.append(model['unk'])<jupyter_output><empty_output><jupyter_text>Dump the processed dataset in a pickle file<jupyter_code># opening the file for writing in binary mode
f = open(b"Data_fast.pkl","wb")

# Saving the processed dataset in a pickle file
pickle.dump(zip(keras_left_context,keras_right_context,keras_middle,ppd['target_status']),f)<jupyter_output><empty_output><jupyter_text>Shape of 'keras_left_context' Can you decode as to what each dimensions of the shape represent ?<jupyter_code>np.shape(keras_left_context)<jupyter_output><empty_output><jupyter_text>Shape of 'keras_right_context' Can you decode as to what each dimensions of the shape represent ?<jupyter_code>np.shape(keras_right_context)<jupyter_output><empty_output><jupyter_text>Shape of 'keras_middle' Can you decode as to what each dimensions of the shape represent ?<jupyter_code>np.shape(keras_middle)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09527678274677795, 'fraction_numerical': 0.0043206913106096975, 'mean_word_length': 4.33806426177804, 'pattern_counts': {'":': 0, '<': 168, '<?xml version=': 0, '>': 168, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '226986', 'license': 'MIT', 'n_tokens_mistral': 8134, 'n_tokens_neox': 7865, 'n_words': 3047, 'path': 'Data_preparation.ipynb', 'repo_name': 'RaudraKush/sarcasm-target-detection', 'size': '27079'}