<jupyter_start><jupyter_text>Patent Corpus Text Cleaning* Patent are a special kind of textual data that contain plenty of technical terms, specific words serving as transition phrases, and numerous academic words that describe invention outcomes. Before modeling topics with LDA, we also utilize three modules to remove general words from the corpus of patents as follows:    * Stop words such as the, that, and these;    * High-frequency words in patent claims such as claimed, comprising, and invention;    * General academic words such as research, approach, and data.**Among all the terms only technical terms provide the most meaningful information that reflects technological topics and innovations.**<jupyter_code>import os
import pandas as pd 
os.chdir('/Users/sheeroh/Box Sync/2_projects/insightDSNYC/data/')
from loadData_workflow import loadData
#load the saved pickle file patent document
patdocs = pd.read_pickle( 'patdocs_clean.pkl')

#check if all data has been successfully loaded
#patdocs.head(2)<jupyter_output><empty_output><jupyter_text>Text Cleaning<jupyter_code>from nltk.corpus import stopwords # Import the stop word list
import nltk
import re

def patent_to_words( raw_review ):
    letters_only = re.sub("[^a-zA-Z]", " ", raw_review) 
    lower_case = letters_only.lower()        # Convert to lower case
    words = lower_case.split()               # Split into words
    from nltk.corpus import stopwords # Import the stop word list
    #print (stopwords.words("english"))
    words = [w for w in words if not w in stopwords.words("english")] 
    #print (words)
    xx = stopwords.words("english")
    # Add first, second and one
    xx.extend(["first","second","one","two","also","may","least","present","determine",
    "included","includes","include","provided","provides","wherein","method","methods",
    "comprises","comprised","comprising","used","uses","using","use","say","says","said","disclose","discloses","disclosed",
    "containing","contain","contains","contained","make","made","makes","end","couple","relates", 'invention','including',
    "b","c","d", 'new','described', 'gift', 'A', "research", 'group', 'according',"approach", 'data', 'system', 'x', 'claimed', 'claim', 'therein'])
    stops = set(xx)               
    #Remove stop words
    meaningful_words = [w for w in words if not w in stops]
    return( " ".join( meaningful_words)) 
patent_to_words(patdocs["abstract"][0])<jupyter_output><empty_output><jupyter_text>Clean and Tokenize patents into lists (each patent is a words array)Saved clean_abstract to file so as to pick it up from here later on.<jupyter_code>import numpy as np
import json

# Get the number of patents based on the dataframe column size
num_patents = patdocs["abstract"].size
print(num_patents)

# Initialize an empty list to hold the clean reviews
clean_abstract = []

# Loop over each review; create an index i that goes from 0 to the length of the patent list 
for i in range( 0, num_patents ):
    patent = patent_to_words(patdocs["abstract"][i])
    if i%10000==0:print(i)
    array = patent.split()
    clean_abstract.append(array)
with open('clean_abstract.txt', 'w') as outfile:
    json.dump(clean_abstract, outfile)<jupyter_output><empty_output><jupyter_text>Next, using gensim's Phrases function, we create additional bigrams to include in the topic modeling. We used [12] as a reference to create the bigrams.<jupyter_code>from gensim.models.phrases import Phrases
import gensim
from gensim import corpora, models, similarities
from gensim.corpora import Dictionary
# Identify Bigrams using gensim's Phrases function

#bigram = Phraser(phrases)
bigram = models.Phrases(clean_abstract, delimiter=b'_')
print(bigram)

final_abstract = []
for i in range(0,num_patents):
    sent = clean_abstract[i] 
    temp_bigram = bigram[sent]
    final_abstract.append(temp_bigram)
    #if i%10000==0:print(i)
final_column = pd.Series(final_abstract)
   
#list(bigram[final_abstract])
#print(bigram[sent])
#len(bigram.vocab)
# create a new column final_column
final_column = pd.Series(final_abstract)

#print(final_column)
patdocs['final_column']= final_column.values
patdocs.head(2)
patdocs.to_csv('patdocs_final_column.csv')<jupyter_output><empty_output><jupyter_text>Convert tokenized document to dictionary and document-term matrix<jupyter_code>os.chdir("model")

# turn our tokenized documents into a id <-> term dictionary
dictionary = corpora.Dictionary(final_abstract)
dictionary.save('abstract_new.dict') # store the dictionary, for future reference-
     
# convert tokenized documents into a document-term matrix (bag-of-words)
corpus = [dictionary.doc2bow(text) for text in final_abstract]
corpora.MmCorpus.serialize('abstract_new.mm', corpus)

print('Number of unique tokens: %d' % len(dictionary))
print('Number of documents: %d' % len(corpus))

#print(corpus)
#mm_corpus = gensim.corpora.MmCorpus('abstract.mm')
#print(mm_corpus)
len(final_abstract)-1000<jupyter_output><empty_output><jupyter_text>Split the data into training and text<jupyter_code>import random
import pickle
random.seed(7)# 42 is not always th2 answer, let's try something different :)

train_set = random.sample(list(range(0,len(final_abstract))),len(final_abstract)-1000)
test_set = [x for x in list(range(0,len(final_abstract))) if x not in train_set]

train_texts = [final_abstract[i] for i in train_set]
test_texts = [final_abstract[i] for i in test_set]

pickle.dump([train_set,test_set,train_texts,test_texts],open('./abstract_train_test_sets_new.pkl','wb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11710739341608203, 'fraction_numerical': 0.0064759848893685915, 'mean_word_length': 5.864197530864198, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '113832', 'license': 'MIT', 'n_tokens_mistral': 1779, 'n_tokens_neox': 1712, 'n_words': 502, 'path': 'notebooks/02_Text_Preprocessing-clean.ipynb', 'repo_name': 'ShiroJean/insightDSNYC', 'size': '5559'}