<jupyter_start><jupyter_text>Stats for combined data<jupyter_code># Fetching all the text-target files from the folder and merging them 
path_all_text_target = '/content/drive/MyDrive/dataset_patent/text_target'
files_all_text_target = [f for f in listdir(path_all_text_target) if isfile(join(path_all_text_target, f))]
print(files_all_text_target)
combined_all_text_target = pd.concat([pd.read_csv(path_all_text_target+'/'+f) for f in files_all_text_target ])
combined_all_text_target.head()
# Scikit-learn shuffle 
combined_all_text_target = shuffle(combined_all_text_target, random_state=0)
combined_all_text_target.head()
# Reseting index such that it starts from 0,1,2, ... so on 
combined_all_text_target.reset_index(inplace=True, drop=True)
combined_all_text_target.head(10)
# A copied file to play with the text-target dataset
df_analysis = combined_all_text_target.copy()
# Calculating total numbers of the sample 
print("Size of the data: ", combined_all_text_target.shape[0])
# Checking counts of respective labels 
combined_all_text_target['target'].value_counts()
df_analysis['text'] = df_analysis['text'].apply(str)
# Tokenization using NLTK  - Taking 5 minutes on colab 


# df_analysis['tokenized_sents'] = df_analysis.apply(lambda row: nltk.word_tokenize(row['text']), axis=1)
# Tokenization using keras - Takind 1 minutes on colab
df_analysis['keras_tokenized_sents'] = df_analysis.apply(lambda row: text_to_word_sequence(row['text']), axis=1)
df_analysis.head()<jupyter_output><empty_output><jupyter_text>Let's see how both tokenization works<jupyter_code># nltk tokenization 

# print(df_analysis['tokenized_sents'][0])
# Keras tokenization 
print(df_analysis['keras_tokenized_sents'][0])<jupyter_output>['according', 'to', 'the', 'slurry', 'for', 'forming', 'a', 'mold', 'the', 'mold', 'and', 'the', 'method', 'for', 'producing', 'a', 'mold', 'a', 'casting', 'can', 'be', 'stably', 'molded', 'and', 'the', 'mold', 'can', 'easily', 'collapse', 'in', 'on', 'itself']<jupyter_text>It seems like the tokenization perform better considering following factors 1.   Execution time is better (As seen above in code)2.   Converting all tokens in lower case3.   It is not considering punctuation as new token<jupyter_code>df_analysis['total_words_k_token'] = df_analysis['keras_tokenized_sents'].apply(lambda x : len(x))
df_analysis.head()
# Total number of words in text - overall
print('Total number of words ' + str(df_analysis['total_words_k_token'].sum()))
# Highest number of words in text 
print('Highest number of words in text is : ' + str(df_analysis['total_words_k_token'].max()))

# Lowest number of words in text 
print('Lowest number of words in text is : ' + str(df_analysis['total_words_k_token'].min()))

# Average number of words in text 
print('Average number of words in text is : ' + str(df_analysis['total_words_k_token'].mean()))
# Checking the rows having lowest number of words = 0
df_analysis[df_analysis['total_words_k_token']==0]<jupyter_output><empty_output><jupyter_text>Depending on usage we would like to remove such rows<jupyter_code># Saving a generated file on drive = Having size of 1.41GB
# df_analysis.to_csv( "/content/drive/MyDrive/dataset_patent/generated_files/df_analysis.csv", index=False, encoding='utf-8-sig')
# df_analysis = pd.read_csv('/content/drive/MyDrive/dataset_patent/generated_files/df_analysis.csv')
df_analysis.head()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.histplot(df_analysis.total_words_k_token, kde=True, ax=ax, color='#BCD70E')
plt.title('KDE plot for total words token')
plt.show()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.boxplot(df_analysis.total_words_k_token, ax=ax, color='#ff0000')
plt.title('Barplot for total words token')
plt.show()
text = '''
The titular threat of The Blob has always struck me as the ultimate movie
monster: an insatiably hungry, amoeba-like mass able to penetrate
virtually any safeguard, capable of--as a doomed doctor chillingly
describes it--"assimilating flesh on contact.
Snide comparisons to gelatin be damned, it's a concept with the most
devastating of potential consequences, not unlike the grey goo scenario
proposed by technological theorists fearful of
artificial intelligence run rampant.
'''
blob = TextBlob(text)
b_tags = blob.tags           # [('The', 'DT'), ('titular', 'JJ'),
                    #  ('threat', 'NN'), ('of', 'IN'), ...]

b_np = blob.noun_phrases   # WordList(['titular threat', 'blob',
                    #            'ultimate movie monster',
                    #            'amoeba-like mass', ...])

for sentence in blob.sentences:
    print(sentence.sentiment.polarity)

print(b_tags)
print(b_np)    
text_p = df_analysis['text'][2]
print(text_p)<jupyter_output>According to the tire pressure decrease detection device, the method, and the program of the present invention, the influence of the periodic noise can be eliminated and the estimate precision of the tire resonance frequency can be improved.<jupyter_text>In a system disclosed in Japanese Patent Laying-Open No. 2010-035277 (PTL 1), determination of a charging mode and a power supply mode is performed based on a pilot signal transmitted from a power cable.', 'However, to improve accuracy in determination, it is preferable that two or more determination systems are provided. On this point, in a system for determining the power charging mode and the power supply mode based only on a pilot signal, there was a room for improvement in enhancing the accuracy in determination.<jupyter_code># Applying text blob extracting information 
text_p_blob = TextBlob(text_p)

# Extracting tags from the text 
text_p_tags = text_p_blob.tags
print(text_p_tags)
text_p_phrase = text_p_blob.noun_phrases
print(text_p_phrase)
for sentence in text_p_blob.sentences:
    print(sentence.sentiment.polarity)<jupyter_output><empty_output><jupyter_text>Cleaning data (considering duplicacy and null values)<jupyter_code>pos_text_target = pd.read_csv('/content/drive/MyDrive/dataset_patent/text_target/pos_text_target.csv',index_col= False)
neg_text_target = pd.read_csv('/content/drive/MyDrive/dataset_patent/text_target/neg_text_target.csv',index_col= False)
neut_text_target = pd.read_csv('/content/drive/MyDrive/dataset_patent/text_target/neut_text_target.csv',index_col= False)
# Chcking sahpe of dataframe before preprocessing
print(pos_text_target.shape)
print(neg_text_target.shape)
print(neut_text_target.shape)
# Checking null values in the table
print(pos_text_target.isnull().values.any())
print(neg_text_target.isnull().values.any())
print(neut_text_target.isnull().values.any())

# Dropping null values 
pos_text_target.dropna(axis = 0, inplace=True)
neg_text_target.dropna(axis = 0, inplace=True)
neut_text_target.dropna(axis = 0, inplace=True)

# Checking shape of frame after dropping null values 
print(pos_text_target.shape)
print(neg_text_target.shape)
print(neut_text_target.shape)
# Checking wether there is duplicate values or not
pos_text_target[pos_text_target.duplicated('text')]
pos_text_target['text'][180]
# Finding rows having same text data
pos_text_target[pos_text_target['text']=='The heat-absorbing glass plate of the present invention satisfies both low solar transmittance and high visible light transmittance. Particularly, according to the present invention, it is possible to obtain a heat-absorbing glass having a ratio Tv/Te of the visible light transmittance higher than conventional glass, relative to the amount of total iron t-Fe2O3 as calculated as Fe2O3 as represented by mass % based on oxides.']
# Removing duplicated rows and printing shape
pos_text_target.drop_duplicates(keep='first',inplace=True)
print(pos_text_target.shape)
# Checking wether there is duplicate values or not
neg_text_target[neg_text_target.duplicated('text')]
neg_text_target['text'][467]
# Finding rows having same text data
neg_text_target[neg_text_target['text'] == "An aspect of the present invention provides a method and apparatus for encoding an image, which can improve image compression efficiency.', 'Another aspect of the present invention provides a method and apparatus for decoding an image, which can improve image compression efficiency.', 'Still another aspect of the present invention provides a method and apparatus for inter prediction, which can improve image compression efficiency.', 'Still yet another aspect of the present invention provides a method and apparatus for deriving a temporal motion vector predictor, which can improve image compression efficiency."]
# Removing duplicated rows 
neg_text_target.drop_duplicates(keep='first',inplace=True)
print(neg_text_target.shape)
neut_text_target[neut_text_target.duplicated('text')]
neut_text_target['text'][239]
neut_text_target[neut_text_target['text']=="In order to solve the above-described problems, a method of producing a composite reinforcing material of the present invention comprises a step of kneading at least a graphite-based carbon material and a reinforcing material into a base material,', 'the graphite-based carbon material having a rhombohedral graphite layer (3R) and a hexagonal graphite layer (2H), wherein a Rate (3R) of the rhombohedral graphite layer (3R) and the hexagonal graphite layer (2H), based on an X-ray diffraction method, which is defined by following Equation 1 is 31% or more:"]
# Removing duplicated rows 
neut_text_target.drop_duplicates(keep='first',inplace=True)
print(neut_text_target.shape)
# Shape of frame after dropping values  

print(pos_text_target.shape)
print(neg_text_target.shape)
print(neut_text_target.shape)
# Tokenization using keras - Takind 1 minutes on colab
pos_text_target['keras_tokenized_sents'] = pos_text_target.apply(lambda row: text_to_word_sequence(row['text']), axis=1)
neg_text_target['keras_tokenized_sents'] = neg_text_target.apply(lambda row: text_to_word_sequence(row['text']), axis=1)
neut_text_target['keras_tokenized_sents'] = neut_text_target.apply(lambda row: text_to_word_sequence(row['text']), axis=1)

# Counting total words per text in dataframe 
pos_text_target['total_words_k_token'] = pos_text_target['keras_tokenized_sents'].apply(lambda x : len(x))
neg_text_target['total_words_k_token'] = neg_text_target['keras_tokenized_sents'].apply(lambda x : len(x))
neut_text_target['total_words_k_token'] = neut_text_target['keras_tokenized_sents'].apply(lambda x : len(x))

pos_text_target['total_words_k_token'].describe()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.histplot(pos_text_target.total_words_k_token, kde=True, ax=ax, color='#48a832')
plt.title('KDE plot for positive total words token')
plt.show()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.boxplot(pos_text_target.total_words_k_token, ax=ax, color='#48a832')
plt.title('Barplot for positive total words token')
plt.show()
# Counting total numbers of text having tokens greater than 500
pos_text_target[pos_text_target['total_words_k_token'] > 500].count()  
neg_text_target['total_words_k_token'].describe()

fig, ax = plt.subplots(figsize=(15, 7.5))
sns.histplot(neg_text_target.total_words_k_token, kde=True, ax=ax, color='#a83248')
plt.title('KDE plot for negative total words token')
plt.show()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.boxplot(pos_text_target.total_words_k_token, ax=ax, color='#a83248')
plt.title('Barplot for negative total words token')
plt.show()
# Counting total numbers of text having tokens greater than 500
neg_text_target[neg_text_target['total_words_k_token'] > 500].count()  
neut_text_target['total_words_k_token'].describe()

fig, ax = plt.subplots(figsize=(15, 7.5))
sns.histplot(neut_text_target.total_words_k_token, kde=True, ax=ax, color='#3266a8')
plt.title('KDE plot for neutral total words token')
plt.show()
fig, ax = plt.subplots(figsize=(15, 7.5))
sns.boxplot(pos_text_target.total_words_k_token, ax=ax, color='#3266a8')
plt.title('Barplot for neutral total words token')
plt.show()
# Counting total numbers of text having tokens greater than 500
neut_text_target[neut_text_target['total_words_k_token'] > 500].count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1054204366232257, 'fraction_numerical': 0.012285216236407404, 'mean_word_length': 6.698402555910543, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '66445', 'license': 'Apache-2.0', 'n_tokens_mistral': 3978, 'n_tokens_neox': 3782, 'n_words': 1082, 'path': 'Jupyter notebook/Prepare_split_analysis.ipynb', 'repo_name': 'Renuk9390/Patent_Sentiment_Analysis', 'size': '12047'}