<jupyter_start><jupyter_text>Distribution of different Parts of Speech in Regular and Non-Regular Articles In this notebook, we will look at the distribution of different parts of speech in regular and non-regular articles. We would be looking at how many times different parts of speeches were used in an article. We first calulate a `Instance per article length` metric which evaluates the percentage of that POS appearing in the article. We then look at the distribution of `instance per article length` with the count of `articles` for both regular and non regular articlesTo ensure that the distribution are comparable, we have undersampled the "Regular" classOur general Hypothesis before this experiment is that the POS speech used in Regular and Non regular articles would be differentLet us first look at the different parts of speech detected. We use Spacy package from Python to conduct this experiment **Different Parts of speech**`ADJ` : adjective`ADP` : adposition`ADV` : adverb`AUX` : auxiliary verb`CONJ` : coordinating conjunction`DET` : determiner`INTJ` : interjection`NOUN` : noun`NUM` : numeral`PART` : particle`PRON` : pronoun`PROPN` : proper noun`PUNCT` : punctuation`SCONJ` : subordinating conjunction`SYM`: symbol`VERB` : verb`X` : other<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
#Importing Libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import spacy
import json 
import time
#Reading the from the sample file created for the Exploratory data analysis. Change the location if necessary
file_loc = "/content/drive/My Drive/Bloomberg Editorial Classifier/data/collected/combined_data_sample.csv"
df = pd.read_csv(file_loc)
df.head()
#Collecting Part of speech
regular_pos_all = {'ADJ': [], 'ADP': [], 'ADV': [], 'AUX': [], 'CONJ': [], 
                   'DET': [], 'INTJ': [], 'NOUN': [], 'NUM': [], 'PART': [], 
                   'PRON': [], 'PROPN': [], 'PUNCT': [], 'SCONJ': [], 'SYM': [],
                   'VERB': [], 'X':[],'SPACE':[], 'CCONJ' : []}

op_pos_all = {'ADJ': [], 'ADP': [], 'ADV': [], 'AUX': [], 'CONJ': [], 'DET': [],
              'INTJ': [], 'NOUN': [], 'NUM': [], 'PART': [], 'PRON': [],
              'PROPN': [], 'PUNCT': [], 'SCONJ': [], 'SYM': [], 'VERB': [], 
              'X':[], 'SPACE':[], 'CCONJ' : []}

start_time = time.time()
for j, row in df.iterrows():
    nlp = spacy.load('en_core_web_sm')
    article = row['article_text']
    try:
        doc = nlp(article)
        regular_pos = {}
        op_pos = {}
        for token in doc:
            if row['label'] == 'regular':
                if token.pos_ not in regular_pos.keys():
                    regular_pos[token.pos_] = 1
                else:
                    regular_pos[token.pos_] +=1
            else:
                if token.pos_ not in op_pos.keys():
                    op_pos[token.pos_] = 1
                else:
                    op_pos[token.pos_] +=1

        for i in op_pos.keys():
            op_pos_all[i].append(op_pos[i]/len(doc))

        for i in regular_pos.keys():
            regular_pos_all[i].append(regular_pos[i]/len(doc))
        break
    
    except:
        print(j)
        continue
print("Time for execution = %s seconds ---" % round((time.time() - start_time),2))

# Saving to a json file for future reference
import json
with open('regular_pos_all.json', 'w') as fp:
    json.dump(regular_pos_all, fp)
with open('op_pos_all.json', 'w') as fp2:
    json.dump(op_pos_all, fp2)
# Read from json file which was saved - Execute from here for re-running the results

with open('regular_pos_all.json') as json_file:
    regular_pos_all = json.load(json_file)
with open('op_pos_all.json') as json_file:
    op_pos_all = json.load(json_file)<jupyter_output><empty_output><jupyter_text>As some of the attributes are not that significant in our analysis, we drop them from the plotWe Drop the following POS- `CONJ` : coordinating conjunction- `SCONJ` : subordinating conjunction- `SYM`: symbol- `VERB` : verb- `X` : other<jupyter_code>#Preparing DataFrame for plotting from the dictionary
reg_pos_df = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in regular_pos_all.items() ]))
reg_pos_df = reg_pos_df.drop(columns=['CONJ', 'SCONJ', 'X', 'SYM', 'PUNCT'])

op_pos_df = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in op_pos_all.items() ]))
op_pos_df = op_pos_df.drop(columns=['CONJ', 'SCONJ', 'X', 'SYM', 'PUNCT'])
#Preparing the charts
fig, ax = plt.subplots(4,3, figsize=(16,16))
fig.suptitle('Distribution of different Parts of Speech', fontweight='bold')
plt.subplots_adjust(hspace=0.5)
plt.style.use('ggplot')
pos = reg_pos_df.columns

for i in range(4):
    for j in range(3):
        bins = np.histogram(np.hstack((op_pos_df[pos[0]],
                                       reg_pos_df[pos[0]]))[~np.isnan(np.hstack((op_pos_df[pos[0]],
                                                                                      reg_pos_df[pos[0]])))], 
                            bins=20)[1] 
        _ = ax[i, j].hist(op_pos_df[pos[0]], alpha=0.5, bins = bins, label='Non-Regular')
        _ = ax[i, j].hist(reg_pos_df[pos[0]], alpha=0.5, bins = bins, label='Regular')
        _ = ax[i, j].set_title(pos[0], fontweight='bold')
        _ = ax[i, j].legend()
        
        pos = pos[1:]
    
_ = ax[3, 0].set_xlabel('Instances by article length', fontweight='bold')
_ = ax[3, 1].set_xlabel('Instances by article length', fontweight='bold')
_ = ax[3, 2].set_xlabel('Instances by article length', fontweight='bold')

_ = ax[0, 0].set_ylabel('Articles', fontweight='bold')
_ = ax[1, 0].set_ylabel('Articles', fontweight='bold')
_ = ax[2, 0].set_ylabel('Articles', fontweight='bold')
_ = ax[3, 0].set_ylabel('Articles', fontweight='bold')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14888466194016947, 'fraction_numerical': 0.007954348953830192, 'mean_word_length': 3.2560706401766004, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '462738', 'license': 'MIT', 'n_tokens_mistral': 1943, 'n_tokens_neox': 1859, 'n_words': 519, 'path': 'src/eda/part_of_speech/POS_EDAV.ipynb', 'repo_name': 'ameyakarnad/bloomberg_editorial_classifier', 'size': '5783'}