<jupyter_start><jupyter_text>POSTme!: Tool for Social Media Messaging Optimization   Notebook : Part of Speech Tagging              Goal: Parts of speech (POS) are specific lexical categories to which words are assigned, based on their syntactic context and role. Usually, words can fall into one of the following major categories. The process of classifying and labeling POS tags for words called parts of speech tagging or POS tagging. Knowledge about the structure and syntax of language is helpful in many areas like text processing, annotation, and parsing for further operations such as text classification or summarization. The aim of this notebook is to calculate the number of certain POS tags.         [source](https://towardsdatascience.com/a-practitioners-guide-to-natural-language-processing-part-i-processing-understanding-text-9f4abfd13e72)  Import Dependencies<jupyter_code>!pip install contractions
!pip install textsearch
!pip install tqdm
import nltk
import contractions
import re
import tqdm

import pandas as pd
import numpy as np

import textblob

from google.colab import drive #to import and export data directly
nltk.download("punkt")
nltk.download('averaged_perceptron_tagger')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.<jupyter_text>POS taggingFor this task, we will only count the number of specific parts of speech and categorize them under certain groups. As an example, singular nouns (NN), plural nouns(NNS), singular proper nouns(NNP), and plural proper nouns (NNPS) will be categorized as nouns.  List of Part-of-Speech Tags| POS Tag | Description | Example ||---------|---------------------------------------|-----------------------------------------|| CC | coordinating conjunction | and || CD | cardinal number | 1, third || DT | determiner | the || EX | existential there | there is || FW | foreign word | d’hoevre || IN | preposition/subordinating conjunction | in, of, like || JJ | adjective | big || JJR | adjective, comparative | bigger || JJS | adjective, superlative | biggest || LS | list marker | 1) || MD | modal | could, will || NN | noun, singular or mass | door || NNS | noun plural | doors || NNP | proper noun, singular | John || NNPS | proper noun, plural | Vikings || PDT | predeterminer | both the boys || POS | possessive ending | friend‘s || PRP | personal pronoun | I, he, it || PRP\$ | possessive pronoun | my, his || RB | adverb | however, usually, naturally, here, good || RBR | adverb, comparative | better || RBS | adverb, superlative | best || RP | particle | give up || TO | to | to go, to him || UH | interjection | uhhuhhuhh || VB | verb, base form | take || VBD | verb, past tense | took || VBG | verb, gerund/present participle | taking || VBN | verb, past participle | taken || VBP | verb, sing. present, non-3d | take || VBZ | verb, 3rd person sing. present | takes || WDT | wh-determiner | which || WP | wh-pronoun | who, what || WP\$ | possessive wh-pronoun | whose || WRB | wh-abverb | where, when |[Source](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)<jupyter_code>pos_family = {
    'noun' : ['NN','NNS','NNP','NNPS'],
    'pron' : ['PRP','PRP$','WP','WP$'],
    'verb' : ['VB','VBD','VBG','VBN','VBP','VBZ'],
    'adj' :  ['JJ','JJR','JJS'],
    'adv' :  ['RB','RBR','RBS','WRB']
}<jupyter_output><empty_output><jupyter_text>In order to mark up the words in a text(corpus) as corresponding to a particular part of speech we use the TextBlob library. TextBlob is an open-source library for processing textual data and performing NLP tasks with ease. Check out the documentation here: [TextBlob](https://textblob.readthedocs.io/en/dev/)<jupyter_code># function to check and get the part of speech tag count of a words in a given sentence
# note this may take some time to execute on larger corpora

def check_pos_tag(x, flag):
    cnt = 0
    try:
        wiki = textblob.TextBlob(x)
        for tup in wiki.tags:
            ppo = list(tup)[1]
            if ppo in pos_family[flag]:
                cnt += 1
    except:
        pass
    return cnt<jupyter_output><empty_output><jupyter_text>Twitter Data<jupyter_code>df_twitter.info()
feature_col = 'data_text'
df_twitter['noun_count'] = df_twitter[feature_col].apply(lambda x: check_pos_tag(x, 'noun'))
df_twitter['verb_count'] = df_twitter[feature_col].apply(lambda x: check_pos_tag(x, 'verb'))
df_twitter['adj_count'] = df_twitter[feature_col].apply(lambda x: check_pos_tag(x, 'adj'))
df_twitter['adv_count'] = df_twitter[feature_col].apply(lambda x: check_pos_tag(x, 'adv'))
df_twitter['pron_count'] = df_twitter[feature_col].apply(lambda x: check_pos_tag(x, 'pron'))
twitter_pos = df_twitter[['data_id','noun_count', 'verb_count','adj_count', 'adv_count', 'pron_count']]
twitter_pos.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 268 entries, 0 to 267
Data columns (total 6 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   data_id     268 non-null    int64
 1   noun_count  268 non-null    int64
 2   verb_count  268 non-null    int64
 3   adj_count   268 non-null    int64
 4   adv_count   268 non-null    int64
 5   pron_count  268 non-null    int64
dtypes: int64(6)
memory usage: 12.7 KB<jupyter_text>Linkedin Data<jupyter_code>df_linkedin.info()
feature_col = "PostMessage"
df_linkedin['noun_count'] = df_linkedin[feature_col].apply(lambda x: check_pos_tag(x, 'noun'))
df_linkedin['verb_count'] = df_linkedin[feature_col].apply(lambda x: check_pos_tag(x, 'verb'))
df_linkedin['adj_count'] = df_linkedin[feature_col].apply(lambda x: check_pos_tag(x, 'adj'))
df_linkedin['adv_count'] = df_linkedin[feature_col].apply(lambda x: check_pos_tag(x, 'adv'))
df_linkedin['pron_count'] = df_linkedin[feature_col].apply(lambda x: check_pos_tag(x, 'pron'))
linkedin_pos = df_linkedin[['DataID','noun_count', 'verb_count','adj_count', 'adv_count', 'pron_count']]
linkedin_pos = linkedin_pos.rename(columns={"DataID":"data_id"})
linkedin_pos.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 273 entries, 0 to 272
Data columns (total 6 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   data_id     273 non-null    int64
 1   noun_count  273 non-null    int64
 2   verb_count  273 non-null    int64
 3   adj_count   273 non-null    int64
 4   adv_count   273 non-null    int64
 5   pron_count  273 non-null    int64
dtypes: int64(6)
memory usage: 12.9 KB
{'fraction_non_alphanumeric': 0.1348802395209581, 'fraction_numerical': 0.017964071856287425, 'mean_word_length': 4.042264150943396, 'pattern_counts': {'":': 1, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '590798', 'license': 'MIT', 'n_tokens_mistral': 2292, 'n_tokens_neox': 2129, 'n_words': 656, 'path': 'notebooks/03_Part of Speech (POS).ipynb', 'repo_name': 'marcelaulloa/POSTme', 'size': '6680'}