<jupyter_start><jupyter_text>POSTme!: Tool for Social Media Messaging Optimization   Notebook : Named Entity Recognition              Goal: Named entity recognition (NER) , also known as entity chunking/extraction , is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc. This notebook is used to count certin named entities in social media posts   Install Dependencies<jupyter_code>!pip install contractions
!pip install textsearch
!pip install tqdm
import nltk
import pandas as pd
import numpy as np

import contractions
import re
import tqdm

from google.colab import drive #to download and upload data directly to google drive

nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
import spacy
from spacy import displacy

nlp = spacy.load('en_core_web_sm')
from collections import Counter<jupyter_output><empty_output><jupyter_text>Named Entities with spaCYThere are Named Entity Taggers available through popular nlp libraries like "nltk" and "spacy". Each of these libraries follows a different approach to solve the problem. For this task, we use spaCY library. [spaCY](https://spacy.io/api/data-formatsnamed-entities)<jupyter_code>nlp.pipeline<jupyter_output><empty_output><jupyter_text>![](https://i.imgur.com/EEYbJUl.png)<jupyter_code>#first, we calculate many different categories, later we can decide what to include and exclude

pos_family = {
    'person' : ['PERSON'],
    'norp' : ['NORP'],
    'fac' : ['FAC'],
    'org' : ['ORG'],
    'gpe' : ['GPE'],
    'loc' : ['LOC'],
    'product' : ['PRODUCT'],
    'event' : ['EVENT'],
    'work_of_art' : ['WORK_OF_ART'],
    'law' : ['LAW'],
    'language' : ['LANGUAGE'],
    'date' : ['DATE'],
    'time' : ['TIME'],
    'percent' : ['PERCENT'],
    'money' : ['MONEY'],
}
ner_entities = ['person_count', 'norp_count', 'fac_count', 'organization_count', 'gpe_count', 'loc_count', 'product_count', 'event_count', 'workOfart_count', 'law_count', 'language_count',
                'date_count', 'time_count', 'percent_count', 'money_count']
#function to count named entities
def check_ner(x, flag):
    cnt = 0
    try:
        nlp_text = nlp(x)
        for tup in [(word.text, word.ent_type_) for word in nlp_text]:
            ppo = list(tup)[1]
            if ppo in pos_family[flag]:
                cnt += 1
    except:
        pass
    return cnt<jupyter_output><empty_output><jupyter_text>Linkedin Data<jupyter_code>df_linkedin.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 273 entries, 0 to 272
Data columns (total 21 columns):
 #   Column                       Non-Null Count  Dtype  
---  ------                       --------------  -----  
 0   Unnamed: 0                   273 non-null    int64  
 1   DateTime                     273 non-null    object 
 2   LinkeInPage                  273 non-null    object 
 3   DataID                       273 non-null    int64  
 4   Tags                         97 non-null     object 
 5   PostCampaign                 0 non-null      float64
 6   PostType                     273 non-null    object 
 7   PostMessage                  273 non-null    object 
 8   Reactions                    273 non-null    int64  
 9   Impressions                  273 non-null    int64  
 10  Engagement                   273 non-null    float64
 11  Clicks                       273 non-null    int64  
 12  Shares                       273 non-null    int64  
 13  Comments            [...]<jupyter_text>we will use the pre-processed text to count named entities<jupyter_code>feature_col = "norm_linkedin_posts"
df_linkedin['person_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'person'))
df_linkedin['norp_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'norp'))
df_linkedin['fac_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'fac'))
df_linkedin['organization_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'org'))
df_linkedin['gpe_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'gpe'))
df_linkedin['loc_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'loc'))
df_linkedin['product_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'product'))
df_linkedin['event_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'event'))
df_linkedin['workOfart_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'work_of_art'))
df_linkedin['law_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'law'))
df_linkedin['language_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'language'))
df_linkedin['date_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'date'))
df_linkedin['time_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'time'))
df_linkedin['percent_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'percent'))
df_linkedin['money_count'] = df_linkedin[feature_col].apply(lambda x: check_ner(x, 'money'))

df2 = df_linkedin[['person_count', 'norp_count', 'fac_count', 'organization_count', 'gpe_count', 'loc_count', 'product_count', 'event_count', 'workOfart_count', 'law_count', 'language_count',
                'date_count', 'time_count', 'percent_count', 'money_count']]
df2.head(1)
linkedin_ner = df_linkedin[['DataID','person_count', 'norp_count','organization_count', 'gpe_count', 'loc_count', 'product_count',
                'date_count','money_count']]
linkedin_ner = linkedin_ner.rename(columns={"DataID":"data_id"})
linkedin_ner.head()
linkedin_ner.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 273 entries, 0 to 272
Data columns (total 9 columns):
 #   Column              Non-Null Count  Dtype
---  ------              --------------  -----
 0   data_id             273 non-null    int64
 1   person_count        273 non-null    int64
 2   norp_count          273 non-null    int64
 3   organization_count  273 non-null    int64
 4   gpe_count           273 non-null    int64
 5   loc_count           273 non-null    int64
 6   product_count       273 non-null    int64
 7   date_count          273 non-null    int64
 8   money_count         273 non-null    int64
dtypes: int64(9)
memory usage: 19.3 KB<jupyter_text>Twitter Data<jupyter_code>df_twitter.info()
feature_col = "norm_twitter_posts"
df_twitter['person_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'person'))
df_twitter['norp_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'norp'))
df_twitter['fac_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'fac'))
df_twitter['organization_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'org'))
df_twitter['gpe_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'gpe'))
df_twitter['loc_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'loc'))
df_twitter['product_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'product'))
df_twitter['event_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'event'))
df_twitter['workOfart_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'work_of_art'))
df_twitter['law_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'law'))
df_twitter['language_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'language'))
df_twitter['date_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'date'))
df_twitter['time_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'time'))
df_twitter['percent_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'percent'))
df_twitter['money_count'] = df_twitter[feature_col].apply(lambda x: check_ner(x, 'money'))

df = df_twitter[['data_id','person_count', 'norp_count', 'fac_count', 'organization_count', 'gpe_count', 'loc_count', 'product_count', 'event_count', 'workOfart_count', 'law_count', 'language_count',
                'date_count', 'time_count', 'percent_count', 'money_count']]
df.head()
twitter_ner = df_twitter[['data_id','person_count', 'norp_count','organization_count', 'gpe_count', 'loc_count', 'product_count',
                'date_count','money_count']]
twitter_ner.head()
twitter_ner.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 205 entries, 0 to 204
Data columns (total 9 columns):
 #   Column              Non-Null Count  Dtype
---  ------              --------------  -----
 0   data_id             205 non-null    int64
 1   person_count        205 non-null    int64
 2   norp_count          205 non-null    int64
 3   organization_count  205 non-null    int64
 4   gpe_count           205 non-null    int64
 5   loc_count           205 non-null    int64
 6   product_count       205 non-null    int64
 7   date_count          205 non-null    int64
 8   money_count         205 non-null    int64
dtypes: int64(9)
memory usage: 14.5 KB<jupyter_text>Check and Visualize only 1 TEXT in ROW<jupyter_code>df[df["person_count"]== df["person_count"].max()]
text_nlp = nlp(df_twitter["post_message"][3])
text_nlp_pro = nlp(df_twitter["processed_text"][146])
text_nlp_pro2 = nlp(df_twitter["pro_text2"][3])
ner_tagged = [(word.text, word.ent_type_) for word in text_nlp]
print(ner_tagged)
x = [(word.ent_type_) for word in text_nlp]
print(x)
named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in ner_tagged:
    if tag:
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None
c = Counter([item[1] for item in named_entities])
c.most_common()
from spacy import displacy

# visualize named entities
displacy.render(text_nlp, style='ent', jupyter=True)
displacy.render(text_nlp_pro, style='ent', jupyter=True)
displacy.render(text_nlp_pro2, style='ent', jupyter=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12939328490084429, 'fraction_numerical': 0.022776359709405065, 'mean_word_length': 3.8097261567516525, 'pattern_counts': {'":': 1, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '591216', 'license': 'MIT', 'n_tokens_mistral': 3565, 'n_tokens_neox': 3331, 'n_words': 770, 'path': 'notebooks/02_Named Entity Recognition (NER).ipynb', 'repo_name': 'marcelaulloa/POSTme', 'size': '10186'}