<jupyter_start><jupyter_text>Generate Parts of Speech tags using various python libraries  ```2.1 Generating POS tags using Polyglot library2.2 Generating POS tags using Stanford CoreNLP 2.3 Generating POS tags using Spacy library2.4 Why do we need to develop our own POS tagger?```  Generating POS tags using Polyglot library  ```2.1.1 Download polyglot POS model for English language2.1.2 Load POS model2.1.3 Import dependencies2.1.4 Detect the language2.1.5 Tokenization of the sentences2.1.6 Generate POS tags for given sentence```  2.1.1 Download polyglot POS model for English language<jupyter_code>from polyglot.downloader import downloader
print(downloader.supported_languages_table("pos2"))<jupyter_output><empty_output><jupyter_text>2.1.2 Load POS model<jupyter_code>from polyglot.downloader import downloader
downloader.download("embeddings2.en")
downloader.download("pos2.en")<jupyter_output><empty_output><jupyter_text>2.1.3 Import dependencies<jupyter_code>import polyglot
from polyglot.text import Text, Word<jupyter_output><empty_output><jupyter_text>2.1.4 Detect the language<jupyter_code>text = Text("<NAME>.")
print("Language Detected: Code={}, Name={}\n".format(text.language.code, text.language.name))<jupyter_output><empty_output><jupyter_text>2.1.5 Tokenization of the sentences<jupyter_code>words_list = Text("Beautiful is better than ugly. "
           "Explicit is better than implicit. "
           "Simple is better than complex.")
print(words_list.words)<jupyter_output><empty_output><jupyter_text>2.1.6 Generate POS tags for given sentence<jupyter_code>sent = """We will meet at eight o'clock on Thursday morning."""
text = Text(sent)
text.pos_tags
print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
for word, tag in text.pos_tags:
    print(u"{:<16}{:>2}".format(word, tag))
text = Text("This is a car")
text.pos_tags
print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
for word, tag in text.pos_tags:
    print(u"{:<16}{:>2}".format(word, tag))
text = Text("Alexander the Great...!")
text.pos_tags
print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
for word, tag in text.pos_tags:
    print(u"{:<16}{:>2}".format(word, tag))
text = Text("Alexander the Great, was a king of the ancient Greek kingdom of Macedon.")
text.pos_tags
print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
for word, tag in text.pos_tags:
    print(u"{:<16}{:>2}".format(word, tag))<jupyter_output>Word            POS Tag
------------------------------
Alexander       PROPN
the             DET
Great           PROPN
,               PUNCT
was             VERB
a               DET
king            NOUN
of              ADP
the             DET
ancient         ADJ
Greek           ADJ
kingdom         NOUN
of              ADP
Macedon         PROPN
.               PUNCT<jupyter_text>Generating POS tags using Stanford CoreNLP  ```2.2.1 Run Stanford CoreNLP in server mode2.2.2 Define a function2.2.3 Generate POS tags for given sentence```  2.2.1 Run Stanford CoreNLP in server mode<jupyter_code>## Run the server using all jars in the current directory (e.g., the CoreNLP home directory)
# java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer --add-modules java.se.ee -port 9000 -timeout 15000
from pycorenlp import StanfordCoreNLP
nlp = StanfordCoreNLP('http://localhost:9000')<jupyter_output><empty_output><jupyter_text>2.2.2 Define a function<jupyter_code>def stnfordpostagdemofunction(text):
    output = nlp.annotate(text, properties={
        'annotators': 'pos',
        'outputFormat': 'json'
    })
    for s in output["sentences"]:
        print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
        for t in s["tokens"]:
            print(u"{:<16}{:>2}".format(str(t["word"]),str(t["pos"])))<jupyter_output><empty_output><jupyter_text>2.2.3 Generate POS tags for given sentence<jupyter_code>if __name__ == "__main__":
    stnfordpostagdemofunction("This is a car.")
   
if __name__ == "__main__":
    stnfordpostagdemofunction("Alexander the Great, was a king of the ancient Greek kingdom of Macedon.")
if __name__ == "__main__":
    stnfordpostagdemofunction("We will meet at eight o'clock on Thursday morning.")
if __name__ == "__main__":
    stnfordpostagdemofunction("The name of your medicine is Paracetamol 500mg Tablets (called paracetamol throughout this leaflet).")<jupyter_output>Word            POS Tag
------------------------------
The             DT
name            NN
of              IN
your            PRP$
medicine        NN
is              VBZ
Paracetamol     NNP
500             CD
mg              NN
Tablets         NNS
-LRB-           -LRB-
called          VBN
paracetamol     NNP
throughout      IN
this            DT
leaflet         NN
-RRB-           -RRB-
.                .<jupyter_text>Generating POS tags using Spacy library  ```2.3.1 Import dependencies2.3.2 Load model2.3.3 Generate POS tag for given sentence```  2.3.1 Import dependencies<jupyter_code>import spacy<jupyter_output><empty_output><jupyter_text>2.3.2 Load model<jupyter_code>nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>2.3.3 Generate POS tag for given sentence<jupyter_code>doc = nlp(u'Apple is looking at buying U.K. startup for $1 billion')
print("{:<32}{:<32}{}".format("Word", "Word Category", "POS Tag")+"\n"+"-"*80)
for token in doc:
    #print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,token.shape_, token.is_alpha, token.is_stop)
    print(u"{:<32}{:<32}{}".format(token.text,token.pos_, token.tag_))<jupyter_output>Word                            Word Category                   POS Tag
--------------------------------------------------------------------------------
Apple                           PROPN                           NNP
is                              VERB                            VBZ
looking                         VERB                            VBG
at                              ADP                             IN
buying                          VERB                            VBG
U.K.                            PROPN                           NNP
startup                         NOUN                            NN
for                             ADP                             IN
$                               SYM                             $
1                               NUM                             CD
billion                         NUM                             CD<jupyter_text>Why do we need to develop our own POS tagger?  * Dealing with domain specific terminology* Dealing with ambiguity<jupyter_code>doc = nlp(u'The name of your medicine is Paracetamol 500mg Tablets (called paracetamol throughout this leaflet). ')
print("{:<32}{:<32}{}".format("Word", "Word Category", "POS Tag")+"\n"+"-"*80)
for token in doc:
    #print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,token.shape_, token.is_alpha, token.is_stop)
    print(u"{:<32}{:<32}{}".format(token.text,token.pos_, token.tag_))<jupyter_output>Word                            Word Category                   POS Tag
--------------------------------------------------------------------------------
The                             DET                             DT
name                            NOUN                            NN
of                              ADP                             IN
your                            ADJ                             PRP$
medicine                        NOUN                            NN
is                              VERB                            VBZ
Paracetamol                     PROPN                           NNP
500                             NUM                             CD
mg                              ADJ                             JJ
Tablets                         NOUN                            NNS
(                               PUNCT                           -LRB-
called                          VERB                            VBN
paracetamol                     NO[...]
{'fraction_non_alphanumeric': 0.13142713819914723, 'fraction_numerical': 0.020817657386506146, 'mean_word_length': 1.7642980935875217, 'pattern_counts': {'":': 4, '<': 68, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '47093', 'license': 'MIT', 'n_tokens_mistral': 2433, 'n_tokens_neox': 2301, 'n_words': 600, 'path': 'POS-tag-workshop-master/POS_tagger_Demo.ipynb', 'repo_name': 'RITUait/NLP-Projects', 'size': '7985'}