<jupyter_start><jupyter_text>**Urud Named Entity Recognition**by **<NAME>**  **Part 1: Using Libraries***   **SpaCy** (for NER)*   **StanfordNLP** (for Urdu PoS)  Named Entity Recognizer (NER)NER locate and classify named entity mentions in unstructured text into pre-defined categories, such as:*   Person*   Location*   Organization*   Date*   ...*italicized text*  **Using NER**SpaCy is a free open-source library for Natural Language Processing in Python.<jupyter_code>#!pip install -U spacy
# Uncomment the above statement, if you are not in colab and spacy is not installed there
import spacy<jupyter_output><empty_output><jupyter_text>Loading model of English language<jupyter_code>nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>Finding Named Entities<jupyter_code>doc = nlp(u'<NAME> has been retained as Pakistan captain while Babar Azam has been named as the vice captain \
            for the home series against Sri Lanka, a press release by Pakistan Cricket Board (PCB) said. \
            The final squads will be named on September 23.')
[(ent.text, ent.label_) for ent in doc.ents]<jupyter_output><empty_output><jupyter_text>Python Syntax<jupyter_code>#[(ent.text, ent.label_) for ent in doc.ents]
NElist = []
for ent in doc.ents:
  NElist.append((ent.text, ent.label_))
print(NElist)<jupyter_output>[('<NAME>', 'PERSON'), ('Pakistan', 'GPE'), ('Babar Azam', 'ORG'), ('Sri Lanka', 'GPE'), ('Pakistan Cricket Board', 'ORG'), ('PCB', 'ORG'), ('September 23', 'DATE')]<jupyter_text>Part of Speech (PoS)*   Universal Tagset (ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, SYM, VERB, X)*   Language Specific Tagset Urdu PoS Tagger is available in StanfordNLPInstalling Stanford NLP<jupyter_code>!pip install StanfordNLP<jupyter_output>Collecting StanfordNLP
[?25l  Downloading https://files.pythonhosted.org/packages/41/bf/5d2898febb6e993fcccd90484cba3c46353658511a41430012e901824e94/stanfordnlp-0.2.0-py3-none-any.whl (158kB)

[K     |██                              | 10kB 19.2MB/s eta 0:00:01
[K     |████▏                           | 20kB 3.3MB/s eta 0:00:01
[K     |██████▏                         | 30kB 4.8MB/s eta 0:00:01
[K     |████████▎                       | 40kB 3.1MB/s eta 0:00:01
[K     |██████████▎                     | 51kB 3.8MB/s eta 0:00:01
[K     |████████████▍                   | 61kB 4.5MB/s eta 0:00:01
[K     |██████████████▌                 | 71kB 5.2MB/s eta 0:00:01
[K     |████████████████▌               | 81kB 5.8MB/s eta 0:00:01
[K     |██████████████████▋             | 92kB 6.5MB/s eta 0:00:01
[K     |████████████████████▋           | 102kB 5.1MB/s eta 0:00:01
[K     |██████████████████████▊         | 112kB 5.1MB/s eta 0:00:01
[K     |████████████████████████▊       | 122kB 5.1MB/[...]<jupyter_text>Downloading model for Urdu``` This is formatted as code```<jupyter_code>import stanfordnlp
stanfordnlp.download('ur')<jupyter_output>Using the default treebank "ur_udtb" for language "ur".
Would you like to download the models for: ur_udtb now? (Y/n)
y

Default download directory: /root/stanfordnlp_resources
Hit enter to continue or type an alternate directory.


Downloading models for: ur_udtb
Download location: /root/stanfordnlp_resources/ur_udtb_models.zip<jupyter_text>Using NLP for Urdu<jupyter_code>nlp = stanfordnlp.Pipeline(lang="ur")
doc = nlp("لڑکیاں لائبریری میں کتابیں پڑھ رہی تھیں۔")<jupyter_output>Use device: cpu
---
Loading: tokenize
With settings: 
{'model_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb_tokenizer.pt', 'lang': 'ur', 'shorthand': 'ur_udtb', 'mode': 'predict'}
---
Loading: pos
With settings: 
{'model_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb_tagger.pt', 'pretrain_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb.pretrain.pt', 'lang': 'ur', 'shorthand': 'ur_udtb', 'mode': 'predict'}
---
Loading: lemma
With settings: 
{'model_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb_lemmatizer.pt', 'lang': 'ur', 'shorthand': 'ur_udtb', 'mode': 'predict'}
Building an attentional Seq2Seq model...
Using a Bi-LSTM encoder
Using soft attention for LSTM.
Finetune all embeddings.
[Running seq2seq lemmatizer with edit classifier]
---
Loading: depparse
With settings: 
{'model_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb_parser.pt', 'pretrain_path': '/root/stanfordnlp_resources/ur_udtb_models/ur_udtb.pretrain.pt', 'lang': '[...]<jupyter_text>Linguistic Features of the words<jupyter_code>for s in doc.sentences:
  for w in s.words:
    print(w.text, " : ", w)<jupyter_output>لڑکیاں  :  <Word index=1;text=لڑکیاں;lemma=لڑکی;upos=NOUN;xpos=NN;feats=Case=Nom|Gender=Masc|Number=Plur|Person=3;governor=5;dependency_relation=nsubj>
لائبریری  :  <Word index=2;text=لائبریری;lemma=لائبریری;upos=NOUN;xpos=NN;feats=Case=Acc|Gender=Fem|Number=Sing|Person=3;governor=5;dependency_relation=advmod>
میں  :  <Word index=3;text=میں;lemma=میں;upos=ADP;xpos=PSP;feats=AdpType=Post;governor=2;dependency_relation=case>
کتابیں  :  <Word index=4;text=کتابیں;lemma=کتاب;upos=NOUN;xpos=NN;feats=Case=Nom|Gender=Masc|Number=Plur|Person=3;governor=5;dependency_relation=obj>
پڑھ  :  <Word index=5;text=پڑھ;lemma=پڑھ;upos=VERB;xpos=VM;feats=Voice=Act;governor=0;dependency_relation=root>
رہی  :  <Word index=6;text=رہی;lemma=رہ;upos=AUX;xpos=VAUX;feats=Aspect=Perf|Gender=Fem|Number=Sing|VerbForm=Part;governor=5;dependency_relation=aux>
تھیں  :  <Word index=7;text=تھیں;lemma=تھا;upos=AUX;xpos=VAUX;feats=Gender=Fem|Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin;governor=6;dependency_relati[...]<jupyter_text>**Exercise:** Print the UPoS and XPoS corresponding to the each word in the following format.word =  لڑکیاں 		,univ pos :  NOUN 		 ,other pos :  NNword =  لائبریری 		,univ pos :  NOUN 		 ,other pos :  NNword =  میں 		,univ pos :  ADP 		 ,other pos :  PSPword =  کتابیں 		,univ pos :  NOUN 		 ,other pos :  NNword =  پڑھ 		,univ pos :  VERB 		 ,other pos :  VMword =  رہی 		,univ pos :  AUX 		 ,other pos :  VAUXword =  تھیں 		,univ pos :  AUX 		 ,other pos :  VAUXword =  ۔ 		,univ pos :  PUNCT 		 ,other pos :  SYM<jupyter_code>#write code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15810723514211886, 'fraction_numerical': 0.029877260981912145, 'mean_word_length': 4.635122838944495, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '45548', 'license': 'MIT', 'n_tokens_mistral': 2787, 'n_tokens_neox': 2516, 'n_words': 499, 'path': 'CLT20 ws1.ipynb', 'repo_name': 'khazeena/Urdu-NER', 'size': '6215'}