<jupyter_start><jupyter_text>BNLPBNLP is a natural language processing toolkit for Bengali Language. This tool will help you to tokenize Bengali text, Embedding Bengali words, Bengali POS Tagging, Construct Neural Model for Bengali NLP purposes.Here we are prodiving training approach of different model using **BNLP**  Installation<jupyter_code>!pip install -U bnlp_toolkit<jupyter_output>Collecting bnlp_toolkit
  Downloading https://files.pythonhosted.org/packages/16/be/44d78b55ad8121cce1ca0bdbc7cf1db8d3f585006bacb08bd53ec8653957/bnlp_toolkit-3.0.0-py3-none-any.whl
Requirement already satisfied, skipping upgrade: nltk in /usr/local/lib/python3.6/dist-packages (from bnlp_toolkit) (3.2.5)
Requirement already satisfied, skipping upgrade: gensim in /usr/local/lib/python3.6/dist-packages (from bnlp_toolkit) (3.6.0)
Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from bnlp_toolkit) (1.4.1)
Collecting sklearn-crfsuite
  Downloading https://files.pythonhosted.org/packages/25/74/5b7befa513482e6dee1f3dd68171a6c9dfc14c0eaa00f885ffeba54fe9b0/sklearn_crfsuite-0.3.6-py2.py3-none-any.whl
Collecting sentencepiece
[?25l  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
[K     |████████████████████████████[...]<jupyter_text>Downloading Bengali Processed Wikipedia Data<jupyter_code>#drive data download code
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
downloaded = drive.CreateFile({'id':"1rQUQLsXg0TZnlrAgmNMkCXGDnYbjlLmM"})
downloaded.GetContentFile('bn_wiki_data.txt.zip')

!unzip bn_wiki_data.txt.zip
!rm -rf bn_wiki_data.txt.zip<jupyter_output>Archive:  bn_wiki_data.txt.zip
  inflating: bn_wiki_data.txt<jupyter_text>TrainingHere we present `bengali sentencepiece`, `bengali word2vec`, `bengali fasttext` training on `bengali wikipedia data`Training time will depend on data size.  Training Bengali Sentencepice ModelAfter successfully compiling the below code will produce two file:* `wiki_sp.model` * `wiki_sp.vecab`<jupyter_code>from bnlp import SentencepieceTokenizer

bsp = SentencepieceTokenizer()
data = "bn_wiki_data.txt"
model_prefix = "wiki_sp"
vocab_size = 30000
bsp.train(data, model_prefix, vocab_size)<jupyter_output>punkt not found. downloading...
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
wiki_sp.model and wiki_sp.vocab is saved on your current directory<jupyter_text>Training Bengali Word2Vec ModelAfter successfully compiling it will produce three file. * `wiki_word2vec.model`* `wiki_word2vec.vector`* `wiki_word2vec.model.trainables.syn1neg.npy`* `wiki_word2vec..model.wv.vectors.npy`<jupyter_code>from bnlp import BengaliWord2Vec
bwv = BengaliWord2Vec()
data_file = "bn_wiki_data.txt"
model_name = "wiki_word2vec.model"
vector_name = "wiki_word2vec.vector"
bwv.train(data_file, model_name, vector_name)<jupyter_output>/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:252: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL<jupyter_text>Training Bengali Fasttext ModelFirst of all install `fasttext` using `pip install fasttext` and restart runtime.After successfully training it will produce: * `wiki_fasttext.bin`<jupyter_code>!pip install fasttext
from bnlp.embedding.fasttext import BengaliFasttext

bft = BengaliFasttext()
data = "bn_wiki_data.txt"
model_name = "wiki_fasttext.bin"
epoch = 1
bft.train(data, model_name, epoch)<jupyter_output><empty_output><jupyter_text>Training Bengali POS TAGGING CRF modelAfter successfully training it will produce a trained model with accuracy on evaluation data: * `pos_model.pkl`<jupyter_code>from bnlp import POS
bn_pos = POS()
model_name = "pos_model.pkl"
tagged_sentences = [[('রপ্তানি', 'JJ'), ('দ্রব্য', 'NC'), ('-', 'PU'), ('তাজা', 'JJ'), ('ও', 'CCD'), ('শুকনা', 'JJ'), ('ফল', 'NC'), (',', 'PU'), ('আফিম', 'NC'), (',', 'PU'), ('পশুচর্ম', 'NC'), ('ও', 'CCD'), ('পশম', 'NC'), ('এবং', 'CCD'),('কার্পেট', 'NC'), ('৷', 'PU')], [('মাটি', 'NC'), ('থেকে', 'PP'), ('বড়জোর', 'JQ'), ('চার', 'JQ'), ('পাঁচ', 'JQ'), ('ফুট', 'CCL'), ('উঁচু', 'JJ'), ('হবে', 'VM'), ('৷', 'PU')]]

bn_pos.train(model_name, tagged_sentences)<jupyter_output>1
1
Training Started........
it will take time according to your dataset size..
Training Finished!
Evaluating with Test Data...
Accuracy is: 
0.1111111111111111
Model Saved!<jupyter_text>Training Bengali NER modelAfter successfully training it will produce a trained model with accuracy on evaluation data:* `ner_model.pkl`<jupyter_code>from bnlp import NER
bn_ner = NER()
model_name = "ner_model.pkl"
tagged_sentences = [[('ত্রাণ', 'O'),('ও', 'O'),('সমাজকল্যাণ', 'O'),('সম্পাদক', 'S-PER'),('সুজিত', 'B-PER'),('রায়', 'I-PER'),('নন্দী', 'E-PER'),('প্রমুখ', 'O'),('সংবাদ', 'O'),('সম্মেলনে', 'O'),('উপস্থিত', 'O'),('ছিলেন', 'O')], [('ত্রাণ', 'O'),('ও', 'O'),('সমাজকল্যাণ', 'O'),('সম্পাদক', 'S-PER'),('সুজিত', 'B-PER'),('রায়', 'I-PER'),('নন্দী', 'E-PER'),('প্রমুখ', 'O'),('সংবাদ', 'O'),('সম্মেলনে', 'O'),('উপস্থিত', 'O'),('ছিলেন', 'O')], [('ত্রাণ', 'O'),('ও', 'O'),('সমাজকল্যাণ', 'O'),('সম্পাদক', 'S-PER'),('সুজিত', 'B-PER'),('রায়', 'I-PER'),('নন্দী', 'E-PER'),('প্রমুখ', 'O'),('সংবাদ', 'O'),('সম্মেলনে', 'O'),('উপস্থিত', 'O'),('ছিলেন', 'O')]]

bn_ner.train(model_name, tagged_sentences)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1836561331974176, 'fraction_numerical': 0.0326197757390418, 'mean_word_length': 7.153739612188366, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '373081', 'license': 'MIT', 'n_tokens_mistral': 2574, 'n_tokens_neox': 2549, 'n_words': 460, 'path': 'notebook/bnlp_colab_training.ipynb', 'repo_name': 'zarif98sjs/bnlp', 'size': '5886'}