<jupyter_start><jupyter_text>**ShallowNLP** TutorialThe **ShallowNLP** module in *ktrain* is a small collection of text-analytic utilities to help analyze text data in English, Chinese, Russian, and other languages.   All methods in **ShallowNLP** are for use on a normal laptop CPU - no GPUs are required. Thus, it is well-suited to those with minimal computational resources and no GPU access.   Let's begin by importing the `shallownlp` module.<jupyter_code>from ktrain.text import shallownlp as snlp<jupyter_output>Using DISABLE_V2_BEHAVIOR with TensorFlow
using Keras version: 2.2.4-tf<jupyter_text>SECTION 1: Ready-to-Use Named-Entity-Recognition**ShallowNLP** includes pre-trained Named Entity Recognition (NER) for English, Chinese, and Russian. English NERExtracting entities from:><NAME> was head of the Chinese Academy of Medical Sciences and is the current president of Nankai University.<jupyter_code>ner = snlp.NER('en')
text = """
<NAME> was head of the Chinese Academy of Medical Sciences and is 
the current president of Nankai University.
"""
ner.predict(text)<jupyter_output><empty_output><jupyter_text>The `ner.predict` method automatically merges tokens by entity. To see the unmerged results, set `merge_tokens=False`:<jupyter_code>ner.predict(text, merge_tokens=False)<jupyter_output><empty_output><jupyter_text>The `ner.predict` method typically operates on single sentences, as in the example above.  For multi-sentence documents, sentences can be extracted with `snlp.sent_tokenize`:<jupyter_code>document = """<NAME> is a great actor.  <NAME> is not."""
sents = []
for idx, sent in enumerate(snlp.sent_tokenize(document)):
    sents.append(sent)
    print('sentence #%d: %s' % (idx+1, sent))
ner.predict(sents[0])
ner.predict(sents[1])<jupyter_output><empty_output><jupyter_text>Chinese NERExtracting entities from the Chinese translation of:><NAME> was  head of the Chinese Academy of Medical Sciences and is the current president of Nankai University.<jupyter_code>ner = snlp.NER('zh')
ner.predict('曹雪涛曾任中国医学科学院院长，现任南开大学校长。')<jupyter_output><empty_output><jupyter_text>Discovered entities with English translations:- 曹雪涛 = Cao Xuetao (PER)- 中国医学科学院  = Chinese Academy of Medical Sciences (ORG)- 南开大学 = Nankai University (ORG)The `snlp.sent_tokenize` can also be used with Chinese documents:<jupyter_code>document = """这是关于史密斯博士的第一句话。第二句话是关于琼斯先生的。"""
for idx, sent in enumerate(snlp.sent_tokenize(document)):
    print('sentence #%d: %s' % (idx+1, sent))<jupyter_output>sentence #1: 这是关于史密斯博士的第一句话。
sentence #2: 第二句话是关于琼斯先生的。<jupyter_text>Russian NERExtracting entities from the Russian translation of:><NAME>, the youngest daughter of Russian President <NAME>, was appointed head of a new artificial intelligence institute at Moscow State University.<jupyter_code>ner = snlp.NER('ru')
russian_sentence = """<NAME>, младшая дочь президента России Владимира Путина, 
была назначена руководителем нового института искусственного интеллекта в МГУ."""
ner.predict(russian_sentence)<jupyter_output><empty_output><jupyter_text>Discovered entities with English translations:- <NAME> = <NAME> (PER)- России = Russia (LOC)- Владимира Путина = Vladimir Putin (PER)- МГУ = Moscow State University (ORG)  SECTION 2: Text Classification**ShallowNLP** makes it easy to build a text classifier with minimal computational resources.  **ShallowNLP** includes the following sklearn-based text classification models: a non-neural version of [NBSVM](https://nlp.stanford.edu/pubs/sidaw12_simple_sentiment.pdf), Logistic Regression, and [Linear SVM with SGD training (SGDClassifier)](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html). Logistic regression is the default classifier.  For these examples, we will use [NBSVM](https://nlp.stanford.edu/pubs/sidaw12_simple_sentiment.pdf).A classifier can be trained with minimal effort for both English and Chinese. English Text ClassificationWe'll use the IMDb movie review dataset [available here](https://ai.stanford.edu/~amaas/data/sentiment/) to build a sentiment analysis model for English.<jupyter_code>datadir = r'/home/amaiya/data/aclImdb'
(x_train,  y_train, label_names) = snlp.Classifier.load_texts_from_folder(datadir+'/train')
(x_test,  y_test, _) = snlp.Classifier.load_texts_from_folder(datadir+'/test', shuffle=False)
print('label names: %s' % (label_names))
clf = snlp.Classifier().fit(x_train, y_train, ctype='nbsvm')
print('validation accuracy: %s%%' % (round(clf.evaluate(x_test, y_test)*100, 2)))
pos_text = 'I loved this movie because it was hilarious.'
neg_text = 'I hated this movie because it was boring.'
print('prediction for "%s": %s (pos)' % (pos_text, clf.predict(pos_text)))
print('prediction for "%s": %s (neg)' % (neg_text, clf.predict(neg_text)))<jupyter_output>label names: ['neg', 'pos']
validation accuracy: 92.03%
prediction for "I loved this movie because it was hilarious.": 1 (pos)
prediction for "I hated this movie because it was boring.": 0 (neg)<jupyter_text>Chinese Text ClassificationWe'll use the hotel review dataset [available here](here:https://github.com/Tony607/Chinese_sentiment_analysis/tree/master/data/ChnSentiCorp_htl_ba_6000) to build a sentiment analysis model for Chinese.<jupyter_code>datadir = '/home/amaiya/data/ChnSentiCorp_htl_ba_6000'
(texts,  labels, label_names) = snlp.Classifier.load_texts_from_folder(datadir+'/train')
print('label names: %s' % (label_names))
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(texts, labels, test_size=0.1, random_state=42)
clf = snlp.Classifier().fit(x_train, y_train, ctype='nbsvm')
print('validation accuracy: %s%%' % (round(clf.evaluate(x_test, y_test)*100, 2)))
pos_text = '我喜欢这家酒店，因为它很干净。'  # I loved this hotel because it was very clean.
neg_text = '我讨厌这家酒店，因为它很吵。'  # I hated this hotel because it was noisy.
print('prediction for "%s": %s' % (pos_text, clf.predict(pos_text)))
print('prediction for "%s": %s' % (neg_text, clf.predict(neg_text)))<jupyter_output>Building prefix dict from the default dictionary ...
I0304 17:09:57.303427 140636486911808 __init__.py:111] Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
I0304 17:09:57.305477 140636486911808 __init__.py:131] Loading model from cache /tmp/jieba.cache<jupyter_text>Tuning Hyperparameters of a Text ClassifierThe hyperparameters of a particular classifier can be tuned using the `grid_search` method.  Let's tune the **C** hyperparameter of a Logistic Regression model to see what is the best value for this dataset.<jupyter_code># setup data
datadir = r'/home/amaiya/data/aclImdb'
(x_train,  y_train, label_names) = snlp.Classifier.load_texts_from_folder(datadir+'/train')
(x_test,  y_test, _) = snlp.Classifier.load_texts_from_folder(datadir+'/test', shuffle=False)

# initialize a model to optimize
clf = snlp.Classifier()
clf.create_model('logreg', x_train)

# create parameter space for values of C
parameters = {'clf__C': (1e0, 1e-1, 1e-2)}

# tune
clf.grid_search(parameters, x_train[:5000], y_train[:5000], n_jobs=-1)<jupyter_output>clf__C: 1.0<jupyter_text>It looks like a value of `1.0` is best. We can then re-create the model with this hyperparameter value and proceed to train normally:```pythonclf.create_model('logreg', x_train, hp_dict={'C':1.0})clf.fit(x_train, y_train)clf.evaluate(x_test, y_test)```  SECTION 3: Examples of Searching TextHere we will show some simple searches over multi-language documents.<jupyter_code>document1 ="""
Hello there,

Hope this email finds you well.

Are you available to talk about our meeting?

If so, let us plan to schedule the meeting
at the Hefei National Laboratory for Physical Sciences at the Microscale.

As I always say: живи сегодня надейся на завтра

Sincerely,
<NAME>
合肥微尺度国家物理科学实验室
"""

document2 ="""
This is a random document with Arabic about our meeting.

عش اليوم الأمل ليوم غد

Bye for now.
"""

docs = [document1, document2]<jupyter_output><empty_output><jupyter_text>Searching EnglishThe `search` function returns a list of documents that match query. Each entry shows:1. the ID of the document2. the query (multiple queries can be supplied in a list, if desired)3. the number of word hits in the document<jupyter_code>snlp.search(['physical sciences', 'meeting', 'Arabic'], docs, keys=['doc1', 'doc2'])<jupyter_output><empty_output><jupyter_text>Searching ChineseThe `search` function returns a list of documents that match query. Each entry shows:1. the ID of the document2. the query3. the number of word hits in the document<jupyter_code>snlp.search('合肥微尺度国家物理科学实验室', docs, keys=['doc1', 'doc2'])<jupyter_output><empty_output><jupyter_text>For Chinese, the number of word hits is the number of words in the query that appear in the document. Seven of the words in the string 合肥微尺度国家物理科学实验室 were found in `doc1`.  Other SearchesThe `search` function can also be used for other languages. Arabic<jupyter_code>for result in snlp.search('عش اليوم الأمل ليوم غد', docs, keys=['doc1', 'doc2']):
    print("doc id:%s"% (result[0]))
    print('query:%s' % (result[1]))
    print('# of matches in document:%s' % (result[2]))<jupyter_output>doc id:doc2
query:عش اليوم الأمل ليوم غد
# of matches in document:1<jupyter_text>Russian<jupyter_code>snlp.search('сегодня надейся на завтра', docs, keys=['doc1', 'doc2'])<jupyter_output><empty_output><jupyter_text>Extract Chinese, Russian, or Arabic from mixed-language documents<jupyter_code>snlp.find_chinese(document1)
snlp.find_russian(document1)
snlp.find_arabic(document2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11064761706121598, 'fraction_numerical': 0.016894358118677653, 'mean_word_length': 6.480499219968799, 'pattern_counts': {'":': 6, '<': 70, '<?xml version=': 0, '>': 73, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '513006', 'license': 'Apache-2.0', 'n_tokens_mistral': 3409, 'n_tokens_neox': 3319, 'n_words': 956, 'path': 'examples/text/shallownlp-examples.ipynb', 'repo_name': 'xirect/ktrain', 'size': '9668'}