<jupyter_start><jupyter_text>AML Package for Text Analytics - Text Classification In this notebook, you can learn how to train and deploy a text classification model with **Azure Machine Learning Package for Text Analytics** (AMLPTA). The goal of text classification is to assign a piece of text to one or more predefined classes or categories. This text could, for example, be a document, news article, search query, email, tweet, support tickets.There are broad applications of text classification such as: + Categorizing newspaper articles and news wire contents into topics+ Organizing web pages into hierarchical categories + Filtering spam email+ Sentiment analysis+ Predicting user intent from search queries+ Routing support tickets+ Analyzing customer feedback The text classification model building and deployment workflow for a model with AMLPTA is as follows:1. Load the data2. Train the model3. Apply the classifier 4. Evaluate model performance5. Save the pipeline6. Test the pipeline8. Deploy the model as a web serviceConsult the [package reference documentation](https://aka.ms/aml-packages/text) for the detailed reference for each module and class.The sample code in this notebook uses a scikit-learn pipeline.  Prerequisites 1. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.1. The following accounts and application must be set up and installed:   - Azure Machine Learning Experimentation account    - An Azure Machine Learning Model Management account   - Azure Machine Learning Workbench installed   If these three are not yet created or installed, follow the [Azure Machine Learning Quickstart and Workbench installation](../service/quickstart-installation.md) article. 1. The Azure Machine Learning Package for Text Analytics must be installed. Learn how to [install this package here](https://aka.ms/aml-packages/text).  Sample data and Jupyter notebook Download and Explore the sample dataThe following example uses the [20 newsgroups dataset](http://qwone.com/~jason/20Newsgroups/)  that is available through the scikit-learn library to demonstrate how to create a text classifier with Azure Machine Learning Package for Text Analytics. The 20 newsgroups dataset has around 18,000 newsgroups posts on 20 different topics divided into two subsets: one for training and the other one for performance evaluation. The split into train and test is based upon each message post date whether before or after a specific date.<jupyter_code># Import Packages 
# Use Azure Machine Learning history magic to control history collection
# History is off by default, options are "on", "off", or "show"
#%azureml history on
%matplotlib inline
# Use the Azure Machine Learning data collector to log various metrics
from azureml.logging import get_azureml_logger
import os
import warnings
logger = get_azureml_logger()

# Log cell runs into run history
logger.log('Cell','Set up run')

warnings.filterwarnings("ignore")<jupyter_output>---
Metadata-Version: 2.0
Name: azureml-tatk
Version: 0.1.18121.30a1
Summary: Microsoft Azure Machine Learning Package for Text Analytics
Home-page: https://microsoft.sharepoint.com/teams/TextAnalyticsPackagePreview
Author: Microsoft Corporation
Author-email: <EMAIL>
Installer: pip
License: UNKNOWN
Location: c:\users\tatk\appdata\local\amlworkbench\python\lib\site-packages
Requires: ruamel.yaml, dill, gensim, sklearn-crfsuite, bqplot, h5py, pdfminer.six, azure-storage, pytest, ipywidgets, lxml, qgrid, keras, requests, docker, matplotlib, ipython, pandas, nltk, pyspark, scikit-learn, scipy, jsonpickle, numpy, nose, azure-ml-api-sdk, validators, unidecode
Classifiers:<jupyter_text>Set the location of the dataSet the location where you have downloaded the data and load them into pandas dataframes.You can also use your own data where the input data file is *.tsv format.<jupyter_code>import os
import pandas as pd

#set the working directory where to save the training data files
resources_dir = os.path.join(os.path.expanduser("~"), "tatk", "resources")
data_dir = os.path.join(os.path.expanduser("~"), "tatk", "data")

from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(data_home=data_dir, subset='train')
X_train, y_train = twenty_train.data, twenty_train.target
df_train = pd.DataFrame({"text":X_train, "label":y_train})

twenty_test = fetch_20newsgroups(data_home=data_dir, subset='test')
X_test, y_test = twenty_test.data, twenty_test.target   
df_test = pd.DataFrame({"text":X_test, "label":y_test})

# Training Dataset Location
#training_file_path = <specify-your-own-training-data-file-path-here>
# df_train = pd.read_csv(training_file_path,
#                        sep = '\t',                        
#                        header = 0, names= <specify-your-column-name-list-here>)
df_train.head()
print("df_train.shape= {}".format(df_train.shape))

# Test Dataset Location
#test_file_path = <specify-your-own-test-data-file-path-here>
# df_test = pd.read_csv(test_file_path,
#                       sep = '\t',                        
#                       header = 0, names= <specify-your-column-name-list-here>)

print("df_test.shape= {}".format(df_test.shape))
df_test.head()<jupyter_output>df_train.shape= (11314, 2)
df_test.shape= (7532, 2)<jupyter_text>Get the correspondance between categories and their name.<jupyter_code>int_to_categories = pd.DataFrame({'category':range(20), 'category_name': list(twenty_train.target_names)})
int_to_categories<jupyter_output><empty_output><jupyter_text>Now, you can create a preliminary exploration plot histogram of the class frequency in training and test data sets.<jupyter_code>import numpy as np
import math
from matplotlib import pyplot as plt

data = df_train["label"].values
labels = set(data)
print(labels)
bins = range(len(labels)+1) 

plt.hist(data, bins=bins, alpha=0.8, rwidth=0.5)
plt.title('training data distribution over the class labels)')
plt.xlabel('class label')
plt.ylabel('frequency')
plt.grid(True)
plt.show()

data = df_test["label"].values
labels = set(data)
print(labels)
bins = range(len(labels)+1) 


plt.hist(data, bins=bins, alpha=0.8, rwidth=0.5)
plt.title('test data distribution over the class labels)')
plt.xlabel('class label')
plt.ylabel('frequency')
plt.grid(True)
plt.show()<jupyter_output>{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}<jupyter_text>Train the model Specify scikit-learn algorithm and define the text classifierThis step involves training a scikit-learn text classification model using One-versus-Rest LogisticRegression learning algorithm.For the full list of learnings, refer to the [Scikit Learners](http://scikit-learn.org/stable/supervised_learning) documentation.<jupyter_code>from sklearn.linear_model import LogisticRegression
import tatk
from tatk.pipelines.text_classification.text_classifier import TextClassifier

# Define the scikit-learn estimator.
log_reg_learner = LogisticRegression(penalty='l2', dual=False,
                                     tol=0.0001, C=1.0, fit_intercept=True,
                                     intercept_scaling=1, class_weight=None,
                                     random_state=None, solver='lbfgs', 
                                     max_iter=100, multi_class='ovr',
                                     verbose=1, warm_start=False, n_jobs=3) 

# Define the text classifier model.
# It will fit the learner on the text column "text".
text_classifier = TextClassifier(estimator=log_reg_learner, 
                                 text_cols = ["text"], 
                                 label_cols = ["label"], 
#                                numeric_cols = None,
#                                cat_cols = None, 
                                 extract_word_ngrams=True,
                                 extract_char_ngrams=True)<jupyter_output>'pattern' package not found; tag filters are not available for English
TextClassifier::create_pipeline ==> start
:: number of jobs for the pipeline : 6
0	text_nltk_preprocessor
1	text_word_ngrams
2	text_char_ngrams
3	assembler
4	learner
TextClassifier::create_pipeline ==> end<jupyter_text>Fit the modelUse the default parameters of the package. By default, the text classifier extracts:+ Word unigrams and bigrams+ Character 4 grams<jupyter_code>text_classifier.fit(df_train)<jupyter_output>TextClassifier::fit ==> start
schema: col=label:I4:0 col=text:TX:1 header+
NltkPreprocessor::tatk_fit_transform ==> start
NltkPreprocessor::tatk_fit_transform ==> end 	 Time taken: 0.08 mins
NGramsVectorizer::tatk_fit_transform ==> startNGramsVectorizer::tatk_fit_transform ==> start

			vocabulary size=216393
NGramsVectorizer::tatk_fit_transform ==> end 	 Time taken: 0.39 mins
			vocabulary size=67230
NGramsVectorizer::tatk_fit_transform ==> end 	 Time taken: 0.49 mins
VectorAssembler::transform ==> start, num of input records=11314
(11314, 216393)
(11314, 67230)
all_features::
(11314, 283623)
Time taken: 0.06 mins
VectorAssembler::transform ==> end
LogisticRegression::tatk_fit ==> start<jupyter_text>Examine and set the parameters of the different pipeline stepsAlthough the fitting of a scikit-learn model, preprocessing is being done prior to fitting using a pipeline of preprocessor and featurizer (transformation) steps. Hence, the reference to a "pipeline" for training. During evaluation, the full pipeline, including preprocessing and scikit-learn model prediction, is applied to a testing data set.***Example shown with text_word_ngrams*** Typically, you set the parameters before you fit a model. The following code samples show you how to train the model using the default pipeline and model parameters. To see what parameters are included for "text_word_ngrams", use [get_step_param_names_by_name](https://docs.microsoft.com/python/api/tatk.core.base_text_model.basetextmodel). This function returns the parameters such as lowercase, input_col, output_col and so on.<jupyter_code>text_classifier.get_step_param_names_by_name("text_word_ngrams")<jupyter_output><empty_output><jupyter_text>Next, check the parameter values for "text_char_ngrams":<jupyter_code>text_classifier.get_step_params_by_name("text_word_ngrams")<jupyter_output><empty_output><jupyter_text>If necessary, you can change the default parameters.  With the following code, you can change the range of extracted character n-grams from (4,4) to (3,4) to extract both character tri-grams and 4 grams:<jupyter_code>text_classifier.set_step_params_by_name("text_char_ngrams", ngram_range =(3,4)) 
text_classifier.get_step_params_by_name("text_char_ngrams")<jupyter_output><empty_output><jupyter_text>Export the parameters to a fileIf needed, you can optimize model performance by rerunning the model fitting step with revised parameters:<jupyter_code>import os
params_file_path = os.path.join(resources_dir, "params.tsv")
text_classifier.export_params(params_file_path)
print("Saving the configuration parameters to {}".format(params_file_path))<jupyter_output>Saving the configuration parameters to C:\Users\tatk\tatk\resources\params.tsv<jupyter_text>Apply the classifierApply the trained text classifier on the test dataset to generate class predictions:<jupyter_code>df_test = text_classifier.predict(df_test)
df_test.head()<jupyter_output>TextClassifier ::predict ==> start
NltkPreprocessor::tatk_transform ==> start
NltkPreprocessor::tatk_transform ==> end 	 Time taken: 0.05 mins
NGramsVectorizer::tatk_transform ==> startNGramsVectorizer::tatk_transform ==> start

NGramsVectorizer::tatk_transform ==> end 	 Time taken: 0.14 mins
NGramsVectorizer::tatk_transform ==> end 	 Time taken: 0.35 mins
VectorAssembler::transform ==> start, num of input records=7532
(7532, 216393)
(7532, 67230)
all_features::
(7532, 283623)
Time taken: 0.03 mins
VectorAssembler::transform ==> end
LogisticRegression::tatk_predict_proba ==> start
LogisticRegression::tatk_predict_proba ==> end 	 Time taken: 0.0 mins
LogisticRegression::tatk_predict ==> start
LogisticRegression::tatk_predict ==> end 	 Time taken: 0.0 mins
Time taken: 0.44 mins
TextClassifier ::predict ==> end
Order of Labels in predicted probabilities saved to attribute label_order of the class object<jupyter_text>Evaluate model performanceThe [evaluation module](https://docs.microsoft.com/python/api/tatk.evaluation) evaluates the accuracy of the trained text classifier on the test dataset. The evaluate function generates a confusion matrix and provides a macro-F1 score.<jupyter_code>evaluator = text_classifier.evaluate(df_test)<jupyter_output>TextClassifier ::evaluate ==> start
Time taken: 0.0 mins
TextClassifier ::evaluate ==> end<jupyter_text>Plot the confusion without normalization matrix for visualization.<jupyter_code>evaluator.plot_confusion_matrix(normalize=False,
                                title='Confusion matrix, without normalization', 
                                print_confusion_matrix=False,
                                figsize=(8,8),
                                colors=None)<jupyter_output>Confusion matrix, without normalization<jupyter_text>Plot the normalized confusion matrix for visualization.<jupyter_code>evaluator.plot_confusion_matrix(normalize=True,
                                title='Confusion matrix, with normalization', 
                                print_confusion_matrix=False,
                                figsize=(8,8),
                                colors=None)
evaluator.get_metrics('precision')
evaluator.get_metrics('micro_avg_accuracy')
evaluator.get_metrics('macro_avg_accuracy')<jupyter_output><empty_output><jupyter_text>Save the pipelineSave the classification pipeline into a zip file. Also, save the word-ngrams and character n-grams as text files.<jupyter_code>import os
working_dir = os.path.join(data_dir, 'outputs')  
if not os.path.exists(working_dir):
    os.makedirs(working_dir)

# you can save the trained model as a folder or a zip file
model_file = os.path.join(working_dir, 'sk_model.zip')    
text_classifier.save(model_file)
# %azureml upload outputs/models/sk_model.zip
working_dir
# for debugging, you can save the word n-grams vocabulary to a text file
word_vocab_file_path = os.path.join(working_dir, 'word_ngrams_vocabulary.tsv')
text_classifier.get_step_by_name("text_word_ngrams").save_vocabulary(word_vocab_file_path) 
# %azureml upload outputs/dictionaries/word_ngrams_vocabulary.pkl

# for debugging, you can save the character n-grams vocabulary to a text file
char_vocab_file_path = os.path.join(working_dir, 'char_ngrams_vocabulary.tsv')
text_classifier.get_step_by_name("text_char_ngrams").save_vocabulary(char_vocab_file_path) 
# %azureml upload outputs/dictionaries/char_ngrams_vocabulary.pkl<jupyter_output>save_vocabulary ==> start
saving 216393 n-grams ...
Time taken: 0.01 mins
save_vocabulary ==> end
save_vocabulary ==> start
saving 67230 n-grams ...
Time taken: 0.0 mins
save_vocabulary ==> end<jupyter_text>Load the pipelineLoad the classification pipeline and the word-ngrams and character n-grams for inferencing:<jupyter_code># in order to deploy the trained model, you have to load the zip file of the classifier pipeline
loaded_text_classifier = TextClassifier.load(model_file)

from tatk.feature_extraction import NGramsVectorizer
word_ngram_vocab = NGramsVectorizer.load_vocabulary(word_vocab_file_path)
char_ngram_vocab = NGramsVectorizer.load_vocabulary(char_vocab_file_path)<jupyter_output>BaseTextModel::load ==> start
TatkPipeline::load ==> start
Time taken: 0.13 mins
TatkPipeline::load ==> end
Time taken: 0.14 mins
BaseTextModel::load ==> end
loading 216393 n-grams ...
loading 67230 n-grams ...<jupyter_text>Test the pipelineTo evaluate a test dataset, apply the loaded text classification pipeline:<jupyter_code>predictions = loaded_text_classifier.predict(df_test)
loaded_evaluator = loaded_text_classifier.evaluate(predictions)
loaded_evaluator.get_metrics('macro_f1')<jupyter_output>TextClassifier ::predict ==> start
NltkPreprocessor::tatk_transform ==> start
NltkPreprocessor::tatk_transform ==> end 	 Time taken: 0.05 mins
NGramsVectorizer::tatk_transform ==> startNGramsVectorizer::tatk_transform ==> start

NGramsVectorizer::tatk_transform ==> end 	 Time taken: 0.15 mins
NGramsVectorizer::tatk_transform ==> end 	 Time taken: 0.35 mins
VectorAssembler::transform ==> start, num of input records=7532
(7532, 216393)
(7532, 67230)
all_features::
(7532, 283623)
Time taken: 0.03 mins
VectorAssembler::transform ==> end
LogisticRegression::tatk_predict_proba ==> start
LogisticRegression::tatk_predict_proba ==> end 	 Time taken: 0.0 mins
LogisticRegression::tatk_predict ==> start
LogisticRegression::tatk_predict ==> end 	 Time taken: 0.0 mins
Time taken: 0.44 mins
TextClassifier ::predict ==> end
Order of Labels in predicted probabilities saved to attribute label_order of the class object
TextClassifier ::evaluate ==> start
Time taken: 0.0 mins
TextClassifier ::evaluate ==>[...]<jupyter_text>Operationalization: deploy and consumeIn this section, you deploy the text classification pipeline as an Azure Machine Learning web service using [Azure Machine Learning CLI](https://docs.microsoft.com/azure/machine-learning/desktop-workbench/cli-for-azure-machine-learning). Then, you consume the web service for training and scoring.**Log in to your Azure subscription with Azure CLI**Using an [Azure](https://azure.microsoft.com/) account with a valid subscription, log in using the following CLI command:`az login`+ To switch to another Azure subscription, use the command:`az account set --subscription [your subscription name]`+ To see the current model management account, use the command:  `az ml account modelmanagement show`**Create and set your deployment environment**You only need to set your deployment environment once. If you don't have one yet, set up your deployment environment now using [these instructions](https://docs.microsoft.com/azure/machine-learning/desktop-workbench/deployment-setup-configurationenvironment-setup). 1- Make sure your Azure Machine Learning environment, model management account, and resource group are located in the same region.2- Download the deployment configuration file from Blob storage and save it locally:<jupyter_code># Download the deployment config file from Blob storage `url` and save it locally under `file_name`:
deployment_config_file_url = 'https://aztatksa.blob.core.windows.net/dailyrelease/tatk_deploy_config.yaml'
deployment_config_file_path=os.path.join(resources_dir, 'tatk_deploy_config.yaml')
if not os.path.exists(deployment_config_file_path):
    import urllib.request
    print('downloading {}'.format(deployment_config_file_path))
    urllib.request.urlretrieve(deployment_config_file_url, deployment_config_file_path)
deployment_config_file_path<jupyter_output>downloading C:\Users\tatk\tatk\resources\tatk_deploy_config.yaml<jupyter_text>3- Update the deployment configuration file you downloaded to reflect your resources:Please write the path to a local working directory.<jupyter_code>working_directory =r'<please provide a working directory on your machine here>'
web_service_name = 'azureml-text-classifier'

web_service = text_classifier.deploy(web_service_name= web_service_name, 
                       config_file_path=deployment_config_file_path,
                       working_directory= working_directory)  
print(text_classifier._aml_settings.azure_subscription)

print(text_classifier._aml_settings.model_management_account_name)
print(text_classifier._aml_settings.model_management_account_location)
print(text_classifier._aml_settings.model_management_account_resource_group)

print(text_classifier._aml_settings.env_name)
print(text_classifier._aml_settings.env_location)
print(text_classifier._aml_settings.env_resource_group)<jupyter_output>Azure Internal - Carl
tutorial4vienna-model-mgr
eastus2
ner-rg
env4textclassifier
eastus2
env4textclassifierrg<jupyter_text>Given that the trained model is deployed successfully. Let us invoke the scoring web service on new data  4- Given that the trained model is deployed successfully, get the URL and Key to invoke the web service.<jupyter_code>print("Service URL: {}".format(web_service._service_url))
print("Service Key: {}".format(web_service._api_key))<jupyter_output>Service URL: http://192.168.3.11/api/v1/service/azureml-text-classifier/score
Service Key: edYIppYDk2oGkRT9EvnZUorRfbTCBweO<jupyter_text>5- Load the web service at any time using the url and key above.<jupyter_code>from tatk.operationalization.csi.csi_web_service import CsiWebService
url = "<please type the service URL here>"
key = "<please type the service Key here>"
web_service = CsiWebService(url, key)<jupyter_output><empty_output><jupyter_text>6- Test the web service with the body of two emails taken from the 20 newsgrpoups dataset:<jupyter_code>import json
dict1 ={}
dict1["recordId"] = "a1" 
dict1["data"]= {}
dict1["data"]["text"] = """
I'd be interested in a copy of this code if you run across it.
(Mail to the author bounced)
 > / hpldsla:comp.graphics / email-address-removed / 12:53 am  May 13,
 1993 /
 > I fooled around with this problem a few years ago, and implemented a
 > simple method that ran on a PC.
 > was very simple - about 40 or 50 lines of code.
 . . .
 > Somewhere I still have it
 > and could dig it out if there was interest.
"""

dict2 ={}
dict2["recordId"] = "b2"
dict2["data"] ={}
dict2["data"]["text"] = """
>>Could the people discussing recreational drugs such as mj, lsd, mdma, etc.,
>>take their discussions to alt.drugs? Their discussions will receive greatest
>>contribution and readership there. The people interested in strictly
>>"smart drugs" (i.e. Nootropics) should post to this group. The two groups
>>(alt.drugs & alt.psychoactives) have been used interchangably lately.
>>I do think that alt.psychoactives is a deceiving name. alt.psychoactives
>>is supposedly the "smart drug" newsgroup according to newsgroup lists on
>>the Usenet. Should we establish an alt.nootropics or alt.sdn (smart drugs &
>>nutrients)? I have noticed some posts in sci.med.nutrition regarding
>>"smart nutrients." We may lower that groups burden as well.
>

I was wondering if a group called 'sci.pharmacology' would be relevent.
This would be used for a more formal discussion about pharmacological
issues (pharmacodynamics, neuropharmacology, etc.)

Just an informal proposal (I don't know anything about the net.politics
for adding a newsgroup, etc.)

"""

dict_list =[dict1, dict2]
data ={}
data["values"] = dict_list
input_data_json_str = json.dumps(data)
print (input_data_json_str)
prediction = web_service.score(input_data_json_str)
prediction<jupyter_output>{"values": [{"recordId": "a1", "data": {"text": "\nI'd be interested in a copy of this code if you run across it.\n(Mail to the author bounced)\n > / hpldsla:comp.graphics / email-address-removed / 12:53 am  May 13,\n 1993 /\n > I fooled around with this problem a few years ago, and implemented a\n > simple method that ran on a PC.\n > was very simple - about 40 or 50 lines of code.\n . . .\n > Somewhere I still have it\n > and could dig it out if there was interest.\n"}}, {"recordId": "b2", "data": {"text": "\n>>Could the people discussing recreational drugs such as mj, lsd, mdma, etc.,\n>>take their discussions to alt.drugs? Their discussions will receive greatest\n>>contribution and readership there. The people interested in strictly\n>>\"smart drugs\" (i.e. Nootropics) should post to this group. The two groups\n>>(alt.drugs & alt.psychoactives) have been used interchangably lately.\n>>I do think that alt.psychoactives is a deceiving name. alt.psychoactives\n>>is supposedly the \"sm[...]
{'fraction_non_alphanumeric': 0.08845861807137434, 'fraction_numerical': 0.017210832700582133, 'mean_word_length': 5.108477196598815, 'pattern_counts': {'":': 12, '<': 81, '<?xml version=': 0, '>': 180, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '440172', 'license': 'MIT', 'n_tokens_mistral': 7388, 'n_tokens_neox': 6926, 'n_words': 2206, 'path': 'domain-packages/text-analytics/Text_Classification_Single_Column_with_Sklearn.ipynb', 'repo_name': 'wjones30309/MachineLearningSamples-Notebooks', 'size': '23721'}