<jupyter_start><jupyter_text>Using Text Data with EvalMLIn this demo, we will show you how to use EvalML to build models which use text data.<jupyter_code>import evalml
from evalml import AutoMLSearch<jupyter_output><empty_output><jupyter_text>DatasetWe will be utilizing a dataset of SMS text messages, some of which are categorized as spam, and others which are not ("ham"). This dataset is originally from [Kaggle](https://www.kaggle.com/uciml/sms-spam-collection-dataset), but modified to produce a slightly more even distribution of spam to ham.<jupyter_code>from urllib.request import urlopen
import pandas as pd

input_data = urlopen('https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv')
data = pd.read_csv(input_data)

X = data.drop(['Category'], axis=1)
y = data['Category']

display(X.head())<jupyter_output><empty_output><jupyter_text>The ham vs spam distribution of the data is 3:1, so any machine learning model must get above 75% [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precisionIn_binary_classification) in order to perform better than a trivial baseline model which simply classifies everything as ham.<jupyter_code>y.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Search for best pipeline In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.<jupyter_code>X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=0.2, random_seed=0)<jupyter_output><empty_output><jupyter_text>EvalML uses [Woodwork](https://woodwork.alteryx.com/en/stable/) to automatically detect which columns are text columns, so you can run search normally, as you would if there was no text data. We can print out the logical type of the `Message` column and assert that it is indeed inferred as a natural language column.<jupyter_code>X_train.types<jupyter_output><empty_output><jupyter_text>Because the spam/ham labels are binary, we will use `AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')`. When we call `.search()`, the search for the best pipeline will begin.<jupyter_code>automl = AutoMLSearch(X_train=X_train, y_train=y_train,
                      problem_type='binary',
                      max_batches=1,
                      optimize_thresholds=True)

automl.search()<jupyter_output><empty_output><jupyter_text>View rankings and select pipelineOnce the fitting process is done, we can see all of the pipelines that were searched.<jupyter_code>automl.rankings<jupyter_output><empty_output><jupyter_text>To select the best pipeline we can call `automl.best_pipeline`.<jupyter_code>best_pipeline = automl.best_pipeline<jupyter_output><empty_output><jupyter_text>Describe pipelineYou can get more details about any pipeline, including how it performed on other objective functions.<jupyter_code>automl.describe_pipeline(automl.rankings.iloc[0]["id"])
best_pipeline.graph()<jupyter_output><empty_output><jupyter_text>Notice above that there is a `Text Featurization Component` as the second step in the pipeline. The Woodwork `DataTable` passed in to AutoML search recognizes that `'Message'` is a text column, and converts this text into numerical values that can be handled by the estimator.  Evaluate on holdout Now, we can score the pipeline on the holdout data using the core objectives for binary classification problems.<jupyter_code>scores = best_pipeline.score(X_holdout, y_holdout,  objectives=evalml.objectives.get_core_objectives('binary'))
print(f'Accuracy Binary: {scores["Accuracy Binary"]}')<jupyter_output><empty_output><jupyter_text>As you can see, this model performs relatively well on this dataset, even on unseen data.  Why encode text this way?To demonstrate the importance of text-specific modeling, let's train a model with the same dataset, without letting `AutoMLSearch` detect the text column. We can change this by explicitly setting the data type of the `'Message'` column in Woodwork to `Categorical` using the utility method `infer_feature_types`.<jupyter_code>from evalml.utils import infer_feature_types
X = infer_feature_types(X, {'Message': 'Categorical'})
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=0.2, random_seed=0)
automl_no_text = AutoMLSearch(X_train=X_train, y_train=y_train,
                              problem_type='binary',
                              max_batches=1,
                              optimize_thresholds=True)

automl_no_text.search()<jupyter_output><empty_output><jupyter_text>Like before, we can look at the rankings and pick the best pipeline.<jupyter_code>automl_no_text.rankings
best_pipeline_no_text = automl_no_text.best_pipeline<jupyter_output><empty_output><jupyter_text>Here, changing the data type of the text column removed the `Text Featurization Component` from the pipeline.<jupyter_code>best_pipeline_no_text.graph()
automl_no_text.describe_pipeline(automl_no_text.rankings.iloc[0]["id"])
# get standard performance metrics on holdout data
scores = best_pipeline_no_text.score(X_holdout, y_holdout, objectives=evalml.objectives.get_core_objectives('binary'))
print(f'Accuracy Binary: {scores["Accuracy Binary"]}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08344320964400076, 'fraction_numerical': 0.0030137502354492373, 'mean_word_length': 6.108433734939759, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '380265', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1620, 'n_tokens_neox': 1587, 'n_words': 482, 'path': 'docs/source/demos/text_input.ipynb', 'repo_name': 'skvorekn/evalml', 'size': '5309'}