<jupyter_start><jupyter_text>To run this sample, please first install 'microsoftml' package.- For Windows: 'pip install https://rserverdistribution.azureedge.net/production/revoscalepy/9.2.1/wb/1033/d282048eb04046999211535f7368a0a4/windows/microsoftml-1.5.0-py3-none-any.whl'- 'microsoftml' support for Mac will come soon. Categorical FeaturesMany machine learned models only consider numerical featuresand many machine learning problem do have non numerical features.One kind is categories. Somebody's university is notnumerical, it is one item among a list of unordered possibilities.Before training any model, we need to convert that kind of featuresinto numerical features.One famous problem is the `adult data set `_.The goal is to predict whether somebody earns more than 50K a year or notbased on his age, education, occupation, relationship, ... Most of the features are categorical.<jupyter_code>columns = ["age", "workclass", "fnlwgt", "education", "educationnum", "maritalstatus",
           "occupation", "relationship", "race", "sex", "capitalgain", "capitalloss", 
           "hoursperweek", "nativecountry", "Label"]

import pandas
import os

def preprocess_data(df):
    # The data contains '?' for missing values.
    # We replace them and remove them.
    # We convert every numerical features into either str or float.
    # We remove extra spaces and put every thing in lower case.
    for c in df.columns:
        df[c] = df[c].apply(lambda x: numpy.nan if x == '?' else x)
    df = df.dropna()
    for c in df.columns:
        try:
            newc = df[c].astype(float)
            print("numerical", c)
        except Exception as e:
            print("categorical", c)
            newc = df[c].astype(str).apply(lambda s: s.strip(". ").lower())
        df[c] = newc
    return df

if os.path.exists("adult.train.csv"):
    train = pandas.read_csv("adult.train.csv")
else:
    train = pandas.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
                            header=None, names=columns)
    train = preprocess_data(train)
    # We store the data on disk to avoid loading it every time
    # we execute the script.
    train.to_csv("adult.train.csv", index=False)
print(train.head())<jupyter_output><empty_output><jupyter_text>We do the same for the test data.<jupyter_code>if os.path.exists("adult.test.csv"):
    test = pandas.read_csv("adult.test.csv")
else:
    test = pandas.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test",
                            header=None, names=columns)
    test = preprocess_data(test)
    # We store the data on disk to avoid loading it every time
    # we execute the script.
    test.to_csv("adult.test.csv", index=False)
    
print(test.head())<jupyter_output><empty_output><jupyter_text>We convert the label into boolean.<jupyter_code>train["Label"] = train["Label"] == ">50k"
test["Label"] = test["Label"] == ">50k"<jupyter_output><empty_output><jupyter_text>The data contains numerical and categorical features.Let's choose a random forest because it usually works better than a linear model on non continuous features.Let's first train a model on a couple of numerical features.<jupyter_code>from microsoftml import rx_fast_trees, rx_predict
trees = rx_fast_trees("Label ~ age + fnlwgt + educationnum + capitalgain + capitalloss", data=train)<jupyter_output><empty_output><jupyter_text>Let's see the :epkg:`confusion matrix`.<jupyter_code>y_pred = rx_predict(trees, test)
print(y_pred.head())
print(y_pred.tail())

from sklearn.metrics import confusion_matrix
conf = confusion_matrix(test["Label"], y_pred["PredictedLabel"])
print(conf)<jupyter_output><empty_output><jupyter_text>Not very good. We need to use categorical features.Let's choose education without any conversion.<jupyter_code>try:
    trees2 = rx_fast_trees("Label ~ age + fnlwgt + educationnum + capitalgain + capitalloss + education", data=train)
except Exception as e:
    print(e)<jupyter_output><empty_output><jupyter_text>As expected it fails. We need to convert it into a numerical feature.We have two options. The first one is to use :epkg:`scikit-learn` on the input dataframe(see `OneHotEncoder `_ for example).Or we can ask :epkg:`microsoftml` to that instead with :epkg:`microsoft:categorical`.We create a new variable *educationCat* and we replace it in the formula.<jupyter_code>from microsoftml import categorical

trees2 = rx_fast_trees("Label ~ age + fnlwgt + educationnum + capitalgain + capitalloss + education_cat",
                       data=train,
                       ml_transforms=[categorical(cols=dict(education_cat="education"))])<jupyter_output><empty_output><jupyter_text>We look into the :epkg:`confusion matrix`.<jupyter_code>y_pred2 = rx_predict(trees2, test)
conf = confusion_matrix(test["Label"], y_pred2["PredictedLabel"])
print(conf)<jupyter_output><empty_output><jupyter_text>Still not very good. We add more.<jupyter_code>cats = {}
for col in ["workclass", "education", "maritalstatus", "occupation", 
            "relationship", "race", "sex", "nativecountry"]:
    cats[col + "_cat"] = col

formula = "Label ~ age + fnlwgt + educationnum + capitalgain + capitalloss +" + \
          " + ".join(sorted(cats.keys()))
          
print(cats)
print(formula)

trees3 = rx_fast_trees(formula, data=train,
                       ml_transforms=[categorical(cols=cats)])
y_pred3 = rx_predict(trees3, test)
conf = confusion_matrix(test["Label"], y_pred3["PredictedLabel"])
print(conf)<jupyter_output><empty_output><jupyter_text>.. index:: ROCThis is better. We draw the :epkg:`ROC` curve.<jupyter_code>from sklearn.metrics import roc_curve
fpr, tpr, th = roc_curve(test["Label"], y_pred3["Probability"])

import matplotlib.pyplot as plt
plt.figure()
plt.plot(fpr, tpr, label=">50k")
plt.plot([0, 1], [0, 1], color='navy', linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title('ROC - Adult Data Set')
plt.legend(loc="lower right")<jupyter_output><empty_output><jupyter_text>The advantage of using the categorical transforminside the model is the data transformation does not have to be applied on the test data. It is part of the modelor the `pipeline `_to follow :epkg:`scikit-learn` concept.<jupyter_code>plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1203326011923439, 'fraction_numerical': 0.010668340131785378, 'mean_word_length': 4.567685589519651, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 48, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '154811', 'license': 'MIT', 'n_tokens_mistral': 2092, 'n_tokens_neox': 1974, 'n_words': 555, 'path': 'Notebooks/microsoftml_adult_census.ipynb', 'repo_name': 'Azure-Samples/MachineLearningSamples-mml', 'size': '6374'}