<jupyter_start><jupyter_text>101 - Training and Evaluating Classifiers with `mmlspark`In this example, we try to predict incomes from the *Adult Census* dataset.First, we import the packages (use `help(mmlspark)` to view contents),<jupyter_code>import numpy as np
import pandas as pd
from pyspark.sql.types import DoubleType, StringType, StructField, StructType
from pyspark.sql import SparkSession
import pyspark
spark = SparkSession.builder.appName("MyApp") \
            .config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:0.18.1") \
            .config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
            .getOrCreate()
import mmlspark<jupyter_output><empty_output><jupyter_text>Now let's read the data and split it to train and test sets:<jupyter_code>schema = StructType([
  StructField("age", DoubleType(), False),
  StructField("workclass", StringType(), False),
  StructField("fnlwgt", DoubleType(), False),
  StructField("education", StringType(), False),
  StructField("education_num", DoubleType(), False),
  StructField("marital_status", StringType(), False),
  StructField("occupation", StringType(), False),
  StructField("relationship", StringType(), False),
  StructField("race", StringType(), False),
  StructField("sex", StringType(), False),
  StructField("capital_gain", DoubleType(), False),
  StructField("capital_loss", DoubleType(), False),
  StructField("hours_per_week", DoubleType(), False),
  StructField("native_country", StringType(), False),
  StructField("income", StringType(), False)
])

data = spark.read.format("csv").schema(schema).load("/home/robin/datatsets/adult/adult.data")
data = data.select(["education", "marital-status", "hours-per-week", "income"])
train, test = data.randomSplit([0.75, 0.25], seed=123)
train.limit(10).toPandas()<jupyter_output><empty_output><jupyter_text>`TrainClassifier` can be used to initialize and fit a model, it wraps SparkML classifiers.You can use `help(mmlspark.TrainClassifier)` to view the different parameters.Note that it implicitly converts the data into the format expected by the algorithm: tokenizeand hash strings, one-hot encodes categorical variables, assembles the features into a vectorand so on.  The parameter `numFeatures` controls the number of hashed features.<jupyter_code>from mmlspark.train import TrainClassifier
from pyspark.ml.classification import LogisticRegression
model = TrainClassifier(model=LogisticRegression(), labelCol="income", numFeatures=256).fit(train)<jupyter_output><empty_output><jupyter_text>After the model is trained, we score it against the test dataset and view metrics.<jupyter_code>from mmlspark.train import ComputeModelStatistics, TrainedClassifierModel
prediction = model.transform(test)
metrics = ComputeModelStatistics().transform(prediction)
metrics.limit(10).toPandas()<jupyter_output><empty_output><jupyter_text>Finally, we save the model so it can be used in a scoring program.<jupyter_code>model.write().overwrite().save("AdultCensus.mml")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12459016393442623, 'fraction_numerical': 0.008524590163934427, 'mean_word_length': 6.986910994764398, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '252649', 'license': 'Apache-2.0', 'n_tokens_mistral': 954, 'n_tokens_neox': 889, 'n_words': 233, 'path': 'notebooks/samples/Classification - Adult Census.ipynb', 'repo_name': 'jianzhnie/AutoML-Tools', 'size': '3050'}