<jupyter_start><jupyter_text>Introducing CivisML 2.0Note: We are continually releasing changes to CivisML, and this notebook is useful for any versions 2.0.0 and above.Data scientists are on the front lines of their organization’s most important customer growth and engagement questions, and they need to guide action as quickly as possible by getting models into production.  CivisML is a machine learning service that makes it possible for data scientists to massively increase the speed with which they can get great models into production.  And because it’s built on open-source packages, CivisML remains transparent and data scientists remain in control.In this notebook, we’ll go over the new features introduced in CivisML 2.0.  For a walkthrough of CivisML’s fundamentals, check out this introduction to the mechanics of CivisML: https://github.com/civisanalytics/civis-python/blob/master/examples/CivisML_parallel_training.ipynbCivisML 2.0 is full of new features to make modeling faster, more accurate, and more portable. This notebook will cover the following topics:- CivisML overview- Parallel training and validation- Use of the new ETL transformer, `DataFrameETL`, for easy, customizable ETL- Stacked models: combine models to get one bigger, better model- Model portability: get trained models out of CivisML- Multilayer perceptron models: neural networks built in to CivisML- Hyperband: a smarter alternative to grid searchCivisML can be used to build models that answer all kinds of business questions, such as what movie to recommend to a customer, or which customers are most likely to upgrade their accounts. For the sake of example, this notebook uses a publicly available dataset on US colleges, and focuses on predicting the type of college (public non-profit, private non-profit, or private for-profit).<jupyter_code># first, let's import the packages we need
import requests
from io import StringIO
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import model_selection

# import the Civis Python API client
import civis
# ModelPipeline is the class used to build CivisML models
from civis.ml import ModelPipeline
# Suppress warnings for demo purposes. This is not recommended as a general practice.
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Downloading dataBefore we build any models, we need a dataset to play with. We're going to use the most recent College Scorecard data from the Department of Education.This dataset is collected to study the performance of US higher education institutions. You can learn more about it in [this technical paper](https://collegescorecard.ed.gov/assets/UsingFederalDataToMeasureAndImprovePerformance.pdf), and you can find details on the dataset features in [this data dictionary](https://collegescorecard.ed.gov/data/).<jupyter_code># Downloading data; this may take a minute
# Two kind of nulls
df = pd.read_csv("https://ed-public-download.app.cloud.gov/downloads/Most-Recent-Cohorts-All-Data-Elements.csv", sep=",", na_values=['NULL', 'PrivacySuppressed'], low_memory=False)
# How many rows and columns?
df.shape
# What are some of the column names?
df.columns<jupyter_output><empty_output><jupyter_text>Data MungingBefore running CivisML, we need to do some basic data munging, such as removing missing data from the dependent variable, and splitting the data into training and test sets.Throughout this notebook, we'll be trying to predict whether a college is public (labelled as 1), private non-profit (2), or private for-profit (3). The column name for this dependent variable is "CONTROL".<jupyter_code># Make sure to remove any rows with nulls in the dependent variable
df = df[np.isfinite(df['CONTROL'])]
# split into training and test sets
train_data, test_data = model_selection.train_test_split(df, test_size=0.2)
# print a few sample columns
train_data.head()<jupyter_output><empty_output><jupyter_text>Some of these columns are duplicates, or contain information we don't want to use in our model (like college names and URLs). CivisML can take a list of columns to exclude and do this part of the data munging for us, so let's make that list here.<jupyter_code>to_exclude = ['ADM_RATE_ALL', 'OPEID', 'OPEID6', 'ZIP', 'INSTNM', 
              'INSTURL', 'NPCURL', 'ACCREDAGENCY', 'T4APPROVALDATE', 
              'STABBR', 'ALIAS', 'REPAY_DT_MDN', 'SEPAR_DT_MDN']<jupyter_output><empty_output><jupyter_text>Basic CivisML UsageWhen building a supervised model, there are a few basic things you'll probably want to do:1. Transform the data into a modelling-friendly format2. Train the model on some labelled data3. Validate the model4. Use the model to make predictions about unlabelled dataCivisML does all of this in three lines of code. Let's fit a basic sparse logistic model to see how. The first thing we need to do is build a `ModelPipeline` object. This stores all of the basic configuration options for the model. We'll tell it things like the type of model, dependent variable, and columns we want to exclude. CivisML handles basic ETL for you, including categorical expansion of any string-type columns.<jupyter_code># Use a push-button workflow to fit a model with reasonable default parameters
sl_model = ModelPipeline(model='sparse_logistic',
                         model_name='Example sparse logistic',
                         primary_key='UNITID',
                         dependent_variable=['CONTROL'],
                         excluded_columns=to_exclude)<jupyter_output><empty_output><jupyter_text>Next, we want to train and validate the model by calling `.train` on the `ModelPipeline` object. CivisML uses 4-fold cross-validation on the training set. You can train on local data or query data from Redshift. In this case, we have our data locally, so we just pass the data frame.<jupyter_code>sl_train = sl_model.train(train_data)<jupyter_output><empty_output><jupyter_text>This returns a `ModelFuture` object, which is *non-blocking*-- this means that you can keep doing things in your notebook while the model runs on Civis Platform in the background. If you want to make a blocking call (one that doesn't complete until your model is finished), you can use `.result()`.<jupyter_code># non-blocking
sl_train
# blocking
sl_train.result()<jupyter_output><empty_output><jupyter_text>Parallel Model Tuning and ValidationWe didn't actually specify the number of jobs in the `.train()` call above, but behind the scenes, the model was actually training in parallel! In CivisML 2.0, model tuning and validation will automatically be distributed across your computing cluster, without ever using more than 90% of the cluster resources. This means that you can build models faster and try more model configurations, leaving you more time to think critically about your data. If you decide you want more control over the resources you're using, you can set the `n_jobs` parameter to a specific number of jobs, and CivisML won't run more than that at once. We can see how well the model did by looking at the validation metrics.<jupyter_code># loop through the metric names and print to screen
metrics = [print(key) for key in sl_train.metrics.keys()]
# ROC AUC for each of the three categories in our dependent variable
sl_train.metrics['roc_auc']<jupyter_output><empty_output><jupyter_text>Impressive!This is the basic CivisML workflow: create the model, train, and make predictions. There are other configuration options for more complex use cases; for example, you can create a custom estimator, pass custom dependencies, manage the computing resources for larger models, and more. For more information, see the Machine Learning section of the [Python API client docs](https://civis-python.readthedocs.io).Now that we can build a simple model, let's see what's new to CivisML 2.0!  Custom ETLCivisML can do several data transformations to prepare your data for modeling. This makes data preprocessing easier, and makes it part of your model pipeline rather than an additional script you have to run. CivisML's built-in ETL includes:- Categorical expansion: expand a single column of strings or categories into separate binary variables.- Dropping columns: remove columns not needed in a model, such as an ID number.- Removing null columns: remove columns that contain no data.With CivisML 2.0, you can now recreate and customize this ETL using `DataFrameETL`, our open source ETL transformer, [available on GitHub](https://github.com/civisanalytics/civisml-extensions).By default, CivisML will use DataFrameETL to automatically detect non-numeric columns for categorical expansion. Our example college dataset has a lot of integer columns which are actually categorical, but we can make sure they're handled correctly by passing CivisML a custom ETL transformer.<jupyter_code># The ETL transformer used in CivisML can be found in the civismlext module
from civismlext.preprocessing import DataFrameETL<jupyter_output><empty_output><jupyter_text>This creates a list of columns to categorically expand, identified using the data dictionary available [here](https://collegescorecard.ed.gov/data/).<jupyter_code># column indices for columns to expand
to_expand = list(df.columns[:21]) + list(df.columns[23:36]) + list(df.columns[99:290]) + \
    list(df.columns[[1738, 1773, 1776]])
# create ETL estimator to pass to CivisML
etl = DataFrameETL(cols_to_drop=to_exclude, 
                   cols_to_expand=to_expand, # we made this column list during data munging
                   check_null_cols='warn')<jupyter_output><empty_output><jupyter_text>Model StackingNow it's time to fit a model. Let's take a look at model stacking, which is new to CivisML 2.0.Stacking lets you combine several algorithms into a single model which performs as well or better than the component algorithms. We use stacking at Civis to build more accurate models, which saves our data scientists time comparing algorithm performance. In CivisML, we have two stacking workflows: `stacking_classifier` (sparse logistic, GBT, and random forest, with a logistic regression model as a "meta-estimator" to combine predictions from the other models); and `stacking_regressor` (sparse linear, GBT, and random forest, with a non-negative linear regression as the meta-estimator). Use them the same way you use `sparse_logistic` or other pre-defined models. If you want to learn more about how stacking works under the hood, take a look at [this talk](https://www.youtube.com/watch?v=3gpf1lGwecA&t=1058s) by the person at Civis who wrote it!Let's fit both a stacking classifier and some un-stacked models, so we can compare the performance.<jupyter_code>workflows = ['stacking_classifier',
            'sparse_logistic',
            'random_forest_classifier',
            'gradient_boosting_classifier']
models = []
# create a model object for each of the four model types
for wf in workflows:
    model = ModelPipeline(model=wf,
                          model_name=wf + ' v2 example',
                          primary_key='UNITID',
                          dependent_variable=['CONTROL'],
                          etl=etl  # use the custom ETL we created
                          )
    models.append(model)
# iterate over the model objects and run a CivisML training job for each
trains = []
for model in models:
    train = model.train(train_data)
    trains.append(train)<jupyter_output><empty_output><jupyter_text>Let's plot diagnostics for each of the models. In the Civis Platform, these plots will automatically be built and displayed in the "Models" tab. But for the sake of example, let's also explicitly plot ROC curves and AUCs in the notebook.There are three classes (public, non-profit private, and for-profit private), so we'll have three curves per model. It looks like all of the models are doing well, with sparse logistic performing slightly worse than the other three.<jupyter_code>%matplotlib inline
# Let's look at how the model performed during validation
def extract_roc(fut_job, model_name):
    '''Build a data frame of ROC curve data from the completed training job `fut_job`
    with model name `model_name`. Note that this function will only work for a classification
    model where the dependent variable has more than two classes.'''
    aucs = fut_job.metrics['roc_auc']
    roc_curve = fut_job.metrics['roc_curve_by_class']
    n_classes = len(roc_curve)
    fpr = []
    tpr = []
    class_num = []
    auc = []
    for i, curve in enumerate(roc_curve):
        fpr.extend(curve['fpr'])
        tpr.extend(curve['tpr'])
        class_num.extend([i] * len(curve['fpr']))
        auc.extend([aucs[i]] * len(curve['fpr']))
    model_vec = [model_name] * len(fpr)
    df = pd.DataFrame({
        'model': model_vec,
        'class': class_num,
        'fpr': fpr,
        'tpr': tpr,
        'auc': auc
    })
    return df

# extract ROC curve information for all of the trained models
workflows_abbrev = ['stacking', 'logistic', 'RF', 'GBT']
roc_dfs = [extract_roc(train, w) for train, w in zip(trains, workflows_abbrev)]
roc_df = pd.concat(roc_dfs)

# create faceted ROC curve plots. Each row of plots is a different model type, and each
# column of plots is a different class of the dependent variable.
g = sns.FacetGrid(roc_df, col="class",  row="model")
g = g.map(plt.plot, "fpr", "tpr", color='blue')<jupyter_output><empty_output><jupyter_text>All of the models perform quite well, so it's difficult to compare based on the ROC curves. Let's plot the AUCs themselves.<jupyter_code># Plot AUCs for each model
%matplotlib inline
auc_df = roc_df[['model', 'class', 'auc']]
auc_df.drop_duplicates(inplace=True)
plt.show(sns.swarmplot(x=auc_df['model'], y=auc_df['auc']))<jupyter_output><empty_output><jupyter_text>Here we can see that all models but sparse logistic perform quite well, but stacking appears to perform marginally better than the others. For more challenging modeling tasks, the difference between stacking and other models will often be more pronounced. Now our models are trained, and we know that they all perform very well. Because the AUCs are all so high, we would expect the models to make similar predictions. Let's see if that's true.<jupyter_code># kick off a prediction job for each of the four models
preds = [model.predict(test_data) for model in models]
# This will run on Civis Platform cloud resources
[pred.result() for pred in preds]
# print the top few rows for each of the models
pred_df = [pred.table.head() for pred in preds]
import pprint
pprint.pprint(pred_df)<jupyter_output>[          control_1  control_2  control_3
UNITID                                   
217882     0.993129   0.006856   0.000015
195234     0.001592   0.990423   0.007985
446385     0.002784   0.245300   0.751916
13508115   0.003109   0.906107   0.090785
459499     0.005351   0.039922   0.954726,
              control_1  control_2  control_3
UNITID                                      
217882    9.954234e-01   0.000200   0.004377
195234    6.766601e-08   0.999615   0.000385
446385    4.571749e-03   0.056303   0.939125
13508115  1.768058e-02   0.699806   0.282514
459499    1.319468e-02   0.285295   0.701510,
           control_1  control_2  control_3
UNITID                                   
217882        0.960      0.034      0.006
195234        0.012      0.974      0.014
446385        0.020      0.508      0.472
13508115      0.006      0.914      0.080
459499        0.032      0.060      0.908,
           control_1  control_2  control_3
UNITID                                   
217882[...]<jupyter_text>Looks like the probabilities here aren't exactly the same, but are directionally identical-- so, if you chose the class that had the highest probability for each row, you'd end up with the same predictions for all models. This makes sense, because all of the models performed well.  Model PortabilityWhat if you want to score a model outside of Civis Platform? Maybe you want to deploy this model in an app for education policy makers. In CivisML 2.0, you can easily get the trained model pipeline out of the `ModelFuture` object.<jupyter_code>train_stack = trains[0] # Get the ModelFuture for the stacking model
trained_model = train_stack.estimator<jupyter_output><empty_output><jupyter_text>This `Pipeline` contains all of the steps CivisML used to train the model, from ETL to the model itself. We can print each step individually to get a better sense of what is going on.<jupyter_code># print each of the estimators in the pipeline, separated by newlines for readability
for step in train_stack.estimator.steps:
    print(step[1])
    print('\n')<jupyter_output>DataFrameETL(check_null_cols='warn',
       cols_to_drop=['ADM_RATE_ALL', 'OPEID', 'OPEID6', 'ZIP', 'INSTNM', 'INSTURL', 'NPCURL', 'ACCREDAGENCY', 'T4APPROVALDATE', 'STABBR', 'ALIAS', 'REPAY_DT_MDN', 'SEPAR_DT_MDN'],
       cols_to_expand=['UNITID', 'OPEID', 'OPEID6', 'INSTNM', 'CITY', 'STABBR', 'ZIP', 'ACCREDAGENCY', 'INSTURL', 'NPCURL', 'SCH_DEG', 'HCM2', 'MAIN', 'NUMBRANCH', 'PREDDEG', 'HIGHDEG', 'CONTROL', 'ST_FIPS', 'REGION', 'LOCALE', 'LOCALE2', 'CCBASIC', 'CCUGPROF', 'CCSIZSET', 'HBCU', 'PBI', 'ANNHI', 'TRIBAL',...RT2', 'CIP54ASSOC', 'CIP54CERT4', 'CIP54BACHL', 'DISTANCEONLY', 'ICLEVEL', 'OPENADMP', 'ACCREDCODE'],
       dataframe_output=False, dummy_na=True, fill_value=0.0)


Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)


StackedClassifier(cv=StratifiedKFold(n_splits=4, random_state=42420, shuffle=True),
         estimator_list=[('sparse_logistic', Pipeline(memory=None,
     steps=[('selectfrommodel', SelectFromModel(estimator=LogitNet(alpha=1, c[...]<jupyter_text>Now we can see that there are three steps: the `DataFrameETL` object we passed in, a null imputation step, and the stacking estimator itself.We can use this outside of CivisML simply by calling `.predict` on the estimator. This will make predictions using the model in the notebook without using CivisML.<jupyter_code># drop the dependent variable so we don't use it to predict itself!
predictions = trained_model.predict(test_data.drop(labels=['CONTROL'], axis=1))
# print out the class predictions. These will be integers representing the predicted
# class rather than probabilities.
predictions<jupyter_output><empty_output><jupyter_text>Hyperparameter optimization with Hyperband and Neural NetworksMultilayer Perceptrons (MLPs) are simple neural networks, which are now built in to CivisML. The MLP estimators in CivisML come from [muffnn](https://github.com/civisanalytics/muffnn), another open source package written and maintained by Civis Analytics using [tensorflow](https://www.tensorflow.org/). Let's fit one using hyperband.Tuning hyperparameters is a critical chore for getting an algorithm to perform at its best, but it can take a long time to run. Using CivisML 2.0, we can use hyperband as an alternative to conventional grid search for hyperparameter optimization-- it runs about twice as fast. While grid search runs every parameter combination for the full time, hyperband runs many combinations for a short time, then filters out the best, runs them for longer, filters again, and so on. This means that you can try more combinations in less time, so we recommend using it whenever possible. The hyperband estimator is open source and [available on GitHub](https://github.com/civisanalytics/civisml-extensions). You can learn about the details in [the original paper, Li et al. (2016)](https://arxiv.org/abs/1603.06560).Right now, hyperband is implemented in CivisML named preset models for the following algorithms: - Multilayer Perceptrons (MLPs)- Stacking- Random forests- GBTs- ExtraTreesUnlike grid search, you don't need to specify values to search over. If you pass `cross_validation_parameters='hyperband'` to `ModelPipeline`, hyperparameter combinations will be randomly drawn from preset distributions.<jupyter_code># build a model specifying the MLP model with hyperband
model_mlp = ModelPipeline(model='multilayer_perceptron_classifier',
                          model_name='MLP example',
                          primary_key='UNITID',
                          dependent_variable=['CONTROL'],
                          cross_validation_parameters='hyperband',
                          etl=etl
                          )
train_mlp = model_mlp.train(train_data, 
                            n_jobs=10) # parallel hyperparameter optimization and validation!
# block until the job finishes
train_mlp.result()<jupyter_output><empty_output><jupyter_text>Let's dig into the hyperband model a little bit. Like the stacking model, the model below starts with ETL and null imputation, but contains some additional steps: a step to scale the predictor variables (which improves neural network performance), and a hyperband searcher containing the MLP.<jupyter_code>for step in train_mlp.estimator.steps:
    print(step[1])
    print('\n')<jupyter_output>INFO:tensorflow:Restoring parameters from /tmp/tmpe49np0dv/saved_model
DataFrameETL(check_null_cols='warn',
       cols_to_drop=['ADM_RATE_ALL', 'OPEID', 'OPEID6', 'ZIP', 'INSTNM', 'INSTURL', 'NPCURL', 'ACCREDAGENCY', 'T4APPROVALDATE', 'STABBR', 'ALIAS', 'REPAY_DT_MDN', 'SEPAR_DT_MDN'],
       cols_to_expand=['UNITID', 'OPEID', 'OPEID6', 'INSTNM', 'CITY', 'STABBR', 'ZIP', 'ACCREDAGENCY', 'INSTURL', 'NPCURL', 'SCH_DEG', 'HCM2', 'MAIN', 'NUMBRANCH', 'PREDDEG', 'HIGHDEG', 'CONTROL', 'ST_FIPS', 'REGION', 'LOCALE', 'LOCALE2', 'CCBASIC', 'CCUGPROF', 'CCSIZSET', 'HBCU', 'PBI', 'ANNHI', 'TRIBAL',...RT2', 'CIP54ASSOC', 'CIP54CERT4', 'CIP54BACHL', 'DISTANCEONLY', 'ICLEVEL', 'OPENADMP', 'ACCREDCODE'],
       dataframe_output=False, dummy_na=True, fill_value=0.0)


Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)


MinMaxScaler(copy=False, feature_range=(0, 1))


HyperbandSearchCV(cost_parameter_max={'n_epochs': 50},
         cost_parameter_min={'n_epochs': 5}, cv=None,[...]<jupyter_text>`HyperbandSearchCV` essentially works like `GridSearchCV`. If you want to get the best estimator without all of the extra CV information, you can access it using the `best_estimator_` attribute.<jupyter_code>train_mlp.estimator.steps[3][1].best_estimator_<jupyter_output><empty_output><jupyter_text>To see how well the best model performed, you can look at the `best_score_`.<jupyter_code>train_mlp.estimator.steps[3][1].best_score_<jupyter_output><empty_output><jupyter_text>And to look at information about the different hyperparameter configurations that were tried, you can look at the `cv_results_`.<jupyter_code>train_mlp.estimator.steps[3][1].cv_results_<jupyter_output><empty_output><jupyter_text>Just like any other model in CivisML, we can use hyperband-tuned models to make predictions using `.predict()` on the `ModelPipeline`.<jupyter_code>predict_mlp = model_mlp.predict(test_data)
predict_mlp.table.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08071885470606154, 'fraction_numerical': 0.02319307253818372, 'mean_word_length': 4.433096926713948, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 90, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '40134', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 7051, 'n_tokens_neox': 6526, 'n_words': 2642, 'path': 'examples/Introducing_CivisML_v2.ipynb', 'repo_name': 'mcoirad-gmmb/civis-python', 'size': '22981'}