<jupyter_start><jupyter_text>[`vtreat`](https://github.com/WinVector/pyvtreat) Nested Model Bias WarningFor quite a while we have been teaching estimating variable re-encodings on the exact same data theyare later *naively* using to train a model on leads to an undesirable nested model bias.  The `vtreat`package (both the [`R` version](https://github.com/WinVector/vtreat) and [`Python` version](https://github.com/WinVector/pyvtreat)) both incorporate a cross-frame methodthat allows one to use all the training data both to build learn variable re-encodings and to correctly train a subsequent model (for an example please see our recent [PyData LA talk](http://www.win-vector.com/blog/2019/12/pydata-los-angeles-2019-talk-preparing-messy-real-world-data-for-supervised-machine-learning/)).The next version of `vtreat` will warn the user if they have improperly used the same data for both `vtreat` impact code inference and downstream modeling.  So in addition to us warning you not to do this, the package now also checks and warns against this situation.  Set up the Example This example is copied from [some of our classification documentation](https://github.com/WinVector/pyvtreat/blob/master/Examples/Classification/Classification.md).Load modules/packages.<jupyter_code>import pkg_resources
import pandas
import numpy
import numpy.random
import vtreat
import vtreat.util

numpy.random.seed(2019)<jupyter_output><empty_output><jupyter_text>Generate example data. * `y` is a noisy sinusoidal function of the variable `x`* `yc` is the output to be predicted: : whether `y` is > 0.5. * Input `xc` is a categorical variable that represents a discretization of `y`, along some `NaN`s* Input `x2` is a pure noise variable with no relationship to the output<jupyter_code>def make_data(nrows):
    d = pandas.DataFrame({'x': 5*numpy.random.normal(size=nrows)})
    d['y'] = numpy.sin(d['x']) + 0.1*numpy.random.normal(size=nrows)
    d.loc[numpy.arange(3, 10), 'x'] = numpy.nan                           # introduce a nan level
    d['xc'] = ['level_' + str(5*numpy.round(yi/5, 1)) for yi in d['y']]
    d['x2'] = numpy.random.normal(size=nrows)
    d.loc[d['xc']=='level_-1.0', 'xc'] = numpy.nan  # introduce a nan level
    d['yc'] = d['y']>0.5
    return d

training_data = make_data(500)

training_data.head()
outcome_name = 'yc'    # outcome variable / column
outcome_target = True  # value we consider positive<jupyter_output><empty_output><jupyter_text>Demonstrate the Warning Now that we have the data, we want to treat it prior to modeling: we want training data where all the input variables are numeric and have no missing values or `NA`s.First create the data treatment transform design object, in this case a treatment for a binomial classification problem.We use the training data `training_data` to fit the transform and the return a treated training set: completely numeric, with no missing values.<jupyter_code>treatment = vtreat.BinomialOutcomeTreatment(
    outcome_name=outcome_name,      # outcome variable
    outcome_target=outcome_target,  # outcome of interest
    cols_to_copy=['y'],  # columns to "carry along" but not treat as input variables
)  
train_prepared = treatment.fit_transform(training_data, training_data['yc'])<jupyter_output><empty_output><jupyter_text>`train_prepared` is prepared in the correct way to use the same training data for inferring the impact-coded variables, using `.fit_transform()` instead of `.fit().transform()`.We prepare new test or application data as follows.<jupyter_code>test_data = make_data(100)

test_prepared = treatment.transform(test_data)<jupyter_output><empty_output><jupyter_text>The issue is: for training data we should not call `transform()`, but instead use the value returned by `.fit_transform()`.The point is we should not do the following:<jupyter_code>train_prepared_wrong = treatment.transform(training_data)<jupyter_output>/Users/johnmount/opt/anaconda3/envs/ai_academy_3_7/lib/python3.7/site-packages/vtreat/vtreat_api.py:370: UserWarning: possibly called transform on same data used to fit (this causes over-fit, please use fit_transform() instead)
  "possibly called transform on same data used to fit (this causes over-fit, please use fit_transform() instead)")
{'fraction_non_alphanumeric': 0.09802538787023977, 'fraction_numerical': 0.010578279266572637, 'mean_word_length': 5.220760233918129, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '580989', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1305, 'n_tokens_neox': 1231, 'n_words': 443, 'path': 'Examples/Classification/ClassificationWarningExample.ipynb', 'repo_name': 'sdrees/pyvtreat', 'size': '4254'}