<jupyter_start><jupyter_text>Patsy[Patsy](https://patsy.readthedocs.io/en/latest/index.html) is a neat API to transform your data into experimentation model form. For regression and classification problems, you often want your data in the `Xy` form where `X` is a matrix (independent variable) and `y` is a column vector (dependent variable). In regression, let's say you have $x_1, x_2$ as your independent variable and $y$ as your dependent variable. You might want to express the possible models as follows.- $y = b_0 + x_1 + x_2$- $y = b_0 + x_1 + x_2 + x_1 x_2$- $y = b_0 + x_1 + x_2 + x_1^2 + x_2^2$- $y = b_0 + x_1 + x_2 + x_1^2 + x_2^2 + x_1 x_2$If your data is stored in a dataframe, it's not terribly hard (but not so easy either) to transform your dataframe into one with the polynomial and interaction effects. Let's look at a made up dataframe below, and let's say that we want to model our regression problem with $y = b_0 + x_1 + x_2 + x_1^2 + x_2^2 + x_1 x_2$. How would we transform the dataframe?<jupyter_code>import pandas as pd

df = pd.DataFrame({
    'height': [10, 20, 30, 40, 50],
    'weight': [88, 99, 125, 155, 120],
    'y': [50, 60, 70, 80, 90]
})
df<jupyter_output><empty_output><jupyter_text>Here's one possible way to transform the dataframe into the model we desire. There are 6 lines of code!<jupyter_code>X = df[['height', 'weight']]
X['intercept'] = 1
X['height**2'] = df.height**2
X['weight**2'] = df.weight**2
X['height*weight'] = df.height * df.weight
X = X[['intercept', 'height', 'weight', 'height**2', 'weight**2', 'height*weight']]

X<jupyter_output><empty_output><jupyter_text>FormulasThere's an easier way using `patsy`. Patsy uses `R`-style [formulas](https://patsy.readthedocs.io/en/latest/formulas.html) to define the model and takes care of transforming the data for you. There are only 2 lines of code in using patsy! Really, just 1 line of code if the formula was not so long, then we would have passed the string literal to the `dmatrice()` function.<jupyter_code>from patsy import dmatrices

formula = 'y ~ height + weight + I(height**2) + I(weight**2) + height:weight'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>Notice how we had to use `I(height**2)` and `I(weight**2)`? The `**` operator already has a special meaning in patsy (which is to produce interaction effects), and not the Python meaning (which is raising a number to the power of another). The `I()` is called the [identity operator](https://patsy.readthedocs.io/en/latest/formulas.html?highlight=Identity%20functionfactors-and-terms) and preserves the expected Python expression. Also, we got the `Intercept` column by default. If we did not want the `Intercept` column, then we would subtract 1.<jupyter_code>formula = 'y ~ height + weight + I(height**2) + I(weight**2) + height:weight - 1'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>The `return_type` argument is set to `dataframe` so that a tuple of `pandas` dataframes are returned; otherwise, a tuple of `DesignMatrix` is returned instead. A `DesignMatrix` is just a wrapper around `numpy` arrays with metadata.<jupyter_code>formula = 'y ~ height + weight + I(height**2) + I(weight**2) + height:weight - 1'
y, X = dmatrices(formula, df)
X
y<jupyter_output><empty_output><jupyter_text>If we did not want `y`, we simply omit the `y ~` part and use the `dmatrix()` function instead.<jupyter_code>from patsy import dmatrix

formula = 'height + weight + I(height**2) + I(weight**2) + height:weight - 1'
X = dmatrix(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>We already saw the builtin `I()` function, but we can also use other functions to transform our data. Below, we do a log transform on height and weight.<jupyter_code>import numpy as np

formula = 'y ~ np.log(height) + np.log(weight) + I(height**2) + I(weight**2) + height:weight'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>We can also center or standardize values using using the builtin functions `center()` and `standardize()`.<jupyter_code>formula = 'y ~ center(height) + standardize(weight) + I(height**2) + I(weight**2) + center(height):standardize(weight)'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>Categorical dataHow does patsy handle categorical data? Below, we create a dummy data set with two categorical fields; `handedness` has the values left and right; school has the values elementary, middle, high (high for highschool).<jupyter_code>df = pd.DataFrame({
    'height': [10, 20, 30, 40, 50],
    'weight': [88, 99, 125, 155, 120],
    'handedness': ['left', 'right', 'left', 'right', 'left'],
    'school': ['elementary', 'middle', 'high', 'elementary', 'middle'],
    'y': [50, 60, 70, 80, 90]
})
df<jupyter_output><empty_output><jupyter_text>The formulas work the same way as before, and patsy magically creates one-hot encoded variables for the categorical variables. Notice how one of the categorical variables is automatically dropped? For example, we might expect to see `handedness[T.left]` and `handedness[T.right]` as bew columns, but we only see `handedness[T.right]`. Likewise, `school[T.elementary]` is dropped. The resulting column names of the pattern `name[T.value]` is to signal that these fields are the result of one-hot encoding against categorical variables.<jupyter_code>formula = 'y ~ height + weight + handedness + school'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>If we use the `C()` builtin function, we can specify the ordering or levels of the categorical variable. Notice how the first level is the one dropped.<jupyter_code>handedness_levels = ['right', 'left']
school_levels = ['elementary', 'middle', 'high']

formula = 'y ~ height + weight + C(handedness, levels=handedness_levels) + C(school, levels=school_levels)'
y, X = dmatrices(formula, df, return_type='dataframe')

X<jupyter_output><empty_output><jupyter_text>Interaction To specify interaction effects, we have a couple of options. We can use the following operators or symbols.- `:`- `*`- `**`Below, we specify interaction effect using `:`. You will notice that `handedness` is not part of the model.<jupyter_code>formula = 'y ~ handedness:school'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>Below, we use `*`, and you will notice all two-way and single effects are included. The result is typically what we want. Now, `handedness` is part of the model.<jupyter_code>formula = 'y ~ handedness*school'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>The `**` is probably the best way to specify 1 and 2-way interactions between two variables. Additionally, patsy just knows what to do with dropping columns to prevent multicollinearity.<jupyter_code>formula = 'y ~ (handedness + school)**2'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>Here is a model specified up to all 2-way interactions for all the variables.<jupyter_code>formula = 'y ~ (height + weight + handedness + school)**2'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>And if we wanted up to all 3-way interactions?<jupyter_code>formula = 'y ~ (height + weight + handedness + school)**3'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>And if we wanted up to all 4-way interactions?<jupyter_code>formula = 'y ~ (height + weight + handedness + school)**4'
y, X = dmatrices(formula, df, return_type='dataframe')
list(X.columns)<jupyter_output><empty_output><jupyter_text>Null valuesWhen creating your design matrices from formulas, patsy, for better or worse, will drop records that have null values. Below, we have 5 records and there is one record with a null value for `handedness`.<jupyter_code>df = pd.DataFrame({
    'height': [10, 20, 30, 40, 50],
    'weight': [88, 99, 125, 155, 120],
    'handedness': ['left', 'right', 'left', None, 'left'],
    'school': ['elementary', 'middle', 'high', 'elementary', 'middle'],
    'y': [50, 60, 70, 80, 90]
})
df<jupyter_output><empty_output><jupyter_text>If we defined a model, then observe how the record with the null value is removed!<jupyter_code>formula = 'y ~ handedness*school'
y, X = dmatrices(formula, df, return_type='dataframe')
X<jupyter_output><empty_output><jupyter_text>To prevent records with null values from being remove, we have to specify what to do with null values using the `NA_action` argument. By default, `NA_action` is set to `drop` to remove records with null values. The alternative is to set this argument to `raise`, which will raise an exception. Neither of these string values help to retain records with null values. We have to use the object `NAAction` and specify the null types to empty (e.g. `NA_types=[]`). The example is below. Observe how both levels of `handedness` is part of the data? You will have to drop one of these now. Yikes! Always make sure that you check your resulting design matrices!It's a bit awkward how records with null values are dropped by default, but, if you decide to keep them, then columns are not dropped as expected. What's being dropped and not being dropped? Again, check your resulting design matrices to make sure that your row and column counts/outputs are as expected when dealing with null values.<jupyter_code>from patsy import NAAction

formula = 'y ~ handedness*school'
y, X = dmatrices(formula, df, return_type='dataframe', NA_action=NAAction(NA_types=[]))
X<jupyter_output><empty_output><jupyter_text>If you have null values for continuous variables, that is fine. In the example below, one of the `weight` value is `None`.<jupyter_code>df = pd.DataFrame({
    'height': [10, 20, 30, 40, 50],
    'weight': [88, 99, 125, None, 120],
    'handedness': ['left', 'right', 'left', None, 'left'],
    'school': ['elementary', 'middle', 'high', 'elementary', 'middle'],
    'y': [50, 60, 70, 80, 90]
})

formula = 'y ~ height + weight + handedness + school'
y, X = dmatrices(formula, df, return_type='dataframe', NA_action=NAAction(NA_types=[]))
X<jupyter_output><empty_output><jupyter_text>And just for completeness, if we do not have null values for the categorical variables, then patsy drops one of them as expected.<jupyter_code>df = pd.DataFrame({
    'height': [10, 20, 30, 40, 50],
    'weight': [88, 99, 125, None, 120],
    'handedness': ['left', 'right', 'left', 'right', 'left'],
    'school': ['elementary', 'middle', 'high', 'elementary', 'middle'],
    'y': [50, 60, 70, 80, 90]
})

formula = 'y ~ height + weight + handedness + school'
y, X = dmatrices(formula, df, return_type='dataframe', NA_action=NAAction(NA_types=[]))
X<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12650330047924768, 'fraction_numerical': 0.021159236820689032, 'mean_word_length': 5.595110316040548, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '59428', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3777, 'n_tokens_neox': 3514, 'n_words': 1297, 'path': 'sphinx/scikit-intro/source/patsy.ipynb', 'repo_name': 'oneoffcoder/books', 'size': '11059'}