<jupyter_start><jupyter_text>Selecting an alternative type for a column Sometimes ptype infers a type for a column which we know to be incorrect. In this situation, ptype allows us to select a _different_ column type and still take advantage of its per-row type inference (conditional on the new type of the column) to identify anomalous and missing values. To illustrate this, we present the following two scenarios:- We employ ptype on a toy example constructed using mostly 4-digit years (e.g. 1991) and an amalous 2-digit year (99). We assume that the user runs ptype on the data frame and inspects the inferred schema. The schema classifies the column as `integer` rather than `date-iso-8601`. The user then indicates to ptype that the column should be treated as `date-iso-8601` and notices that this lets ptype detect `99` as an anomalous entry.- For a more realistic example, we consider the task of measuring the association between two non-numerical data columns of the Grub Damage dataset, which is a collection of information about grass grub numbers, irrigation and damage. The user loads the dataset into a dataframe using Pandas `read_csv` and transforms the data frame using the schema inferred by ptype to calculate the Cramérs V statistic between the two columns. However, ptype misclassifies one of the columns, which causes the measure to be undefined. The user is able to correct ptype's prediction and then correctly calculate the Cramérs V on the newly transformed dataframe.<jupyter_code># Preamble to run notebook in context of source package.
import sys
sys.path.insert(0, '../')<jupyter_output><empty_output><jupyter_text>Toy ExampleHere, we construct a Pandas dataframe containing 9 data entries valued between 1991 and 1999, where in one row 1999 is encoded by 99. Note that we set `dtype` to `str` so that ptype can infer a schema based on this untyped (string) representation.<jupyter_code>import pandas as pd

x = ['1991', '1992', '1993', '1994', '1995', '1996', '1997', '1998', '99']
column = 'year'

df = pd.DataFrame(x, dtype='str', columns=[column])
df<jupyter_output><empty_output><jupyter_text>First, we show that the default behaviour of ptype is to infer that the column contains integers rather than dates.<jupyter_code>from ptype.Ptype import Ptype

ptype = Ptype()
schema = ptype.schema_fit(df)
schema.show()<jupyter_output><empty_output><jupyter_text>Notice that the column's type is reported as integer and no missing or anomalous entries are detected. We can now change how the column is interpreted by interacting with ptype:<jupyter_code>schema.cols[column].reclassify('date-iso-8601')<jupyter_output><empty_output><jupyter_text>As a result of this interaction, the inferred schema is modified as follows:<jupyter_code>schema.show()<jupyter_output><empty_output><jupyter_text>Now that the user has specified that the column is of type `date-iso-8601`, ptype can update its beliefs regarding anomalous values. As we can see, 99 is now classified as an anomalous entry since it does not fit the corresponding Probabilistic Finite-State Machine, i.e., the machine expects 4-digit formatted values between 1900 and 2099.<jupyter_code># repair 99 as 1999?<jupyter_output><empty_output><jupyter_text>Real-world ExampleIn this example, we use the [Grub Damage](https://www.openml.org/d/338) dataset to analyze the relationship between grass grub numbers, irrigation and damage. We will focus on the task of finding the association between the `zone` and `GG_new` columns.Note that the `GG_new` column is an ordinal variable.<jupyter_code>df = pd.read_csv('../data/grub-damage.csv', encoding="ISO-8859-1", dtype='str')
df.head()<jupyter_output><empty_output><jupyter_text>First, we use ptype to inspect the properties of this dataset.<jupyter_code>ptype = Ptype()

schema = ptype.schema_fit(df)
schema.show()<jupyter_output><empty_output><jupyter_text>As you can see, ptype predicts the type of the `zone` column as `boolean` and labels the values `C` and `M` as anomalies. Note that we can confirm that these values should be consider “normal“ values by consulting the metadata, which states "8. zone - position of paddock (F: foothills, M: midplain, C: coastal) - enumerated".If we accept ptype‘s analysis as it stands, we can obtain the following data frame:<jupyter_code>df2 = schema.transform(df)
df2<jupyter_output><empty_output><jupyter_text>However, since the anomalous values have now been replaced by ``, the Cramers V measure between `zone` and `GG_new columns` is undefined:<jupyter_code># NBVAL_IGNORE_OUTPUT
import numpy as np
import scipy.stats as ss

# see https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V
def cramers_corrected_stat(x, y):
    """ calculate Cramers V statistic for categorial-categorial association.
        uses correction from Bergsma and Wicher, 
        Journal of the Korean Statistical Society 42 (2013): 323-328
    """
    confusion_matrix = pd.crosstab(x, y)
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum().sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2 - ((k-1)*(r-1))/(n-1))    
    rcorr = r - ((r-1)**2)/(n-1)
    kcorr = k - ((k-1)**2)/(n-1)
    return np.sqrt(phi2corr / min( (kcorr-1), (rcorr-1)))


cramers_corrected_stat(df2['zone'], df2['GG_new'])<jupyter_output>/Users/tceritli/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:19: RuntimeWarning: invalid value encountered in double_scalars<jupyter_text>As before, we can interact further with ptype to fix its predictions for the `zone` column.<jupyter_code>schema.cols['zone'].reclassify('string')
schema.show()<jupyter_output><empty_output><jupyter_text>As we can see, the row type predictions have been updated according to the new column type, and so `C` and `M` are now classified as normal values. Now we can tranform the dataframe according to the revised schema.<jupyter_code>df2 = schema.transform(df)
df2<jupyter_output><empty_output><jupyter_text>And finally we can calculate the Cramers V statistic as desired:<jupyter_code>cramers_corrected_stat(df2['zone'], df2['GG_new'])<jupyter_output><empty_output><jupyter_text>Housing Price Dataset<jupyter_code># data source: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data
df = pd.read_csv('../data/housing_price.csv', dtype='str', keep_default_na=False)
df.head()
features = ['1stFlrSF', '2ndFlrSF', 'GrLivArea', 'HalfBath']
target = ['SalePrice']

X = df[features]
y = df[target]

df = df[features+target]
df.head()
schema = ptype.schema_fit(df)
schema.show()
df2 = schema.transform(df)
df2.head()
from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt
%matplotlib inline
plt.rcdefaults()

lm = LinearRegression()

X = df2[features].values
y = df2[target].values

# lm.fit(X, y)
# y_hat = lm.predict(X)

# def scatter_plot(y, y_hat):
#     plt.figure(figsize=(12, 4))

#     sorted_indices = sorted(range(len(y)),key=y.__getitem__)
#     plt.scatter(range(len(y)), y[sorted_indices], label='Actual Price', s=10)
#     plt.scatter(range(len(y)), y_hat[sorted_indices], label='Predicted Price', s=10)

#     plt.title('Actual vs Fitted Values for SalePrice')
#     plt.xlabel('Item (sorted wrt actual price)')
#     plt.ylabel('Price (in dollars)')
#     plt.legend()
#     plt.show()

# scatter_plot(y, y_hat)
schema.cols['HalfBath'].reclassify('integer')
schema.show()
df2 = schema.transform(df)
df2.head()
X = df2[features].values
y = df2[target].values

lm.fit(X, y)
y_hat = lm.predict(X)

def scatter_plot(y, y_hat):
    plt.figure(figsize=(9, 3))

    sorted_indices = sorted(range(len(y)),key=y.__getitem__)
    plt.scatter(range(len(y)), y[sorted_indices], label='Actual Price', s=10)
    plt.scatter(range(len(y)), y_hat[sorted_indices], label='Predicted Price', s=10)

    plt.title('Actual vs Fitted Values for SalePrice')
    plt.xlabel('Item (sorted wrt actual price)')
    plt.ylabel('Price (in dollars)')
    plt.legend()
    plt.show()

scatter_plot(y, y_hat)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10758706467661691, 'fraction_numerical': 0.02164179104477612, 'mean_word_length': 5.351500789889416, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '135916', 'license': 'MIT', 'n_tokens_mistral': 2715, 'n_tokens_neox': 2539, 'n_words': 863, 'path': 'notebooks/reclassify-column.ipynb', 'repo_name': 'Andy-Practice/ptype', 'size': '8040'}