<jupyter_start><jupyter_text>Example1This tutorial describes preprocess using "prepnet" to predict IRIS specify by SVC.<jupyter_code>import prepnet

# Dependency libraries
import numpy as np
import pandas as pd
from sklearn.svm import SVC
from sklearn import datasets
# Load dataset and display distribution
iris = datasets.load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['target'] = iris.target_names[iris.target]
display(df.head())
df.hist(figsize=(8, 8))<jupyter_output><empty_output><jupyter_text>Preprocess using prepnetThe Series wise converters are numerical normalization or NaN imputation or others.There are always keep original columns and indices.The Frame wise converters change columns and indices.Almost conversion can directly invoke from FunctionalContext.But, converters of changing indices cannot directly invoke from FunctionalContext.In this case conversions need to invoke FunctionalContext.post accessor.This accessor always run after other converters.<jupyter_code>df.dtypes
context = prepnet.FunctionalContext()
with context.enter('normalize'):
    context[
        'sepal length (cm)',
        'sepal width (cm)',
        'petal length (cm)',
        'petal width (cm)',
    ].standardize()
with context.enter('outlier-process'):
    context[
        'sepal length (cm)',
        'sepal width (cm)',
        'petal length (cm)',
        'petal width (cm)',
    ].quantile_round(0.99)
with context.enter('post'):
    context.split(5)

preprocessed_df_list = list(context.encode(df))
train_df = pd.concat(preprocessed_df_list[:4], axis=0)
test_df = preprocessed_df_list[-1]
display(train_df.head(10))
train_df.hist(figsize=(8, 8))
test_df.hist(figsize=(8, 8))<jupyter_output><empty_output><jupyter_text>Training and display predictionsThis<jupyter_code># Training SVM multi-classification using one vs rest
from sklearn.metrics import accuracy_score
target_columns = 'target'

classifier = SVC(degree=3)
classifier.fit(train_df.drop(columns=target_columns), train_df[target_columns])
train_pred_labels = classifier.predict(train_df.drop(columns=target_columns))

print(f'Accuracy: {accuracy_score(train_df[target_columns], train_pred_labels):.3f}')
display(pd.crosstab(
    train_df[target_columns].reset_index(drop=True), 
    pd.Series(train_pred_labels, name='pred')
))
test_pred_labels = classifier.predict(test_df.drop(columns=target_columns))

print(f'Accuracy: {accuracy_score(test_df[target_columns], test_pred_labels):.3f}')
display(pd.crosstab(
    test_df[target_columns].reset_index(drop=True), 
    pd.Series(test_pred_labels, name='pred')
))<jupyter_output>Accuracy: 0.900<jupyter_text>Disable normalize and compare accuracy<jupyter_code>context2 = context.disable('normalize')
preprocessed_df_list = list(context2.encode(df))
train_df = pd.concat(preprocessed_df_list[:4], axis=0)
test_df = preprocessed_df_list[-1]
display(train_df.head(10))
train_df.hist(figsize=(8, 8))
test_df.hist(figsize=(8, 8))
# Training SVM multi-classification using one vs rest
from sklearn.metrics import accuracy_score
target_columns = 'target'

classifier = SVC(degree=3)
classifier.fit(train_df.drop(columns=target_columns), train_df[target_columns])
train_pred_labels = classifier.predict(train_df.drop(columns=target_columns))

print(f'Train Accuracy: {accuracy_score(train_df[target_columns], train_pred_labels):.3f}')
display(pd.crosstab(
    train_df[target_columns].reset_index(drop=True), 
    pd.Series(train_pred_labels, name='pred')
))

test_pred_labels = classifier.predict(test_df.drop(columns=target_columns))

print(f'Test Accuracy: {accuracy_score(test_df[target_columns], test_pred_labels):.3f}')
display(pd.crosstab(
    test_df[target_columns].reset_index(drop=True), 
    pd.Series(test_pred_labels, name='pred')
))<jupyter_output>Train Accuracy: 0.967
{'fraction_non_alphanumeric': 0.11687631027253668, 'fraction_numerical': 0.010744234800838574, 'mean_word_length': 7.01890756302521, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '445194', 'license': 'MIT', 'n_tokens_mistral': 1294, 'n_tokens_neox': 1235, 'n_words': 249, 'path': 'examples/01_iris.ipynb', 'repo_name': 'elda27/prepnet', 'size': '3816'}