<jupyter_start><jupyter_text>Iris flower datasetThe [iris flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) is a common dataset used in machine learning.It has been created <NAME> in 1936. It contains the petal length, petal width, sepal length and sepal width of 150 iris flowers from 3 different species.Dataset has been downloaded from [Kaggle](https://www.kaggle.com/uciml/iris).<jupyter_code>from pathlib import Path
import sys
import time

import matplotlib
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
import numpy as np
import pandas as pd

%matplotlib inline

print("matplotlib:", matplotlib.__version__)
print("numpy:", np.__version__)
print("pandas:", pd.__version__)

import autoclasswrapper as wrapper
print("AutoClassWrapper:", wrapper.__version__)<jupyter_output>matplotlib: 3.0.3
numpy: 1.16.2
pandas: 0.24.1
AutoClassWrapper: 1.4.1<jupyter_text>Dataset preparation<jupyter_code>df = pd.read_csv("iris.csv", index_col="Id")
df.head()
df.describe(include='all').T<jupyter_output><empty_output><jupyter_text>Apart iris species, data in this dataset are numerical values only.To demonstrate the ability of AutoClass to handle discrete values, we will convert `PetalWidthCm` column to discrete categorical values.<jupyter_code>def categorize(value):
    if value <= 0.75:
        return "small"
    elif 0.75 < value <= 1.75:
        return "medium"
    elif 1.75 < value:
        return "large"
df["PetalWidthCat"] = df["PetalWidthCm"].apply(categorize)
df.head()<jupyter_output><empty_output><jupyter_text>Save dataset in two different files. One with real values and the other one with discrete values (column `PetalWidthCat`)<jupyter_code>df.drop(["Species", "PetalWidthCm", "PetalWidthCat"], axis=1).to_csv("iris_real.tsv", sep="\t", header=True)
!head iris_real.tsv
df["PetalWidthCat"].to_csv("iris_discrete.tsv", sep="\t", header=True)
!head iris_discrete.tsv<jupyter_output>Id	PetalWidthCat
1	small
2	small
3	small
4	small
5	small
6	small
7	small
8	small
9	small<jupyter_text>Step 1 - prepare input files<jupyter_code># Create object to prepare dataset.
clust = wrapper.Input()

# Load datasets from tsv files.
clust.add_input_data("iris_real.tsv", "real scalar")
clust.add_input_data("iris_discrete.tsv", "discrete")

# Prepare input data:
# - create a final dataframe
# - merge datasets if multiple inputs
clust.prepare_input_data()

# Create files needed by AutoClass.
clust.create_db2_file()
clust.create_hd2_file()
clust.create_model_file()
# We wanted reproducible results to ease documentation.
# But bear in mind, that this parameter is not advised by authors of AutoClass C in production run.
# Use clust.create_sparams_file() instead.
clust.create_sparams_file(reproducible_run=True)
clust.create_rparams_file()<jupyter_output>2019-07-03 14:31:08 INFO     Reading data file 'iris_real.tsv' as 'real scalar' with error 0.01
2019-07-03 14:31:08 INFO     Detected encoding: ascii
2019-07-03 14:31:08 INFO     Found 150 rows and 4 columns
2019-07-03 14:31:08 DEBUG    Checking column names
2019-07-03 14:31:08 DEBUG    Index name 'Id'
2019-07-03 14:31:08 DEBUG    Column name 'SepalLengthCm'
2019-07-03 14:31:08 DEBUG    Column name 'SepalWidthCm'
2019-07-03 14:31:08 DEBUG    Column name 'PetalLengthCm'
2019-07-03 14:31:08 INFO     Checking data format
2019-07-03 14:31:08 INFO     Column 'SepalLengthCm'
2019-07-03 14:31:08 INFO     count    150.000000
2019-07-03 14:31:08 INFO     mean       5.843333
2019-07-03 14:31:08 INFO     std        0.828066
2019-07-03 14:31:08 INFO     min        4.300000
2019-07-03 14:31:08 INFO     50%        5.800000
2019-07-03 14:31:08 INFO     max        7.900000
2019-07-03 14:31:08 INFO     ---
2019-07-03 14:31:08 INFO     Column 'SepalWidthCm'
2019-07-03 14:31:08 INFO     count    150.0000[...]<jupyter_text>Step 2 - prepare run script & run autoclass<jupyter_code># Clean previous status file if a classification has already been performed.
!rm -f autoclass-run-*

# Search autoclass in path.
wrapper.search_autoclass_in_path()

# Create object to run AutoClass.
run = wrapper.Run()

# Prepare run script.
run.create_run_file()

# Run AutoClass.
run.run()<jupyter_output>2019-07-03 14:31:26 INFO     AutoClass C executable found in /home/pierre/.soft/bin/autoclass
2019-07-03 14:31:26 INFO     Writing run file
2019-07-03 14:31:26 INFO     AutoClass C executable found in /home/pierre/.soft/bin/autoclass
2019-07-03 14:31:26 INFO     AutoClass C version: AUTOCLASS C (version 3.3.6unx)
2019-07-03 14:31:26 INFO     Running clustering...<jupyter_text>Step 3 - parse and format results<jupyter_code>timer = 0
step = 2
while not Path("autoclass-run-success").exists():
    timer += step
    sys.stdout.write("\r")
    sys.stdout.write("Time: {} sec.".format(timer))
    sys.stdout.flush()
    time.sleep(step)

results = wrapper.Output()
results.extract_results()
results.aggregate_input_data()
results.write_cdt()
results.write_cdt(with_proba=True)
results.write_class_stats()
results.write_dendrogram()<jupyter_output>2019-07-03 14:31:31 INFO     Extracting autoclass results
2019-07-03 14:31:31 INFO     Found 150 cases classified in 4 classes
2019-07-03 14:31:31 INFO     Aggregating input data
2019-07-03 14:31:31 INFO     Writing classes + probabilities .tsv file
2019-07-03 14:31:31 INFO     Writing .cdt file
2019-07-03 14:31:31 INFO     Writing .cdt file (with probabilities)
2019-07-03 14:31:31 INFO     Writing class statistics
2019-07-03 14:31:31 INFO     Writing dendrogram<jupyter_text>For comparison, add class number to original dataset.<jupyter_code>df_class = pd.read_csv("autoclass_out.tsv", sep="\t", index_col="Id")
df = pd.concat([df, df_class["main-class"]], axis=1, join="outer")
df.head()<jupyter_output><empty_output><jupyter_text>Compute class distribution for iris species<jupyter_code>pd.pivot_table(df, index=["Species"], columns=["main-class"], values=[], aggfunc=len, fill_value=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1119327731092437, 'fraction_numerical': 0.09563025210084034, 'mean_word_length': 4.716618635926993, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '380108', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2446, 'n_tokens_neox': 2062, 'n_words': 559, 'path': 'notebooks/demo_iris.ipynb', 'repo_name': 'koi29doc/DUO', 'size': '5957'}