<jupyter_start><jupyter_text>IrisHere are some of the information provided by the official website:```textThis is perhaps the best known database to be found in the pattern recognition literature.The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant.Predicted attribute: class of iris plant.```And here's the pandas-view of the raw data:```text      f0   f1   f2   f3           label0    5.1  3.5  1.4  0.2     Iris-setosa1    4.9  3.0  1.4  0.2     Iris-setosa2    4.7  3.2  1.3  0.2     Iris-setosa3    4.6  3.1  1.5  0.2     Iris-setosa4    5.0  3.6  1.4  0.2     Iris-setosa..   ...  ...  ...  ...             ...145  6.7  3.0  5.2  2.3  Iris-virginica146  6.3  2.5  5.0  1.9  Iris-virginica147  6.5  3.0  5.2  2.0  Iris-virginica148  6.2  3.4  5.4  2.3  Iris-virginica149  5.9  3.0  5.1  1.8  Iris-virginica[150 rows x 5 columns]```> We didn't use pandas in our code, but it is convenient to visualize some data with it though 🤣>> You can download the raw data (`iris.data`) with [this link](https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data).<jupyter_code># preparations

import os
import torch
import pickle
import cflearn
import numpy as np

np.random.seed(142857)
torch.manual_seed(142857)<jupyter_output><empty_output><jupyter_text>Basic UsagesTraditionally, we need to process the raw data before we feed them into our machine learning models (e.g. encode the label column, which is a string column, into an ordinal column). In carefree-learn, however, we can train neural networks directly on files without worrying about the rest:<jupyter_code>m = cflearn.make().fit("iris.data")<jupyter_output><empty_output><jupyter_text>What's going under the hood is that carefree-learn will try to parse the `iris.data` automatically (with the help of [carefree-data](https://github.com/carefree0910/carefree-data)), split the data into training set and validation set, with which we'll train a fully connected neural network (fcnn).We can further inspect the processed data if we want to know how carefree-learn actually parsed the input data:<jupyter_code>print(m.tr_data.raw.x[0])
print(m.tr_data.raw.y[0])
print(m.tr_data.processed.x[0])
print(m.tr_data.processed.y[0])<jupyter_output>['4.6', '3.6', '1.0', '0.2']
['Iris-setosa']
[-1.5065205  1.2634597 -1.5687355 -1.3129768]
[0]<jupyter_text>It shows that the raw data is carefully normalized into numerical data that neural networks can accept. You may also notice that the first elements are not identical with the first line of the raw data, this is caused by the auto-shuffle mechanism introduced in [carefree-data](https://github.com/carefree0910/carefree-data).What's more, by saying *normalized*, it means that the input features will be automatically normalized to `mean=0.0` and `std=1.0`:<jupyter_code>tr_x = m.tr_data.processed.x
cv_x = m.cv_data.processed.x
stacked = np.vstack([tr_x, cv_x])
print(stacked.mean(0))
print(stacked.std(0))<jupyter_output>[ 3.1739475e-08 -3.7471455e-07 -1.9907951e-07 -8.0267590e-08]
[0.99999976 0.9999997  1.0000002  0.9999999 ]<jupyter_text>> The results shown above means we first normalized the data before we actually split it into train & validation set.After training on files, carefree-learn can predict & evaluate on files directly as well. We'll handle the data parsing and normalization for you automatically:<jupyter_code># `contains_labels` is set to True because `iris.data` itself contains labels
predictions = m.predict("iris.data", contains_labels=True)
# evaluations could be achieved easily with cflearn.evaluate
cflearn.evaluate("iris.data", pipelines=m)<jupyter_output>~~~  [ info ] Results
================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|          fcnn          |    0.926667    |    0.000000    |    0.926667    |    0.994800    |    0.000000    |    0.994800    |
================================================================================================================================<jupyter_text>BenchmarkingAs we know, neural networks are trained with **_stochastic_** gradient descent (and its variants), which will introduce some randomness to the final result, even if we are training on the same dataset. In this case, we need to repeat the same task several times in order to obtain the bias & variance of our neural networks.Fortunately, carefree-learn introduced [`repeat_with`](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributedrepeat_with) API, which can achieve this goal easily with only a few lines of code:<jupyter_code># With num_repeat=3 specified, we'll train 3 models on `iris.data`.
result = cflearn.repeat_with("iris.data", num_repeat=3)
cflearn.evaluate("iris.data", pipelines=result.pipelines)<jupyter_output>~~~  [ info ] Results
================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|          fcnn          |    0.902222    |    0.019116    |    0.883106    |    0.985778    |    0.004722    |    0.981055    |
================================================================================================================================<jupyter_text>We can also compare the performances across different models:<jupyter_code># With models=["linear", "fcnn"], we'll train both linear models and fcnn models.
models = ["linear", "fcnn"]
result = cflearn.repeat_with("iris.data", models=models, num_repeat=3)
cflearn.evaluate("iris.data", pipelines=result.pipelines)<jupyter_output>~~~  [ info ] Results
================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|          fcnn          | -- 0.915556 -- | -- 0.027933 -- | -- 0.887623 -- | -- 0.985467 -- | -- 0.004121 -- | -- 0.981345 -- |
--------------------------------------------------------------------------------------------------------------------------------
|         linear         |    0.620000    |    0.176970    |    0.443030   [...]<jupyter_text>It is worth mentioning that carefree-learn supports [distributed training](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributeddistributed-training), which means when we need to perform large scale benchmarking (e.g. train 100 models), we could accelerate the process through multiprocessing:> In carefree-learn, distributed training doesn't mean training your model on multiple GPUs or multiple machines. Instead, distributed training in carefree-learn means training multiple models at the same time. Please refer to our [documentation](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributeddistributed-training) for more details.<jupyter_code># With num_jobs=2, we will launch 2 processes to run the tasks in a distributed way.
result = cflearn.repeat_with("iris.data", num_repeat=10, num_jobs=2)<jupyter_output><empty_output><jupyter_text>On iris dataset, however, launching distributed training will actually hurt the speed because iris dataset only contains 150 samples, so the relative overhead brought by distributed training will be too large. Please refer the **CAUTION** section of our [documentation](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributedbenchmarking) for more details. Advanced BenchmarkingBut this is not enough, because we want to know whether other models (e.g. scikit-learn models) could achieve a better performance than carefree-learn models. In this case, we can perform an advanced benchmarking with the [Experiment](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributedexperiment) helper class.<jupyter_code>experiment = cflearn.Experiment()
tr_x, tr_y = m.tr_data.processed.xy
cv_x, cv_y = m.cv_data.processed.xy
data_folder = experiment.dump_data_bundle(tr_x, tr_y, cv_x, cv_y)

# Add carefree-learn tasks
for model in ["linear", "fcnn"]:
    experiment.add_task(model=model, data_folder=data_folder)
# Add scikit-learn tasks
run_command = f"python run_sklearn.py"
common_kwargs = {"run_command": run_command, "data_folder": data_folder}
experiment.add_task(model="decision_tree", **common_kwargs)
experiment.add_task(model="random_forest", **common_kwargs)<jupyter_output><empty_output><jupyter_text>Notice that we specified `run_command="python run_sklearn.py"` for scikit-learn tasks, which means [Experiment](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributedexperiment) will try to execute this command in the current working directory for training scikit-learn models. The good news is that we do not need to speciy any command line arguments, because [Experiment](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributedexperiment) will handle those for us.Here is basically what a `run_sklearn.py` should look like ([source code](run_sklearn.py)):```pythonimport osimport picklefrom sklearn.tree import DecisionTreeClassifierfrom sklearn.ensemble import RandomForestClassifierfrom cflearn.dist.runs._utils import get_infoif __name__ == "__main__":    info = get_info()    kwargs = info.kwargs     data    data_list = info.data_list    x, y = data_list[:2]     model    model = kwargs["model"]    if model == "decision_tree":        base = DecisionTreeClassifier    elif model == "random_forest":        base = RandomForestClassifier    else:        raise NotImplementedError    sk_model = base()     train & save    sk_model.fit(x, y.ravel())    with open(os.path.join(info.workplace, "sk_model.pkl"), "wb") as f:        pickle.dump(sk_model, f)```With `run_sklearn.py` defined, we could run those tasks with one line of code:<jupyter_code>results = experiment.run_tasks()<jupyter_output><empty_output><jupyter_text>After finished running, we should be able to see the following file structure in the current working directory:```text|--- __experiment__   |--- __data__      |-- x.npy      |-- y.npy      |-- x_cv.npy      |-- y_cv.npy   |--- fcnn/0      |-- _logs      |-- __meta__.json      |-- cflearn^_^fcnn^_^0000.zip   |--- linear/0      |-- ...   |--- decision_tree/0      |-- __meta__.json      |-- sk_model.pkl   |--- random_forest/0      |-- ...```As we expected, carefree-learn models are saved into zip files, while scikit-learn models are saved into `sk_model.pkl` files. Since these models are not yet loaded, we should manually load them into our environment:<jupyter_code>pipelines = {}
sk_patterns = {}
for workplace, workplace_key in zip(results.workplaces, results.workplace_keys):
    model = workplace_key[0]
    if model not in ["decision_tree", "random_forest"]:
        pipelines[model] = cflearn.task_loader(workplace)
    else:
        model_file = os.path.join(workplace, "sk_model.pkl")
        with open(model_file, "rb") as f:
            sk_model = pickle.load(f)
            # In `carefree-learn`, we treat labels as column vectors.
            # So we need to reshape the outputs from the scikit-learn models.
            sk_predict = lambda x: sk_model.predict(x).reshape([-1, 1])
            sk_predict_prob = lambda x: sk_model.predict_proba(x)
            sk_pattern = cflearn.ModelPattern(
                predict_method=sk_predict,
                predict_prob_method=sk_predict_prob,
            )
            sk_patterns[model] = sk_pattern<jupyter_output><empty_output><jupyter_text>After which we can finally perform benchmarking on these models:<jupyter_code>cflearn.evaluate(cv_x, cv_y, pipelines=pipelines, other_patterns=sk_patterns)<jupyter_output>~~~  [ info ] Results
================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|     decision_tree      | -- 0.960000 -- | -- 0.000000 -- | -- 0.960000 -- | -- 0.998667 -- | -- 0.000000 -- | -- 0.998667 -- |
--------------------------------------------------------------------------------------------------------------------------------
|          fcnn          | -- 0.960000 -- | -- 0.000000 -- | -- 0.960000 --[...]<jupyter_text>Seems that scikit-learn models are better than carefree-learn models! This is not surprising because neural networks often require more data than traditional machine learning algorithms. However, we can boost carefree-learn models with AutoML, as shown in the next section. AutoML on IrisAs mentioned in the introduction, carefree-learn is actually a minimal Automatic Machine Learning (AutoML) solution for tabular datasets. Up till now we haven't mentioned any AutoML stuffs yet, so in this section we'll illustrate how to perform AutoML on Iris dataset, as well as how to pack the AutoML results into production.Since carefree-learn has provided the [cflearn.Auto](https://carefree0910.me/carefree-learn-doc/docs/user-guides/auto-ml) API for out-of-the-box usages, AutoML in carefree-learn could be achieved in two lines of code:<jupyter_code>auto = cflearn.Auto("clf", models="fcnn")
auto.fit(tr_x, tr_y, cv_x, cv_y)<jupyter_output><empty_output><jupyter_text>We can make predictions directly with `auto.predict`:<jupyter_code>predictions = auto.predict(cv_x)
print("accuracy:", (predictions == cv_y).mean())<jupyter_output>accuracy: 0.9866666666666667<jupyter_text>And of course, we can compare it with other models:<jupyter_code>all_patterns = sk_patterns.copy()
all_patterns["auto"] = auto.pattern
cflearn.evaluate(cv_x, cv_y, pipelines=pipelines, other_patterns=all_patterns)<jupyter_output>~~~  [ info ] Results
================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|          auto          | -- 0.986667 -- | -- 0.000000 -- | -- 0.986667 -- | -- 0.998933 -- | -- 0.000000 -- | -- 0.998933 -- |
--------------------------------------------------------------------------------------------------------------------------------
|     decision_tree      |    0.960000    | -- 0.000000 -- |    0.960000   [...]<jupyter_text>Bravo! Our AutoML model beats the scikit-learn models 🥳If we are satisfied with the results, we can pack the models up into a zip file<jupyter_code>auto.pack("pack")<jupyter_output><empty_output><jupyter_text>which could be used on our production environments / machines easily:<jupyter_code>unpacked = cflearn.Auto.unpack("pack")
predictions = unpacked.pattern.predict(cv_x)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2389417114510128, 'fraction_numerical': 0.03566999350380913, 'mean_word_length': 2.148754183711417, 'pattern_counts': {'":': 5, '<': 61, '<?xml version=': 0, '>': 66, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '318265', 'license': 'MIT', 'n_tokens_mistral': 4833, 'n_tokens_neox': 4357, 'n_words': 1284, 'path': 'examples/iris/iris.ipynb', 'repo_name': 'vishalbelsare/carefree-learn', 'size': '16933'}