<jupyter_start><jupyter_text>Tabular Data Classification with NNI in AML This simple example is to use NNI NAS 2.0(Retiarii) framework to search for the best neural architecture for tabular data classification task in Azure Machine Learning training platform.The video demo is https://www.youtube.com/watch?v=PDVqBmm7Cro and https://www.bilibili.com/video/BV1oy4y1W7GF.  Step 1: Prepare the dataset The first step is to prepare the dataset. Here we use the Titanic dataset as an example.<jupyter_code>import os
import torch
import pandas as pd

from sklearn.preprocessing import LabelEncoder
from torchvision.datasets.utils import download_url

class TitanicDataset(torch.utils.data.Dataset):
    def __init__(self, root: str, train: bool = True):
        filename = 'train.csv' if train else 'eval.csv'
        if not os.path.exists(os.path.join(root, filename)):
            download_url(os.path.join(
                'https://storage.googleapis.com/tf-datasets/titanic/', filename), root, filename)

        df = pd.read_csv(os.path.join(root, filename))
        object_colunmns = df.select_dtypes(include='object').columns.values
        for idx in df.columns:
            if idx in object_colunmns:
                df[idx] = LabelEncoder().fit_transform(df[idx])
        
        self.x = torch.tensor(df.iloc[:, 1:].values)
        self.y = torch.tensor(df.iloc[:, 0].values)

    def __len__(self):
        return len(self.y)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]
train_dataset = TitanicDataset('./data', train=True)
test_dataset = TitanicDataset('./data', train=False)<jupyter_output><empty_output><jupyter_text>Step 2: Define the Model Space Model space is defined by users to express a set of models that they want to explore, which contains potentially good-performing models. In Retiarii(NNI NAS 2.0) framework, a model space is defined with two parts: a base model and possible mutations on the base model.  Step 2.1: Define the Base Model Defining a base model is almost the same as defining a PyTorch (or TensorFlow) model. Usually, you only need to replace the code ``import torch.nn as nn`` with ``import nni.retiarii.nn.pytorch as nn`` to use NNI wrapped PyTorch modules. Below is a very simple example of defining a base model.<jupyter_code>import nni.retiarii.nn.pytorch as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, input_size):
        super().__init__()

        self.fc1 = nn.Linear(input_size, 16)
        self.bn1 = nn.BatchNorm1d(16)
        self.dropout1 = nn.Dropout(0.0)

        self.fc2 = nn.Linear(16, 16)
        self.bn2 = nn.BatchNorm1d(16)
        self.dropout2 = nn.Dropout(0.0)

        self.fc3 = nn.Linear(16, 2)

    def forward(self, x):

        x = self.dropout1(F.relu(self.bn1(self.fc1(x))))
        x = self.dropout2(F.relu(self.bn2(self.fc2(x))))
        x = F.sigmoid(self.fc3(x))
        return x
    
model_space = Net(len(train_dataset.__getitem__(0)[0]))<jupyter_output><empty_output><jupyter_text>Step 2.2: Define the Model Mutations A base model is only one concrete model, not a model space. NNI provides APIs and primitives for users to express how the base model can be mutated, i.e., a model space that includes many models. The following will use inline Mutation APIs as a simple example.<jupyter_code>import nni.retiarii.nn.pytorch as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, input_size):
        super().__init__()

        self.hidden_dim1 = nn.ValueChoice(
            [16, 32, 64, 128, 256, 512, 1024], label='hidden_dim1')
        self.hidden_dim2 = nn.ValueChoice(
            [16, 32, 64, 128, 256, 512, 1024], label='hidden_dim2')

        self.fc1 = nn.Linear(input_size, self.hidden_dim1)
        self.bn1 = nn.BatchNorm1d(self.hidden_dim1)
        self.dropout1 = nn.Dropout(nn.ValueChoice([0.0, 0.25, 0.5]))

        self.fc2 = nn.Linear(self.hidden_dim1, self.hidden_dim2)
        self.bn2 = nn.BatchNorm1d(self.hidden_dim2)
        self.dropout2 = nn.Dropout(nn.ValueChoice([0.0, 0.25, 0.5]))

        self.fc3 = nn.Linear(self.hidden_dim2, 2)

    def forward(self, x):

        x = self.dropout1(F.relu(self.bn1(self.fc1(x))))
        x = self.dropout2(F.relu(self.bn2(self.fc2(x))))
        x = F.sigmoid(self.fc3(x))
        return x

model_space = Net(len(train_dataset.__getitem__(0)[0]))<jupyter_output><empty_output><jupyter_text>Besides inline mutations, Retiarii also provides ``mutator``, a more general approach to express complex model space.  Step 3: Explore the Defined Model Space In the NAS process, the search strategy repeatedly generates new models, and the model evaluator is for training and validating each generated model. The obtained performance of a generated model is collected and sent to the search strategy for generating better models.Users can choose a proper search strategy to explore the model space, and use a chosen or user-defined model evaluator to evaluate the performance of each sampled model.  Step 3.1: Choose a Search Strategy<jupyter_code>import nni.retiarii.strategy as strategy

simple_strategy = strategy.TPEStrategy()<jupyter_output><empty_output><jupyter_text>Step 3.2: Choose or Write a Model Evaluator In the context of PyTorch, Retiarii has provided two built-in model evaluators, designed for simple use cases: classification and regression. These two evaluators are built upon the awesome library PyTorch-Lightning.<jupyter_code>import nni.retiarii.evaluator.pytorch.lightning as pl

trainer = pl.Classification(train_dataloader=pl.DataLoader(train_dataset, batch_size=16),
                                val_dataloaders=pl.DataLoader(
                                test_dataset, batch_size=16),
                                max_epochs=20)<jupyter_output><empty_output><jupyter_text>Step 4: Configure the Experiment After all the above are prepared, it is time to configure an experiment to do the model search. The basic experiment configuration is as follows:<jupyter_code>from nni.retiarii.experiment.pytorch import RetiariiExeConfig, RetiariiExperiment

exp = RetiariiExperiment(model_space, trainer, [], simple_strategy)

exp_config = RetiariiExeConfig('aml')
exp_config.experiment_name = 'titanic_example'
exp_config.trial_concurrency = 2
exp_config.max_trial_number = 20
exp_config.max_experiment_duration = '2h'
exp_config.trial_gpu_number = 1
exp_config.nni_manager_ip = '' # your nni_manager_ip<jupyter_output><empty_output><jupyter_text>Running NNI experiments on the AML(Azure Machine Learning) training service is also simple, you only need to configure the following additional fields:<jupyter_code>exp_config.training_service.use_active_gpu = True
exp_config.training_service.subscription_id = '' # your subscription id
exp_config.training_service.resource_group = '' # your resource group
exp_config.training_service.workspace_name = '' # your workspace name
exp_config.training_service.compute_target = '' # your compute target
exp_config.training_service.docker_image = ''  # your docker image<jupyter_output><empty_output><jupyter_text>Step 5: Run and View the Experiment You can launch the experiment now! Besides, NNI provides WebUI to help users view the experiment results and make more advanced analysis.<jupyter_code>exp.run(exp_config, 8081 + random.randint(0, 100))<jupyter_output><empty_output><jupyter_text>Step 6: Export the top Model Exporting the top model script is also very convenient.<jupyter_code>print('Final model:')
for model_code in exp.export_top_models():
    print(model_code)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.092845870594424, 'fraction_numerical': 0.021041557075223566, 'mean_word_length': 4.049800796812749, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '410690', 'license': 'MIT', 'n_tokens_mistral': 2473, 'n_tokens_neox': 2356, 'n_words': 700, 'path': 'examples/notebooks/tabular_data_classification_in_AML.ipynb', 'repo_name': 'AmitShtober/nni', 'size': '7604'}