<jupyter_start><jupyter_text>Binary extension MLP for ordinal regression and deep learning -- cement strength dataset This tutorial explains how to train a deep neural network (here: multilayer perceptron) with the binary extension method by Niu at al. 2016 for ordinal regression. **Paper reference:**- Niu, Zhenxing, <NAME>, <NAME>, <NAME>, and <NAME>. "[Ordinal regression with multiple output cnn for age estimation](https://openaccess.thecvf.com/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf)." In Proceedings of the IEEE conference on computer vision and pattern recognition.  0 -- Obtaining and preparing the cement_strength dataset We will be using the cement_strength dataset from [https://github.com/gagolews/ordinal_regression_data/blob/master/cement_strength.csv](https://github.com/gagolews/ordinal_regression_data/blob/master/cement_strength.csv).First, we are going to download and prepare the and save it as CSV files locally. This is a general procedure that is not specific to CORN.This dataset has 5 ordinal labels (1, 2, 3, 4, and 5). Note that the method requires labels to be starting at 0, which is why we subtract "1" from the label column.<jupyter_code>import pandas as pd
import numpy as np


data_df = pd.read_csv("https://raw.githubusercontent.com/gagolews/ordinal_regression_data/master/cement_strength.csv")

data_df["response"] = data_df["response"]-1 # labels should start at 0

data_labels = data_df["response"]
data_features = data_df.loc[:, ["V1", "V2", "V3", "V4", "V5", "V6", "V7", "V8"]]

print('Number of features:', data_features.shape[1])
print('Number of examples:', data_features.shape[0])
print('Labels:', np.unique(data_labels.values))<jupyter_output>Number of features: 8
Number of examples: 998
Labels: [0 1 2 3 4]<jupyter_text>Split into training and test data<jupyter_code>from sklearn.model_selection import train_test_split


X_train, X_test, y_train, y_test = train_test_split(
    data_features.values,
    data_labels.values,
    test_size=0.2,
    random_state=1,
    stratify=data_labels.values)<jupyter_output><empty_output><jupyter_text>Standardize features<jupyter_code>from sklearn.preprocessing import StandardScaler


sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform(X_test)<jupyter_output><empty_output><jupyter_text>1 -- Setting up the dataset and dataloader In this section, we set up the data set and data loaders using PyTorch utilities. This is a general procedure that is not specific to the method.<jupyter_code>import torch


##########################
### SETTINGS
##########################

# Hyperparameters
random_seed = 1
learning_rate = 0.05
num_epochs = 50
batch_size = 128

# Architecture
NUM_CLASSES = 10

# Other
DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print('Training on', DEVICE)
from torch.utils.data import Dataset


class MyDataset(Dataset):

    def __init__(self, feature_array, label_array, dtype=np.float32):
    
        self.features = feature_array.astype(np.float32)
        self.labels = label_array

    def __getitem__(self, index):
        inputs = self.features[index]
        label = self.labels[index]
        return inputs, label

    def __len__(self):
        return self.labels.shape[0]
import torch
from torch.utils.data import DataLoader


# Note transforms.ToTensor() scales input images
# to 0-1 range
train_dataset = MyDataset(X_train_std, y_train)
test_dataset = MyDataset(X_test_std, y_test)


train_loader = DataLoader(dataset=train_dataset,
                          batch_size=batch_size,
                          shuffle=True, # want to shuffle the dataset
                          num_workers=0) # number processes/CPUs to use

test_loader = DataLoader(dataset=test_dataset,
                         batch_size=batch_size,
                         shuffle=False,
                         num_workers=0)

# Checking the dataset
for inputs, labels in train_loader:  
    print('Input batch dimensions:', inputs.shape)
    print('Input label dimensions:', labels.shape)
    break<jupyter_output>Input batch dimensions: torch.Size([128, 8])
Input label dimensions: torch.Size([128])<jupyter_text>2 - Equipping MLP with a modified output layer In this section, we are using the output layer as it was originally implemented by Niu et al. This is actually very similar to a standard output layer as we can see below:<jupyter_code>class MLP(torch.nn.Module):

    def __init__(self, in_features, num_classes, num_hidden_1=300, num_hidden_2=300):
        super().__init__()
        
        self.num_classes = num_classes
        self.my_network = torch.nn.Sequential(
            
            # 1st hidden layer
            torch.nn.Linear(in_features, num_hidden_1, bias=False),
            torch.nn.LeakyReLU(),
            torch.nn.Dropout(0.2),
            torch.nn.BatchNorm1d(num_hidden_1),
            
            # 2nd hidden layer
            torch.nn.Linear(num_hidden_1, num_hidden_2, bias=False),
            torch.nn.LeakyReLU(),
            torch.nn.Dropout(0.2),
            torch.nn.BatchNorm1d(num_hidden_2),
        )
        
        ### Specify Niu et al. layer
        self.fc = torch.nn.Linear(num_hidden_2, (num_classes-1)*2)
        ###--------------------------------------------------------------------###
        
    def forward(self, x):
        x = self.my_network(x) 
        logits = self.fc(x)
        
        ### Reshape logits for Niu et al. loss
        logits = logits.view(-1, (self.num_classes-1), 2)
        ###--------------------------------------------------------------------###        
        return logits
    
    
    
torch.manual_seed(random_seed)
model = MLP(in_features=8, num_classes=NUM_CLASSES)
model.to(DEVICE)

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)<jupyter_output><empty_output><jupyter_text>3 - Using the extended binary loss for model training During training, all you need to do is to 1) convert the integer class labels into the extended binary label format using the `levels_from_labelbatch` provided via `coral_pytorch`:```python        levels = levels_from_labelbatch(class_labels,                                         num_classes=NUM_CLASSES)```2) Apply the extended binary loss:```pythondef niu_et_al_loss(logits, levels):    val = (-torch.sum((F.log_softmax(logits, dim=2)[:, :, 1]*levels                      + F.log_softmax(logits, dim=2)[:, :, 0]*(1-levels)), dim=1))    return torch.mean(val)loss = niu_et_al_loss(logits, levels)```<jupyter_code>#!pip install coral-pytorch
import torch.nn.functional as F
from coral_pytorch.dataset import levels_from_labelbatch


def niu_et_al_loss(logits, levels):
    val = (-torch.sum((F.log_softmax(logits, dim=2)[:, :, 1]*levels
                      + F.log_softmax(logits, dim=2)[:, :, 0]*(1-levels)), dim=1))
    return torch.mean(val)


for epoch in range(num_epochs):
    
    model = model.train()
    for batch_idx, (features, class_labels) in enumerate(train_loader):

        ##### Convert class labels for extended binary loss
        levels = levels_from_labelbatch(class_labels, 
                                        num_classes=model.num_classes)
        ###--------------------------------------------------------------------###

        features = features.to(DEVICE)
        levels = levels.to(DEVICE)
        logits = model(features)
        
        #### Niu et al. loss loss 
        loss = niu_et_al_loss(logits, levels)
        ###--------------------------------------------------------------------###   
        
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        ### LOGGING
        if not batch_idx % 200:
            print ('Epoch: %03d/%03d | Batch %03d/%03d | Loss: %.4f' 
                   %(epoch+1, num_epochs, batch_idx, 
                     len(train_loader), loss))<jupyter_output>Epoch: 001/050 | Batch 000/007 | Loss: 6.8903
Epoch: 002/050 | Batch 000/007 | Loss: 2.6211
Epoch: 003/050 | Batch 000/007 | Loss: 2.8005
Epoch: 004/050 | Batch 000/007 | Loss: 1.1442
Epoch: 005/050 | Batch 000/007 | Loss: 0.7746
Epoch: 006/050 | Batch 000/007 | Loss: 1.3391
Epoch: 007/050 | Batch 000/007 | Loss: 0.9629
Epoch: 008/050 | Batch 000/007 | Loss: 0.8995
Epoch: 009/050 | Batch 000/007 | Loss: 1.2059
Epoch: 010/050 | Batch 000/007 | Loss: 0.9444
Epoch: 011/050 | Batch 000/007 | Loss: 0.9605
Epoch: 012/050 | Batch 000/007 | Loss: 0.8075
Epoch: 013/050 | Batch 000/007 | Loss: 0.7014
Epoch: 014/050 | Batch 000/007 | Loss: 0.8392
Epoch: 015/050 | Batch 000/007 | Loss: 0.6986
Epoch: 016/050 | Batch 000/007 | Loss: 0.5953
Epoch: 017/050 | Batch 000/007 | Loss: 0.5665
Epoch: 018/050 | Batch 000/007 | Loss: 0.8357
Epoch: 019/050 | Batch 000/007 | Loss: 0.6777
Epoch: 020/050 | Batch 000/007 | Loss: 0.6479
Epoch: 021/050 | Batch 000/007 | Loss: 0.7408
Epoch: 022/050 | Batch 000/007 | L[...]<jupyter_text>4 -- Evaluate modelFinally, after model training, we can evaluate the performance of the model. For example, via the mean absolute error and mean squared error measures.For this, we are going to use the `niu_logits_to_label` utility function below.<jupyter_code>def niu_logits_to_labels(logits):
    probas = F.softmax(logits, dim=2)[:, :, 1]
    predict_levels = probas > 0.5
    predicted_labels = torch.sum(predict_levels, dim=1)
    return predicted_labels


def compute_mae_and_mse(model, data_loader, device):

    with torch.no_grad():
    
        mae, mse, acc, num_examples = 0., 0., 0., 0

        for i, (features, targets) in enumerate(data_loader):

            features = features.to(device)
            targets = targets.float().to(device)

            logits = model(features)
            predicted_labels = niu_logits_to_labels(logits)

            num_examples += targets.size(0)
            mae += torch.sum(torch.abs(predicted_labels - targets))
            mse += torch.sum((predicted_labels - targets)**2)

        mae = mae / num_examples
        mse = mse / num_examples
        return mae, mse
train_mae, train_mse = compute_mae_and_mse(model, train_loader, DEVICE)
test_mae, test_mse = compute_mae_and_mse(model, test_loader, DEVICE)
print(f'Mean absolute error (train/test): {train_mae:.2f} | {test_mae:.2f}')
print(f'Mean squared error (train/test): {train_mse:.2f} | {test_mse:.2f}')<jupyter_output>Mean absolute error (train/test): 0.29 | 0.31
Mean squared error (train/test): 0.32 | 0.36
{'fraction_non_alphanumeric': 0.137262393733881, 'fraction_numerical': 0.050434616486770466, 'mean_word_length': 2.902348117778606, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '40383', 'license': 'MIT', 'n_tokens_mistral': 3741, 'n_tokens_neox': 3328, 'n_words': 870, 'path': 'pytorch_ipynb/ordinal/niu2016_cement.ipynb', 'repo_name': 'a1625395374/deeplearning-models', 'size': '10476'}