<jupyter_start><jupyter_text>Define directories with data  [ assumes you have ran download_data.sh ]<jupyter_code>top_data_dir = '../../data/main_data_for_chairs'
top_pretrained_feat_dir = '../../data/main_data_for_chairs/pretrained_features'
top_image_dir = osp.join(top_data_dir, 'images/shapenet/')
vis_utils.top_image_dir = top_image_dir<jupyter_output><empty_output><jupyter_text>Load data (Assumes you preprocessed the data, with notebooks/prepare_chair_in_context_data.ipynb)<jupyter_code># Load game-data
data_name = 'game_data.pkl'
game_data, word_to_int, int_to_word, int_to_sn_model, sn_model_to_int, sorted_sn_models =\
unpickle_data(osp.join(top_data_dir, data_name))

# Load pre-trained 2D/3D features.
vgg_feats_file = osp.join(top_pretrained_feat_dir, 'shapenet_chair_vgg_fc7_embedding.pkl')
vgg_feats = vgg_image_features(int_to_sn_model, 'chair', vgg_feats_file, python2_to_3=True)

pc_feats_file = osp.join(top_pretrained_feat_dir, 'shapenet_chair_pcae_128bneck_chamfer_embedding.npz')
pc_feats = pc_ae_features(int_to_sn_model, pc_feats_file)<jupyter_output><empty_output><jupyter_text>Prepare data for training (train/val splits, max-size, keep-only-correct etc.)<jupyter_code>max_seq_len = 33              # Maximum size (in tokens) per utterance.
split_sizes = [0.8, 0.1, 0.1] # Train-val-test sizes.
random_seed = 2004
unique_test_geo = True        # If true, the test/train/val splits have 'targets' that are disjoint sets.
only_correct = True           # Drop all not correctly guessed instances.

net_data, split_ids, _, net_data_mask = \
make_dataset_for_rnn_based_model(game_data, 
                                 split_sizes, 
                                 max_seq_len,
                                 drop_too_long=True,
                                 only_correct=only_correct,
                                 unique_test_geo=unique_test_geo,
                                 replace_not_in_train=True,
                                 geo_condition=None, 
                                 bias_train=False, 
                                 seed=random_seed)


dataloaders = {}
num_workers = 8
batch_size = 2048 # Yes this is a big batch-size.

for split in ['train', 'val', 'test']:
    dataset = ShapeglotDataset(net_data[split])
    dataloaders[split] = torch.utils.data.DataLoader(dataset=dataset,
                                                     batch_size=batch_size,
                                                     shuffle=split=='train',
                                                     num_workers=num_workers)<jupyter_output>unique geometries in train/test/val 3608 451 451<jupyter_text>Define Models<jupyter_code>embedding_dim = 100  # Dimension of INPUT space of LSTM
n_hidden = 100       # LSTM hidden size
vocab_size = len(int_to_word)
train_epochs = 100
learning_rate = 0.005
reg_gamma = 0.005    # Weight regularization on FC layers

visual_encoder = PretrainedFeatures(torch.Tensor(vgg_feats), embed_size=embedding_dim)
pc_encoder = PretrainedFeatures(torch.Tensor(pc_feats), embed_size=embedding_dim)
lang_encoder = LanguageEncoder(n_hidden, embedding_dim, vocab_size)
mlp_decoder = MLPDecoder(n_hidden*2, [100, 50, 1], True)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
listener = Listener(lang_encoder, visual_encoder, mlp_decoder, pc_encoder).to(device)
optimizer = optim.Adam(listener.parameters(), lr=learning_rate)<jupyter_output><empty_output><jupyter_text>Train the model (if you let it run for 150 epochs should reach ~78% accuracy)<jupyter_code>best_val_accuracy = 0.0
best_test_accuracy = 0.0

for epoch in range(1, train_epochs+1):
    for phase in ['train', 'val', 'test']:
        if phase == 'train':            
            listener.train()
        else:
            listener.eval()

        running_loss = 0.0
        running_corrects = 0
        
        # Iterate over data.
        for context_ids, targets, tokens in dataloaders[phase]:
            context_ids = context_ids.to(device)
            targets = targets.to(device)
            tokens = tokens.to(device)
    
            with torch.set_grad_enabled(phase == 'train'):
                logits = listener(context_ids, tokens)
                loss = smoothed_cross_entropy(logits, targets)                            
                reg_loss = 0.0
                
                for p in visual_encoder.named_parameters():
                    if 'fc.weight' == p[0]:
                        reg_loss += p[1].norm(2)
                
                for p in pc_encoder.named_parameters():
                    if 'fc.weight' == p[0]:
                        reg_loss += p[1].norm(2)
                        
                reg_loss *= reg_gamma                
                loss += reg_loss
                
                _, preds = torch.max(logits, 1)

                # backward + optimize only if in training phase
                if phase == 'train':
                    listener.zero_grad()
                    loss.backward()
                    optimizer.step()
            
            # statistics
            running_loss += loss.item() * targets.size(0)
            running_corrects += torch.sum(preds == targets)
        
        n_examples = len(dataloaders[phase].dataset)
        epoch_loss = running_loss / n_examples
        epoch_acc = running_corrects.double() / n_examples
        
        if phase == 'val':
            if epoch_acc > best_val_accuracy:
                best_val_accuracy = epoch_acc
                val_improved = True
            else:
                val_improved = False
        
        if phase == 'test' and val_improved:
            best_test_accuracy = epoch_acc
                        
        print('{} Epoch {} Loss: {:.4f} Acc: {:.4f}'.format(epoch, phase, epoch_loss, epoch_acc))
    print('Best test so far: {:.4f}'.format(best_test_accuracy))<jupyter_output>1 Epoch train Loss: 1.1431 Acc: 0.3885
1 Epoch val Loss: 1.1139 Acc: 0.3899
1 Epoch test Loss: 1.1163 Acc: 0.4039
Best test so far: 0.4039
2 Epoch train Loss: 1.0708 Acc: 0.4920
2 Epoch val Loss: 1.0199 Acc: 0.5501
2 Epoch test Loss: 1.0176 Acc: 0.5522
Best test so far: 0.5522
3 Epoch train Loss: 1.0080 Acc: 0.5686
3 Epoch val Loss: 0.9749 Acc: 0.6013
3 Epoch test Loss: 0.9694 Acc: 0.6083
Best test so far: 0.6083
4 Epoch train Loss: 0.9664 Acc: 0.6108
4 Epoch val Loss: 0.9389 Acc: 0.6328
4 Epoch test Loss: 0.9209 Acc: 0.6575
Best test so far: 0.6575
5 Epoch train Loss: 0.9358 Acc: 0.6409
5 Epoch val Loss: 0.9121 Acc: 0.6440
5 Epoch test Loss: 0.9028 Acc: 0.6692
Best test so far: 0.6692
6 Epoch train Loss: 0.9127 Acc: 0.6555
6 Epoch val Loss: 0.8992 Acc: 0.6684
6 Epoch test Loss: 0.8857 Acc: 0.6849
Best test so far: 0.6849
7 Epoch train Loss: 0.8881 Acc: 0.6762
7 Epoch val Loss: 0.8775 Acc: 0.6867
7 Epoch test Loss: 0.8667 Acc: 0.6988
Best test so far: 0.6988
8 Epoch train Loss: 0.8745 [...]<jupyter_text>Collect responses of a trained (!) listener (last epoch) and visualize what its predictions.<jupyter_code>listener.eval()
guesses = []
scores = []
phase = 'test'
score = 0.0
for context_ids, targets, tokens in dataloaders[phase]:
    context_ids = context_ids.to(device)
    targets = targets.to(device)
    tokens = tokens.to(device)        
    with torch.set_grad_enabled(phase == 'train'):
        logits = listener(context_ids, tokens)
        scores.append(F.softmax(logits, dim=1).to('cpu').numpy())
        _, preds = torch.max(logits, 1)
        guesses.append(preds.to('cpu').numpy())
        score += torch.sum(preds == targets)
score = score.item() / float(len(dataloaders[phase].dataset))
guesses = np.hstack(guesses)
scores = np.vstack(scores)
test_data = dataloaders['test'].dataset
rid = np.random.randint(len(test_data))
chair_ids, target, utterance = test_data[rid]
guess = guesses[rid] == target
print ('Predicted Probabilities: {:.2f} {:.2f} {:.2f}'.format(*scores[rid]))
visualize_example(chair_ids, utterance, target, sorted_sn_models, int_to_word, guess=guess)<jupyter_output>Predicted Probabilities: 0.00 0.19 0.80
Utterance: it is round , open at the lower back , and only three legs can be seen 
Guessed correct: True
Target: right-most
{'fraction_non_alphanumeric': 0.09988996209805599, 'fraction_numerical': 0.045726861474507885, 'mean_word_length': 2.4987168520102654, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '422074', 'license': 'MIT', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2637, 'n_words': 637, 'path': 'shapeglot/notebooks/train_listener.ipynb', 'repo_name': 'optas/shapeglot', 'size': '8179'}