<jupyter_start><jupyter_text>SUMMARYThis notebook reproduces my best submission that scores **1.31** on the private LB and reaches the **47th place**. The notebook implements ensemble of CNN-LSTM models using model predictions saved as Kaggle datasets. A detailed summary of the solution is published [in this discussion topic](https://www.kaggle.com/c/bms-molecular-translation/discussion/243845). The table with the main model parameters and CV performance (before beam searchg and normalization) is provided below.![models](https://i.postimg.cc/cLrTp1Pc/Screen-2021-06-04-at-10-17-02.jpg)  PREPARATIONS<jupyter_code>##### PACKAGES

import numpy as np 
import pandas as pd 
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Below we define a list with seven base models. For each of these models, test molecule predictions are stored as Kaggle datasets after beam-search with `k = 5` (thanks @tugstugi) and `RDKit`-based normalization (thanks @nofreewill). The models are sorted by their performance after beam search and normalization in the ascending order (the first model performs best).<jupyter_code>##### BASE MODELS

model_list = ['/kaggle/input/bms-norm-v22/submission_norm.csv',
              '/kaggle/input/bms-norm-v17/submission_norm.csv',
              '/kaggle/input/bms-normalization-v21/submission_norm.csv',
              '/kaggle/input/bms-normalization-v2733/submission_norm.csv',
              '/kaggle/input/bms-normalization-v20/submission_norm.csv',
              '/kaggle/input/bms-normalization-v6/submission_norm.csv',
              '/kaggle/input/bms-normalization-public/submission_norm.csv']
##### PREPARE MODEL PREDICTIONS

models = []

for model in model_list:
    sub = pd.read_csv(model)
    sub = sub.sort_values('image_id').reset_index(drop = True)
    print('- {}: {}'.format(model, sub.shape))
    models.append(sub)<jupyter_output>- /kaggle/input/bms-norm-v22/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-norm-v17/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-normalization-v21/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-normalization-v2733/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-normalization-v20/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-normalization-v6/submission_norm.csv: (1616107, 3)
- /kaggle/input/bms-normalization-public/submission_norm.csv: (1616107, 3)<jupyter_text>I also import another model that produces partial predictions for 273k molecules that proved to be more challenging to translate in my previous experiments. The predictions are done using a beam search with a larger `k`.<jupyter_code>##### IMPORT PARTIAL PREDICTIONS

part_sub = pd.read_csv('/kaggle/input/bms-normalization-bad-27/submission_norm.csv')
part_sub = part_sub.sort_values('image_id').reset_index(drop = True)
print(part_sub.shape)<jupyter_output>(273054, 3)<jupyter_text>Following the idea of @nofreewill, I store three possible events in the process of InChI normalization: `['valid', 'none', 'error']`. Value `valid` means that RDKit was able to convert prediction to a valid InChI string.<jupyter_code>##### CHECK PREDICTION FORMAT

sub = models[0].copy()
display(sub.head())
print('\nEvents:')
display(sub['event'].value_counts())<jupyter_output><empty_output><jupyter_text>ENSEMBLINGThe ensembling is done in the following way:1. If 4/7 models have the same output, set the final prediction to this mode value.2. Else:    - go through each model starting from the best-performing one and set prediction to a first model with valid output    - if there are now valid outputs, set prediction to the output of the lowest-CV model    - if output of the partial model is available and valid, overwrite prediction for this molecule<jupyter_code>##### ENSEMBLING

# placeholders
num_equals  = []
model_preds = []

# loop through test molecules
for i in tqdm(range(len(sub))):
    
    # extract base model predictions and mode
    preds     = [model.iloc[i]['InChI'] for model in models]
    mode      = max(set(preds), key = preds.count)
    num_equal = preds.count(mode)
    num_equals.append(num_equal)
    
    # set prediction to mode
    if num_equal >= 4:
        sub.loc[i, 'InChI'] = mode
        model_preds.append('mode')
        
    else:
        
        # look for valid pred from all models
        valid_pred = False
        for m in range(len(models)):
            if models[m].loc[i, 'event'] == 'valid':
                sub.loc[i, 'InChI'] = models[m].loc[i, 'InChI']
                model_preds.append(model_list[m])
                valid_pred = True
                break
                
        # set preds to lowest-CV model
        if not valid_pred:
            sub.loc[i, 'InChI'] = models[0].loc[i, 'InChI']
            model_preds.append(model_list[0])
                
        # set preds to better model if possible
        if not valid_pred:
            image_id = sub.loc[i, 'image_id']
            if image_id in list(part_sub['image_id'].values):
                if part_sub.loc[part_sub['image_id'] == image_id]['event'].item() == 'valid':
                    sub.loc[i, 'InChI'] = part_sub.loc[part_sub['image_id'] == image_id, 'InChI'].item()
                    model_preds.append('part_model')
##### CHECK NO. EQUAL PREDS DISTRIBUTION

pd.Series(num_equals).value_counts()<jupyter_output><empty_output><jupyter_text>In many cases, our models agree quite well with each other.<jupyter_code>##### CHECK MODEL PREDS DISTRIBUTION

pd.Series(model_preds).value_counts()<jupyter_output><empty_output><jupyter_text>As expected, most predictions are coming from model `v22`. Very few molecules are translated by the last models in a row, suggesting that usually at least one of the models is able to provide valid edictions.  SUBMISSION<jupyter_code>##### EXPORT SUBMISSION

sub = sub[['image_id', 'InChI']]
sub.to_csv('submission.csv', index = False)
sub.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11527426160337553, 'fraction_numerical': 0.02210970464135021, 'mean_word_length': 3.737010391686651, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '470909', 'license': 'MIT', 'n_tokens_mistral': 2026, 'n_tokens_neox': 1900, 'n_words': 510, 'path': 'notebooks/12_ensembling.ipynb', 'repo_name': 'kozodoi/BMS-Molecular-Translation', 'size': '5925'}