<jupyter_start><jupyter_text>Google Quest Q&A Labeling  1st place solution  by <NAME>, <NAME>, <NAME>, and <NAME>The purpose of this competition is to analyze StackExchange questions & answers predicting whether the question is interesting, whether the answer is helpful or misleading etc. So in theory, top solutions can help Q&A systems in getting more human-like.In a nutshell, our team trained 4 models: 2 [BERT](https://arxiv.org/abs/1810.04805) ones, one [RoBERTa](https://arxiv.org/abs/1907.11692), and one [BART](https://arxiv.org/abs/1910.13461). Key ideas are:- pretraining language models with StackExchange data and auxiliary targets- pseudo-labeling- postprocessing predictionsDetails are outlined [in this post](https://www.kaggle.com/c/google-quest-challenge/discussion/129840), code is shared in [this repository](https://github.com/oleg-yaroshevskiy/quest_qa_labeling).  **Install necessary packages** - [mag](https://github.com/ex4sperans/mag) is a lightweight library to keep track of experiments - sacremoses is a dependency for transformers - sacreBLEU and fairseq are dependencies for the BART model<jupyter_code>%%time
!pip install /kaggle/input/pythonmag/mag > /dev/null
!pip install ../input/sacremoses/sacremoses-master/ > /dev/null
!pip install /kaggle/input/sacrebleu/sacreBLEU-master/ > /dev/null
!pip install /kaggle/input/fairseq-hacked/fairseq > /dev/null
import os
import json
from collections import Counter
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Inference Model 1. BERT base uncasedThis is an uncased BERT model, its LM is finetuned with StackExchange data.<jupyter_code>%%time
!python /kaggle/input/old-bert-code/predict_test.py \
  --model_dir /kaggle/input/stackx-80-aux-ep-3       \
  --sub_file model1_bert_base_uncased_pred.csv<jupyter_output>2021-12-02 23:17:04.974678: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
100%|███████████████████████████████████████████| 60/60 [00:10<00:00,  5.98it/s]
100%|███████████████████████████████████████████| 60/60 [00:09<00:00,  6.43it/s]
100%|███████████████████████████████████████████| 60/60 [00:09<00:00,  6.38it/s]
100%|███████████████████████████████████████████| 60/60 [00:09<00:00,  6.44it/s]
100%|███████████████████████████████████████████| 60/60 [00:09<00:00,  6.21it/s]
CPU times: user 1.83 s, sys: 436 ms, total: 2.26 s
Wall time: 1min 54s<jupyter_text>Model 2. BERT base casedThis is a cased BERT model, its LM is finetuned with StackExchange data, code has been refactored w.r.t. to the first model.<jupyter_code>%%time
!python ../input/bert-base-random-code/run.py                \
  --sub_file=model2_bert_base_cased_pred.csv                  \
  --data_path=/kaggle/input/google-quest-challenge/            \
  --max_sequence_length=500                                     \
  --max_title_length=26                                          \
  --max_question_length=260                                       \
  --max_answer_length=210                                          \
  --batch_size=8                                                    \
  --bert_model=/kaggle/input/bert-base-pretrained/stackx-base-cased/<jupyter_output>2021-12-02 23:18:59.025886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Initial arguments Namespace(batch_accumulation=4, batch_size=8, bert_model='/kaggle/input/bert-base-pretrained/stackx-base-cased/', data_path='/kaggle/input/google-quest-challenge/', epochs=5, folds=5, head_tail=True, label='qa', leak_free_pseudo=False, lr=2e-05, max_answer_length=210, max_question_length=260, max_sequence_length=500, max_title_length=26, num_classes=30, pseudo_file=None, seed=42, split_pseudo=False, sub_file='model2_bert_base_cased_pred.csv', warmup=200, workers=8)
Preparing dataset: 100%|█████████████████████| 476/476 [00:03<00:00, 122.68it/s]
Fold 0
Test: 100%|█████████████████████████████████████| 60/60 [00:09<00:00,  6.62it/s]
Fold 1
Test: 100%|█████████████████████████████████████| 60/60 [00:08<00:00,  6.75it/s]
Fold 2
Test: 100%|█████████████████████████████████████| 60/60 [00:08<00:00,  6.76it/s]
Fold 3
Test[...]<jupyter_text>Model 3. RoBERTaHere we're resorting to both LM finetuning and pseudo-labeling.<jupyter_code># setups

ROBERTA_EXPERIMENT_DIR = "2-4-roberta-base-saved-5-head_tail-roberta-stackx-base-v2-pl1kksample20k-1e-05-210-260-500-26-roberta-200"
!mkdir $ROBERTA_EXPERIMENT_DIR
!ln -s /kaggle/input/roberta-stackx-base-pl20k/checkpoints $ROBERTA_EXPERIMENT_DIR/checkpoints

ROBERTA_CONFIG = {
    "_seed": 42,
    "batch_accumulation": 2,
    "batch_size": 4,
    "bert_model": "roberta-base-saved",
    "folds": 5,
    "head_tail": True,
    "label": "roberta-stackx-base-v2-pl1kksample20k",
    "lr": 1e-05,
    "max_answer_length": 210,
    "max_question_length": 260,
    "max_sequence_length": 500,
    "max_title_length": 26,
    "model_type": "roberta",
    "warmup": 200
}
with open(os.path.join(ROBERTA_EXPERIMENT_DIR, "config.json"), "w") as fp:
    json.dump(ROBERTA_CONFIG, fp)
    
!echo kek > $ROBERTA_EXPERIMENT_DIR/command
%%time
!python ../input/roberta-base-code/infer.py                 \
  --experiment $ROBERTA_EXPERIMENT_DIR                       \
  --checkpoint=best_model.pth                                 \
  --bert_model=/kaggle/input/roberta-base-model                \
  --dataframe=/kaggle/input/google-quest-challenge/test.csv     \
  --output_dir=roberta-base-output<jupyter_output>2021-12-02 23:20:35.414736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0

Fold: 0

Test: 100%|███████████████████████████████████| 119/119 [00:12<00:00,  9.35it/s]

Fold: 1

Test: 100%|███████████████████████████████████| 119/119 [00:11<00:00, 10.44it/s]

Fold: 2

Test: 100%|███████████████████████████████████| 119/119 [00:11<00:00, 10.49it/s]

Fold: 3

Test: 100%|███████████████████████████████████| 119/119 [00:11<00:00, 10.35it/s]

Fold: 4

Test: 100%|███████████████████████████████████| 119/119 [00:11<00:00, 10.49it/s]
CPU times: user 1.87 s, sys: 360 ms, total: 2.23 s
Wall time: 1min 55s<jupyter_text>Model 4. BARTBART-large, with pseudo-labeling.<jupyter_code>%%time
!python ../input/bart-code/run.py                      \
  --sub_file=model4_bart_large_pred.csv                 \
  --data_path=/kaggle/input/google-quest-challenge/      \
  --max_sequence_length=500                               \
  --max_title_length=26                                    \
  --max_question_length=260                                 \
  --max_answer_length=210                                    \
  --batch_size=4                                              \
  --bert_model=bart.large<jupyter_output>Initial arguments Namespace(batch_accumulation=4, batch_size=4, bert_model='bart.large', data_path='/kaggle/input/google-quest-challenge/', epochs=5, folds=5, head_tail=True, label='qa', leak_free_pseudo=False, lr=2e-05, max_answer_length=210, max_question_length=260, max_sequence_length=500, max_title_length=26, num_classes=30, pseudo_file=None, seed=42, split_pseudo=False, sub_file='model4_bart_large_pred.csv', warmup=200, workers=8)
| dictionary: 50264 types
Almost done
Preparing dataset: 100%|██████████████████████| 476/476 [00:04<00:00, 99.82it/s]
| dictionary: 50264 types
Almost done
Fold 0
Test: 100%|███████████████████████████████████| 119/119 [00:37<00:00,  3.21it/s]
Fold 1
Test: 100%|███████████████████████████████████| 119/119 [00:36<00:00,  3.22it/s]
Fold 2
Test: 100%|███████████████████████████████████| 119/119 [00:36<00:00,  3.22it/s]
Fold 3
Test: 100%|███████████████████████████████████| 119/119 [00:36<00:00,  3.22it/s]
Fold 4
Test: 100%|██████████████████[...]<jupyter_text>Blending and postprocessing **First, we read the 30 target columns that we need to predict.**<jupyter_code>sample_submission_df = pd.read_csv("/kaggle/input/google-quest-challenge/sample_submission.csv", 
                             index_col='qa_id')
target_columns = sample_submission_df.columns
print(f'There are {len(target_columns)} targets to predict')

train_df = pd.read_csv("/kaggle/input/google-quest-challenge/train.csv")<jupyter_output>There are 30 targets to predict<jupyter_text>**Reading submission files**<jupyter_code>model1_bert_base_uncased_pred_df = pd.read_csv("model1_bert_base_uncased_pred.csv")
model2_bert_base_cased_pred_df = pd.read_csv("model2_bert_base_cased_pred.csv")
model4_bart_large_pred_df = pd.read_csv("model4_bart_large_pred.csv")<jupyter_output><empty_output><jupyter_text>**For RoBERTa, we average predictions from 5 folds**<jupyter_code>roberta_base_dfs = [pd.read_csv(
                    os.path.join("roberta-base-output", "fold-{}.csv".format(fold))) 
                    for fold in range(5)]

model3_roberta_pred_df = roberta_base_dfs[0].copy()

for col in target_columns:
    model3_roberta_pred_df[col] = np.mean([df[col] for df in roberta_base_dfs], axis=0)<jupyter_output><empty_output><jupyter_text>**Blending**<jupyter_code>blended_df = model3_roberta_pred_df.copy()

for col in target_columns:
    blended_df[col] = (
        model1_bert_base_uncased_pred_df[col] * 0.1 +
        model2_bert_base_cased_pred_df[col] * 0.2 + 
        model3_roberta_pred_df[col] * 0.1 + 
        model4_bart_large_pred_df[col] * 0.3
    )<jupyter_output><empty_output><jupyter_text>**Applying postprocessing to the final blend, also discussed [here](https://www.kaggle.com/c/google-quest-challenge/discussion/129840).**<jupyter_code>def postprocess_single(target, ref):
    """
    The idea here is to make the distribution of a particular predicted column
    to match the correspoding distribution of the corresponding column in the
    training dataset (called ref here)
    """
    
    ids = np.argsort(target)
    counts = sorted(Counter(ref).items(), key=lambda s: s[0])
    scores = np.zeros_like(target)
    
    last_pos = 0
    v = 0
    
    for value, count in counts:
        next_pos = last_pos + int(round(count / len(ref) * len(target)))
        if next_pos == last_pos:
            next_pos += 1

        cond = ids[last_pos:next_pos]
        scores[cond] = v
        last_pos = next_pos
        v += 1
        
    return scores / scores.max()
def postprocess_prediction(prediction, actual):
    
    postprocessed = prediction.copy()
    
    for col in target_columns:
        scores = postprocess_single(prediction[col].values, actual[col].values)
        # Those are columns where our postprocessing gave substantial improvement.
        # It also helped for some others, but we didn't include them as the gain was
        # very marginal (less than 0.01)
        if col in (
            "question_conversational",
            "question_type_compare",
            "question_type_definition",
            "question_type_entity",
            "question_has_commonly_accepted_answer",
            "question_type_consequence",
            "question_type_spelling"
        ):
            postprocessed[col] = scores
            
        # scale to 0-1 interval
        v = postprocessed[col].values
        postprocessed[col] = (v - v.min()) / (v.max() - v.min())
    
    return postprocessed
postprocessed = postprocess_prediction(blended_df, train_df)<jupyter_output><empty_output><jupyter_text>**Saving the submission file.**<jupyter_code>postprocessed.to_csv("submission.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1773245765641203, 'fraction_numerical': 0.062132734185966126, 'mean_word_length': 3.5545060999606455, 'pattern_counts': {'":': 14, '<': 63, '<?xml version=': 0, '>': 49, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '423941', 'license': 'MIT', 'n_tokens_mistral': 5134, 'n_tokens_neox': 4236, 'n_words': 754, 'path': 'nlp/1st-place-solution.ipynb', 'repo_name': 'furyhawk/kaggle_practice', 'size': '11615'}