<jupyter_start><jupyter_text>Jigsaw Multilingual Toxic Comment Classification>Note that this blogpost makes use of datasets that may be considered profane, vulgar, or offensive. - toc: true- badges: true- comments: true- author: <NAME><jupyter_code># from transformers import XLMRobertaConfig, XLMRobertaTokenizer, XLMRobertaModel
import pandas as pd
%matplotlib inline<jupyter_output><empty_output><jupyter_text>This is a really interesting competition - and in this competition I want to learn as well as teach - learn by experimenting new techniques and teach by showing the how to do the experiments. I hope you find this blogpost interesting because each week, I will perform a new experiment and get an intuition on what works and what doesn't work.  What's so interesting about this competition?  Well, firstly, the training data is only in `En` while the valid and test set contain multiple languages.<jupyter_code>trn = pd.read_csv("./jigsaw-toxic-comment-train.csv", usecols=['comment_text', 'toxic'])
trn.head()<jupyter_output><empty_output><jupyter_text>Here is a list of 5 toxic comments in the train set:<jupyter_code>list(trn.query("toxic==1").comment_text[:5])<jupyter_output><empty_output><jupyter_text>So, the training set only consists of `comment_text` field in `English` and `toxic` field that represents whether the comment is toxic or not.<jupyter_code>valid = pd.read_csv('./validation.csv')
valid.head()
set(valid.lang)<jupyter_output><empty_output><jupyter_text>**The valid set consists of 'es' (ISO code for Spanish), 'it' (ISO code for Italian) and 'tr' (ISO code for Turkish)**. You can find the ISO codes for reference [here](https://www.andiamo.co.uk/resources/iso-language-codes/). Now what about the test set?<jupyter_code>test = pd.read_csv('./test.csv')
test.head()
set(test.lang)<jupyter_output><empty_output><jupyter_text>**The valid set consists of 'es' (ISO code for Spanish), 'fr' (ISO code for French), 'it' (ISO code for Italian), 'pt' (ISO code for Portuguese, 'ru' (ISO code for Russian)  and 'tr' (ISO code for Turkish)**. You can find the ISO codes for reference [here](https://www.andiamo.co.uk/resources/iso-language-codes/). Interesting right? So can we train a toxic comment classifier on English and use it on another language? I say "No".Why? - **Tokens are different**: The first step of a language model is to create a tokenizer and convert the input text to tokens, how will our model treat two different languages or multiple different languages if it doesn't know how to tokenize them? - **No common embedding represenation**: The tokens are converted to some intermediate representations of some dimension - in case of BERT it's 768, and then the model performs the classification on these intermediate representations. How will our model know which representation to use if it's only trained on English?   So what do we do then?  Well, we make use of multilingual transformer based models. Recently there has been much research in multilingual models - one model for multiliple languages.  From what I've read through the Kaggle forums and recent research papers, there are two main multilingual models - one of them by Facebook AI - [XLM](https://arxiv.org/abs/1901.07291), [XLMR](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/) and another one by Google - [MBert](https://github.com/google-research/bert/blob/master/multilingual.md) (multilingual bert).  There is a great introduction to this competition using mBert available [here](https://www.youtube.com/watch?v=vvr_f-X_LaI). While at the same time there is also much interesting discussion about using XLM-R [here](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/140254). Now which of the two is better - I do not know. BUT, we will found out through expirementation. At the date of writing this blogpost, there are still two more months to go and while at this time I cannot aim to win the competition - I do aim to get a gold medal or at the least implement these new ideas to learn.  Could we use translation?  Another idea is to then translate the test set to English and use a mono lingual English model for toxicity classification. A translated version of the test set is available [here](https://www.kaggle.com/kashnitsky/jigsaw-multilingual-toxic-test-translated).<jupyter_code>val_translated = pd.read_csv("./jigsaw_miltilingual_valid_translated.csv")
val_translated.head()
list(val_translated.query("toxic==1").translated[5:10])<jupyter_output><empty_output><jupyter_text>Above, I have shared 10 translated comments that are toxic. The translated versions of comments do seem to be toxic if you read it in plain English and it is possible that this approach of toxicity classification might work. But, then out training set needs to have the same words for toxic classification too. One approach is to finetune the model on the valid set and use the same API (for translation) on the test set and then make predictions on the test set.> This approach assumes that the translation API used (in this case YANDEX) will not remove the toxicity and that it can mantain a good level of translation accuracy for multiple languages - especially those in the test set.  A downside of this would we that if such a model is served in production - each sentence would have to be translated first before we are able to classify the toxicity on the comment.<jupyter_code>test_translated = pd.read_csv("./jigsaw_miltilingual_test_translated.csv")
test_translated.head()
list(test_translated.translated[95:100])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08055949008498584, 'fraction_numerical': 0.0056657223796034, 'mean_word_length': 5.3046875, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 8, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '256610', 'license': 'Apache-2.0', 'n_tokens_mistral': 1591, 'n_tokens_neox': 1506, 'n_words': 704, 'path': '_notebooks/2020-04-10-JigsawIntro.ipynb', 'repo_name': 'amaarora/jigsaw', 'size': '5652'}