<jupyter_start><jupyter_text>"Multilingual Sequence Classifaction with the MBart Family"> "Need to do some multi-lingual sequence classification? Look no further, at least if you want to use MBart and/or the MBart-50 variety of models.  Working against the `amazon_reviews_multi` dataset I'll show you how to use the `blurr` library to configure the huggingface objects, build DataLoaders, and train a model that you can use for classifying German text.  I'll throw in a bit of the inference code so that you can see how easy `blurr` makes it to use your trained model to boot.  Let's go ..."- toc: false- branch: master- badges: true- hide_binder_badge: true- comments: true- author: <NAME>- categories: [blurr, huggingface, fastai, multilingual, sequence classification]- image: images/articles/blurr-logo-small.png- hide: false- search_exclude: false- show_tags: true<jupyter_code>!pip install ohmeow-blurr -q
!pip install datasets -q
from fastai.text.all import *
from datasets import load_dataset
from transformers import AutoModelForSequenceClassification

from blurr.utils import BLURR
from blurr.data.core import *
from blurr.modeling.core import *
from blurr import __version__ as blurr_version
from fastai import __version__ as fa_version
from torch import __version__ as pt_version
from transformers import __version__ as hft_version

print(f'Using blurr {blurr_version}')
print(f'Using pytorch {pt_version}')
print(f'Using fastai {fa_version}')
print(f'Using transformers {hft_version}')<jupyter_output>Using blurr 0.0.25
Using pytorch 1.8.1+cu101
Using fastai 2.3.1
Using transformers 4.6.1<jupyter_text>Data<jupyter_code>trn_ds, val_ds, tst_ds = load_dataset("amazon_reviews_multi", "de", split=['train', 'validation', 'test'])
trn_df = pd.DataFrame(trn_ds)
val_df = pd.DataFrame(val_ds)
tst_df = pd.DataFrame(tst_ds)<jupyter_output><empty_output><jupyter_text>> Important: Use a subset of the data when building your model to speed up developement time!After you got everything, throw the full dataset at it and go get some coffee :)<jupyter_code># testing with a subset ... (comment out to train on full dataset!!!)

# this won't work because the rows are ordered by our targets!
# trn_ds, val_ds = load_dataset("amazon_reviews_multi", "de", split=['train[:10%]', 'validation[:10%]'])

trn_df = trn_df.sample(frac=0.05)
val_df = val_df.sample(frac=0.05)
trn_df['is_valid'] = False; val_df['is_valid'] = True
df = pd.concat([trn_df, val_df])

print(len(trn_df), len(val_df), len(df))
trn_df.head()
unique_tgt_vals = trn_df.stars.value_counts()
print(unique_tgt_vals)

labels = sorted(list(df.stars.unique()))
print(labels)<jupyter_output>4    2054
3    2054
5    2025
2    1946
1    1921
Name: stars, dtype: int64
[1, 2, 3, 4, 5]<jupyter_text>huggingface objects<jupyter_code>model_name = "facebook/mbart-large-50"
model_cls = AutoModelForSequenceClassification
hf_tok_kwargs = {'src_lang': 'de_DE', 'tgt_lang': 'de_DE'}

hf_arch, hf_config, hf_tokenizer, hf_model = BLURR.get_hf_objects(model_name,  
                                                                  model_cls=model_cls,  
                                                                  tokenizer_kwargs=hf_tok_kwargs, 
                                                                  config_kwargs={'num_labels': len(labels)})

print('arch: ', type(hf_arch))
print('config: ', type(hf_config))
print('tokenizer: ', type(hf_tokenizer))
print('model: ', type(hf_model))<jupyter_output>arch:  <class 'str'>
config:  <class 'transformers.models.mbart.configuration_mbart.MBartConfig'>
tokenizer:  <class 'transformers.models.mbart.tokenization_mbart50_fast.MBart50TokenizerFast'>
model:  <class 'transformers.models.mbart.modeling_mbart.MBartForSequenceClassification'><jupyter_text>> Important: Always good to look into the config as you'll often find good defaults to use in your training and inference!<jupyter_code>hf_config<jupyter_output><empty_output><jupyter_text>DataLoaders<jupyter_code># 3 lines!  Nice!!!
blocks = (HF_TextBlock(hf_arch, hf_config, hf_tokenizer, hf_model, max_length=256), CategoryBlock)
dblock = DataBlock(blocks=blocks, get_x=ColReader('review_body'), get_y=ColReader('stars'), splitter=ColSplitter())

dls = dblock.dataloaders(df, bs=4)<jupyter_output><empty_output><jupyter_text>> Important: It's almost always useful to look at the `shape` of things in your batches (esp. when debugging)For example, when running on a colab GPU I kept getting CUDA OOM even with a batch size of just 4.  So I looked at the input_ids and saw that they were over 1,000 tokens long in some batches.  So I adjusted the `max_length` above to ensure they weren't longer that 128 characters and voila, you have the tutorial before you now.Of course, you should run this with the biggest batch size and sequence size your GPU(s) will support.<jupyter_code>xb, yb = dls.one_batch()
xb['input_ids'].shape
# what does human friendly data look like?
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=1500)<jupyter_output><empty_output><jupyter_text>Training Print out the model so we can build a custom set of parameter groups for an MBart + Sequence Classification task<jupyter_code># hf_model
def mbart_splitter(m):
  model = m.hf_model if (hasattr(m, 'hf_model')) else m
  
  embeds_modules = [
    model.model.encoder.embed_positions, 
    model.model.encoder.embed_tokens,
    model.model.decoder.embed_positions, 
    model.model.decoder.embed_tokens
  ]
 
  embeds = nn.Sequential(*embeds_modules)
  groups = L(embeds, model.model.encoder, model.model.decoder, model.classification_head)
  return groups.map(params).filter(lambda el: len(el) > 0)<jupyter_output><empty_output><jupyter_text>Configure our metrics and callbacks required by `blurr`<jupyter_code># define our metrics (see the sklearn docs for more info)
precision = Precision(average='macro')
recall = Recall(average='macro')
f1 = F1Score(average='macro')

learn_metrics = [accuracy, precision, recall, f1]
learn_cbs = [HF_BaseModelCallback]<jupyter_output><empty_output><jupyter_text>Configure our `Learner` and train away ...<jupyter_code># configure our Learner; 3 lines!
model = HF_BaseModelWrapper(hf_model)
learn = Learner(dls, model, opt_func=Adam, loss_func=CrossEntropyLossFlat(), metrics=learn_metrics, cbs=learn_cbs, splitter=mbart_splitter)
learn.freeze()

print(len(learn.opt.param_groups))
learn.lr_find(suggestions=True)
learn.fit_one_cycle(1, lr_max=7e-5)
learn.show_results(learner=learn, max_n=2, trunc_at=1500)<jupyter_output><empty_output><jupyter_text>We'll freeze all the layers with the exception of the decoder and classification_head layers (the last 2)<jupyter_code>learn.freeze_to(-2)
learn.lr_find(suggestions=True)
learn.fit_one_cycle(3, lr_max=slice(2e-8, 2e-7))
learn.recorder.plot_loss()
learn.show_results(learner=learn, max_n=2, trunc_at=1500)<jupyter_output><empty_output><jupyter_text>Inference<jupyter_code>txt = tst_df.review_body[0]
print(txt)
learn.blurr_predict(txt)
txts = list(tst_df.review_body.values[1:10])
print(txts)
learn.blurr_predict(txts)<jupyter_output>['zunächst macht der <NAME> einen soliden Eindruck. Die Magnethalterung ist auch brauchbar. Was gar nicht geht ist die Tatsache, dass die Halterung für runde Lüftungsdüsen, anders als vom Hersteller beschrieben, nicht geeignet ist! Ständig fällt das Smartphone runter. Durch das häufige Wiederanbringen ist nun auch die Gummierung kaputt, was zur Folge hat, dass die Lüftungsdüse schön zerkratzt wird! Also Schrott, der auch noch mein Auto beschädigt! Für mich ist das nicht brauchbar!', 'Siegel sowie Verpackung war beschädigt und ware war gebraucht mit Verschleiß und Fingerabdrücke. Zurück geschickt und bessere qualitativere Artikel gekauft.', 'Habe dieses Produkt NIE erhalten und das Geld wurde nicht rückerstattet!!!!!!!', 'Die Träger sind schnell abgerissen', 'Druckbild ist leider nicht akzeptabel. Die kompletten seiten werden grau eingefärbt. Verkäufer antwortet nicht auf Emails. Deshalb absolut nicht empfehlenswert.', '🤬🤬🤬 Stoff löst sich nach kurzer Zeit', 'Beim zweiten Gebrauch[...]
{'fraction_non_alphanumeric': 0.09873480525924089, 'fraction_numerical': 0.014016373108409823, 'mean_word_length': 5.565960912052117, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '301315', 'license': 'Apache-2.0', 'n_tokens_mistral': 2809, 'n_tokens_neox': 2690, 'n_words': 764, 'path': '_notebooks/2021-05-25-mbart-sequence-classification-with-blurr.ipynb', 'repo_name': 'kurianbenoy/ohmeow_website', 'size': '8075'}