<jupyter_start><jupyter_text>FARM Building BlocksWelcome to the FARM building blocks tutorial! There are many different ways to make use of this repository, but in this notebook, we will be going through the most import building blocks that will help you harvest the rewards of a successfully trained NLP model.Happy FARMing!  1) Text ClassificationGermEval 2018 (GermEval2018) (https://projects.fzai.h-da.de/iggsa/) is an open data set containing texts that need to be classified by whether they are offensive or not. There are a set of coarse and fine labels, but here we will only be looking at the coarse set which labels each example as either OFFENSE or OTHER. To tackle this task, we are going to build a classifier that is composed of Google's BERT language model and a feed forward neural network prediction head.  Setup<jupyter_code># Let's start by adjust the working directory so that it is the root of the repository
# This should be run just once.

import os
os.chdir('../')
print("Current working directory is {}".format(os.getcwd()))
# Here are the imports we need

import torch
from farm.modeling.tokenization import BertTokenizer
from farm.data_handler.processor import GermEval18CoarseProcessor, GermEval18FineProcessor, GermEval19ImplicitProcessor
from farm.data_handler.data_silo import DataSilo
from farm.modeling.language_model import Bert
from farm.modeling.prediction_head import TextClassificationHead
from farm.modeling.adaptive_model import AdaptiveModel
from farm.experiment import initialize_optimizer
from farm.train import Trainer
# We need to fetch the right device to drive the growth of our model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Devices available: {}".format(device))<jupyter_output>Devices available: cpu<jupyter_text>Data Handling<jupyter_code># Here we initialize a tokenizer that will be used for preprocessing text
# This is the BERT Tokenizer which uses the byte pair encoding method.
# It is currently loaded with a German model

tokenizer = BertTokenizer.from_pretrained(
    pretrained_model_name_or_path="bert-base-cased",
    do_lower_case=False)
# We can test out how it will do on an example sentence

EXAMPLE_SENTENCE = "Selbst ein blindes Huhn findet mal ein Korn."
tokenizer.tokenize(EXAMPLE_SENTENCE)
# In order to prepare the data for the model, we need a set of
# functions to transform data files into PyTorch Datasets.
# We group these together in Processor objects.
# We will need a new Processor object for each new source of data.
# The abstract class can be found in farm.data_handling.processor.Processor

processor = GermEval19ImplicitProcessor(tokenizer=tokenizer,
                                      max_seq_len=128,
                                      data_dir="data/germeval18",
                                      train_filename="INSAB_training.csv",
                                      test_filename="INSAB_test.csv")
# We need a DataSilo in order to keep our train, dev and test sets separate.
# The DataSilo will call the functions in the Processor to generate these sets.
# From the DataSilo, we can fetch a PyTorch DataLoader object which will
# be passed on to the model.
# Here is a good place to define a batch size for the model

BATCH_SIZE = 32

data_silo = DataSilo(
    processor=processor,
    batch_size=BATCH_SIZE)<jupyter_output>03/31/2021 18:41:10 - INFO - farm.data_handler.data_silo -   Loading train set from: data/germeval18/INSAB_training.csv
03/31/2021 18:53:34 - INFO - farm.data_handler.processor -   *** Show 3 random examples ***
03/31/2021 18:53:34 - INFO - farm.data_handler.processor -   *** Example ***
ID: INSAB_training-59820-0
Clear Text: 
 	text: Maps  The Graphic Designer's Barnstar For the numerous maps you've created for many, many articles. (Couldn't find a map barnstar)   
 	label: OTHER
Tokenized: 
 	 tokens: ['Map', '##s', 'The', 'Graphic', 'Designer', "'", 's', 'Bar', '##nst', '##ar', 'For', 'the', 'numerous', 'maps', 'you', "'", 've', 'created', 'for', 'many', ',', 'many', 'articles', '.', '(', 'Couldn', "'", 't', 'find', 'a', 'map', 'barn', '##star', ')']
 	offsets: [0, 3, 6, 10, 18, 26, 27, 29, 32, 35, 38, 42, 46, 55, 60, 63, 64, 67, 75, 79, 83, 85, 90, 98, 100, 101, 107, 108, 110, 115, 117, 121, 125, 129]
 	start_of_word: [True, False, True, True, True, False, False, True, False, False[...]<jupyter_text>Modeling In FARM, we make a strong distinction between the language model and prediction head so that you can mix and match different building blocks for your needs.For example, in the transfer learning paradigm, you might have the one language model that you will be using for both document classification and NER. Or you perhaps you have a pretrained language model which you would like to adapt to your domain, then use for a downstream task such as question answering. All this is possible within FARM and requires only the replacement of a few modular components, as we shall see below.Let's first have a look at how we might set up a model.<jupyter_code># The language model is the foundation on which modern NLP systems are built.
# They encapsulate a general understanding of sentence semantics
# and are not specific to any one task.

# Here we are using Google's BERT model as implemented by HuggingFace. 
# The model being loaded is a German model that we trained. 
# You can also change the MODEL_NAME_OR_PATH to point to a BERT model that you
# have saved or download one connected to the HuggingFace repository.
# See farm.modeling.language_model.PRETRAINED_MODEL_ARCHIVE_MAP for a list of
# available models

MODEL_NAME_OR_PATH = "bert-base-cased"

language_model = Bert.load(MODEL_NAME_OR_PATH)
# A prediction head is a model that processes the output of the language model
# for a specific task.
# Prediction heads will look different depending on whether you're doing text classification
# Named Entity Recognition (NER), question answering or some other task.
# They should generate logits over the available prediction classes and contain methods
# to convert these logits to losses or predictions 

# Here we use TextClassificationHead which receives a single fixed length sentence vector
# and processes it using a feed forward neural network. layer_dims is a list of dimensions:
# [input_dims, hidden_1_dims, hidden_2_dims ..., output_dims]

# Here by default we have a single layer network.
# It takes in a vector of length 768 (the default size of BERT's output).
# It outputs a vector of length 2 (the number of classes in the GermEval18 (coarse) dataset)

LAYER_DIMS = [768, 4]

prediction_head = TextClassificationHead(layer_dims=LAYER_DIMS)
# The language model and prediction head are coupled together in the Adaptive Model.
# This class takes care of model saving and loading and also coordinates
# cases where there is more than one prediction head.

# EMBEDS_DROPOUT_PROB is the probability that an element of the output vector from the
# language model will be set to zero.
EMBEDS_DROPOUT_PROB = 0.1

model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[prediction_head],
    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,
    lm_output_types=["per_sequence"],
    device=device)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code># Here we initialize a Bert Adam optimizer that has a linear warmup and warmdown
# Here you can set learning rate, the warmup proportion and number of epochs to train for

LEARNING_RATE = 2e-5
WARMUP_PROPORTION = 0.1
N_EPOCHS = 1

optimizer, warmup_linear = initialize_optimizer(
    model=model,
    learning_rate=LEARNING_RATE,
    warmup_proportion=WARMUP_PROPORTION,
    n_examples=data_silo.n_samples("train"),
    batch_size=data_silo.batch_size,
    n_epochs=N_EPOCHS)
# Training loop handled by this
# It will also trigger evaluation during training using the dev data
# and after training using the test data.

# Set N_GPU to a positive value if CUDA is available
N_GPU = 0

trainer = Trainer(
    optimizer=optimizer,
    data_silo=data_silo,
    epochs=N_EPOCHS,
    n_gpu=N_GPU,
    warmup_linear=warmup_linear,
    device=device,
)
model = trainer.train(model)<jupyter_output>03/27/2021 18:49:51 - INFO - farm.train -   ***** Running training *****
Train epoch 1/1:   0%|          | 0/141 [00:00<?, ?it/s]/Users/yasen/Desktop/учёба/Алиев/KONVENS2019_and_LREC2020/farm/modeling/optimization.py:341: UserWarning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, *, Number alpha) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:1005.)
  next_m.mul_(beta1).add_(1 - beta1, grad)
Train epoch 1/1:  71%|███████   | 100/141 [1:43:10<44:42, 65.42s/it]  
Evaluating:   0%|          | 0/16 [00:00<?, ?it/s][A
Evaluating:   6%|▋         | 1/16 [00:11<02:48, 11.24s/it][A
Evaluating:  12%|█▎        | 2/16 [00:22<02:37, 11.23s/it][A
Evaluating:  19%|█▉        | 3/16 [00:33<02:25, 11.22s/it][A
Evaluating:  25%|██▌       | 4/16 [00:44<02:15, 11.25s/it][A
Evaluating:  31%|███▏      | 5/16 [00:56<02:05, 11.42s/it][A
Evaluating:  38%|███▊      | 6/16 [01:07<01:53,[...]<jupyter_text>Switch to NER In a transfer learning paradigm, there is a core computation that is shared amongst all tasks. FARM's modular structure means that you can easily swap out different building blocks to make the same language model work for many different tasks.We can adapt the above text classification model to NER by simply switching out the processor and prediction head.<jupyter_code># Import the new building blocks

from farm.data_handler.processor import CONLLProcessor
from farm.modeling.prediction_head import TokenClassificationHead
# This processor will preprocess the data for the CoNLL03 NER task

ner_processor = CONLLProcessor(tokenizer=tokenizer,
                               max_seq_len=128,
                               data_dir="data/conll03de")
# This prediction head is also a feed forward neural network but expects one
# vector per token in the input sequence and will generate a set of logits
# for each input

LAYER_DIMS = [768, 13]

ner_prediction_head = TokenClassificationHead(layer_dims=LAYER_DIMS)
# We can integrate these new pieces with the rest using this code
# It is pretty much the same structure as what we had above for text classification

BATCH_SIZE = 32
EMBEDS_DROPOUT_PROB = 0.1
LEARNING_RATE = 2e-5
WARMUP_PROPORTION = 0.1
N_EPOCHS = 1
N_GPU = 0

data_silo = DataSilo(
    processor=ner_processor,
    batch_size=BATCH_SIZE)

model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[ner_prediction_head],
    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,
    lm_output_types=["per_token"],
    device=device)

optimizer, warmup_linear = initialize_optimizer(
    model=model,
    learning_rate=LEARNING_RATE,
    warmup_proportion=WARMUP_PROPORTION,
    n_examples=data_silo.n_samples("train"),
    batch_size=BATCH_SIZE,
    n_epochs=N_EPOCHS)

trainer = Trainer(
    optimizer=optimizer,
    data_silo=data_silo,
    epochs=N_EPOCHS,
    n_gpu=N_GPU,
    warmup_linear=warmup_linear,
    device=device,
)
model = trainer.train(model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08172905525846702, 'fraction_numerical': 0.03235294117647059, 'mean_word_length': 4.305437352245863, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '238375', 'license': 'Apache-2.0', 'n_tokens_mistral': 3757, 'n_tokens_neox': 3415, 'n_words': 1292, 'path': 'tutorials/1_farm_building_blocks.ipynb', 'repo_name': 'Yakonick/KONVENS2019_and_LREC2020', 'size': '11220'}