<jupyter_start><jupyter_text>Fine-tuning a Danish BERTThis tutorial will take you through how to fine-tune a BERT, both for sentence and token classification. Start by installing the requirements by running the following chunk:<jupyter_code>!pip install -r requirements.txt<jupyter_output><empty_output><jupyter_text>Importing packages<jupyter_code># native packages
import os

# widely use packages
import pandas as pd

# other packages
from simpletransformers.ner import NERModel
from simpletransformers.classification import ClassificationModel

from danlp.datasets import DDT
import pyconll<jupyter_output><empty_output><jupyter_text>--- Token ClassificationLet's start by doing a token classification. Token classification is the act of classifying tokens as is for example used to classify whether a token is an entity and what type of entity it is, e.g. person, organization or location. This is typically called named-entity recognition. Other token classification tasks include part-of-speech tagging as well as others. For this example we will train a BERT for named-entity recognition using the tagged data by DaNLP derived from the Danish dependency Treebank. We will start by loading in the data and examining it.<jupyter_code># Loading the Danish Dependency Tree data
ddt = DDT()
conllu_format = ddt.load_as_conllu(predefined_splits = True)

data = []
for n in range(len(conllu_format)):
    data.append([(i, token.form, token.misc.get("name").pop()) for i, sent in enumerate(conllu_format[n]) for token in sent]) #Getting the sentence #, Word and Tag.
# this dataset contain a training dataset
train = pd.DataFrame(data[0], columns = ['sentence_id', 'words', 'labels']) # note that the names of the columns are important for the model
# a development test dataset
test = pd.DataFrame(data[1], columns = ['sentence_id', 'words', 'labels'])
# and lastly a validation dataset
validation = pd.DataFrame(data[2], columns = ['sentence_id', 'words', 'labels'])

# examing the first ten rows we see some of the structure of the data
train.head(10)<jupyter_output><empty_output><jupyter_text>Okay so now we are ready to train the model. Beware that this process might take some time to it might be ideal to only use some of the data.<jupyter_code># get list of unique labels
unique_labels = list(train['labels'].unique())

# we will need to rename the config file from bert_config.json to config.json
# os.rename('danish_bert_uncased_v2/bert_config.json', 'danish_bert_uncased_v2/config.json')

# preparing the model
model = NERModel('bert', model_name = 'Maltehb/danish-bert-botxo', labels=unique_labels, use_cuda=False, args={'overwrite_output_dir': True, 'reprocess_input_data': True})
# Training the model
model.train_model(train)<jupyter_output><empty_output><jupyter_text>Sentence ClassificationSentence classification is the act of classifying a sentence. This could be classyfying the topic of a sentence or classifying whether a sentence is postive or negative. In this case we will try to predict the score of a trustpilot review based on the text of the review. The dataset used for this is avaliable in the [Github repository](https://github.com/KennethEnevoldsen/tutorial_fine-tuning_danish_bert).<jupyter_code>tp = pd.read_csv("trustpilot.csv")
tp.columns = ['text', 'labels'] # rename variables - not that the renames variable names are important

tp['text'] = tp['text'].astype('str')
tp['labels'] = tp['labels'] - 1 # index to zero
tp.head(10)
# number og unique labels
n_labels = len(tp['labels'].unique())

# initialize the model
sent_model = ClassificationModel('bert', 'Maltehb/danish-bert-botxo', num_labels=n_labels, use_cuda=False, args={'reprocess_input_data': True, 'overwrite_output_dir': True})
# train the model
sent_model.train_model(tp)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09071785432553248, 'fraction_numerical': 0.0026295030239284773, 'mean_word_length': 6.044444444444444, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '382124', 'license': 'MIT', 'n_tokens_mistral': 1138, 'n_tokens_neox': 1097, 'n_words': 418, 'path': 'tutorial.ipynb', 'repo_name': 'KennethEnevoldsen/tutorial_fine-tuning_danish_bert', 'size': '3803'}