---
language:
- fr
tags:
- music
- rap
- lyrics
- bert
library_name: transformers
---
# WatiBERT: Fine-Tuned BERT Model for French Rap Lyrics

## Overview

__WatiBERT__ is a __BERT__ model fine-tuned on __french rap lyrics__ sourced from __Genius__. Used dataset size was __323MB__, corresponding to __85M tokens__ after tokenization.

This model is designed to understand and analyze the __semantic relationships__ within the context of __French rap__, providing a valuable tool for research in __French slang__, and __music lyrics analysis__.

## Model Details

The model is based on the __FlauBERT Large Cased__ architecture and has been fine-tuned with the following hyperparameters:

| Parameter            | Value        |
|----------------------|--------------|
| Epochs               | 5            |
| Train Batch Size     | 16           |
| Learning Rate        | 2e-5         |
| Weight Decay         | 0.01         |
| Warmup Ratio         | 0.1          |
| Dropout              | 0.1          |

## Versions

The model was trained using __AWS SageMaker__ on a single __ml.p3.2xlarge__ instance with the following software versions:

| Requirement          | Version      |
|----------------------|--------------|
| Transformers Library | 4.6          |
| PyTorch              | 1.7          |
| Python               | 3.6          |

## Installation

Install Required Python Libraries:

```bash
pip install transformers
```

## Loading the Model

To load the WatiBERT model, use the following Python code:

```python
from transformers import FlaubertTokenizer, FlaubertWithLMHeadModel

# Load the tokenizer and model
tokenizer = FlaubertTokenizer.from_pretrained("rapminerz/WatiBERT-large-cased")
model = FlaubertWithLMHeadModel.from_pretrained("rapminerz/WatiBERT-large-cased")
```

## Using the Model

BERT Models being masked-models, you can fill missing words to check it out

```python
def fill_mask(sentence, topk):
    inputs = tokenizer(sentence, return_tensors="pt")
    mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
    outputs = model(**inputs)
    logits = outputs.logits
    top_tokens_ids = logits[0, mask_token_index, :].topk(topk, dim=1).indices[0]
    top_tokens = [tokenizer.decode(token_id) for token_id in top_tokens_ids]
    return top_tokens

sentence = "La <special1> est morte hier, ils faisaient pas le poids (gang)"
fill_mask(sentence, 1)
['concurrence']

sentence = "On s'en souviendra comme le coup de tête de <special1>..."
fill_mask(sentence, 1)
['Zidane']

sentence = "Et quand je serai en haut j'achêterai une <special1> à ma daronne !"
fill_mask(sentence, 1)
['villa']

sentence = "Tout ce qui m'importe c'est faire du <special1> !"
fill_mask(sentence, 5)
['chiffre', 'cash', 'fric', 'sale', 'blé']
```

## Usages

This model can be then fined tune to serveral tasks such as : text classification, named entity recognition, question answering, text summerization, text generation, text completion, paraphrasing, language translation, sentiment analysis...

## Purpose and Disclaimer

This model is designed for academic and research purposes only. It is not intended for commercial use. The creators of this model do not endorse or promote any specific views or opinions that may be represented in the dataset.

__Please mention @RapMinerz if you use our models__

## Contact

For any questions or issues, please contact the repository owner, __RapMinerz__, at rapminerz.contact@gmail.com.