|
--- |
|
language: |
|
- en |
|
tags: |
|
- sentence-similarity |
|
- text-classification |
|
datasets: |
|
- dennlinger/wiki-paragraphs |
|
metrics: |
|
- f1 |
|
license: mit |
|
--- |
|
|
|
# BERT-Wiki-Paragraphs |
|
|
|
Authors: Satya Almasian\*, Dennis Aumiller\*, Lucienne-Sophie Marmé, Michael Gertz |
|
Contact us at `<lastname>@informatik.uni-heidelberg.de` |
|
Details for the training method can be found in our work [Structural Text Segmentation of Legal Documents](https://arxiv.org/abs/2012.03619). |
|
The training procedure follows the same setup, but we substitute legal documents for Wikipedia in this model. |
|
Find the associated training data here: [wiki-paragraphs](https://huggingface.co/datasets/dennlinger/wiki-paragraphs) |
|
|
|
Training is performed in a form of weakly-supervised fashion to determine whether paragraphs topically belong together or not. |
|
We utilize automatically generated samples from Wikipedia for training, where paragraphs from within the same section are assumed to be topically coherent. |
|
We use the same articles as ([Koshorek et al., 2018](https://arxiv.org/abs/1803.09337)), |
|
albeit from a 2021 dump of Wikpeida, and split at paragraph boundaries instead of the sentence level. |
|
|
|
## Usage |
|
Preferred usage is through `transformers.pipeline`: |
|
|
|
```python |
|
from transformers import pipeline |
|
pipe = pipeline("text-classification", model="dennlinger/bert-wiki-paragraphs") |
|
|
|
pipe("{First paragraph} [SEP] {Second paragraph}") |
|
``` |
|
A predicted "1" means that paragraphs belong to the same topic, a "0" indicates a disconnect. |
|
|
|
## Training Setup |
|
The model was trained for 3 epochs from `bert-base-uncased` on paragraph pairs (limited to 512 subwork with the `longest_first` truncation strategy). |
|
We use a batch size of 24 wit 2 iterations gradient accumulation (effective batch size of 48), and a learning rate of 1e-4, with gradient clipping at 5. |
|
Training was performed on a single Titan RTX GPU over the duration of 3 weeks. |
|
|
