README.md

---
language:
  - en
tags:
  - sentence-similarity
  - text-classification
datasets:
  - dennlinger/wiki-paragraphs
metrics:
  - f1
license: mit
---

# BERT-Wiki-Paragraphs

Authors: Satya Almasian\*, Dennis Aumiller\*, Lucienne-Sophie Marmé, Michael Gertz  
Contact us at `<lastname>@informatik.uni-heidelberg.de`  
Details for the training method can be found in our work [Structural Text Segmentation of Legal Documents](https://arxiv.org/abs/2012.03619).
The training procedure follows the same setup, but we substitute legal documents for Wikipedia in this model.
Find the associated training data here: [wiki-paragraphs](https://huggingface.co/datasets/dennlinger/wiki-paragraphs)

Training is performed in a form of weakly-supervised fashion to determine whether paragraphs topically belong together or not.
We utilize automatically generated samples from Wikipedia for training, where paragraphs from within the same section are assumed to be topically coherent.  
We use the same articles as ([Koshorek et al., 2018](https://arxiv.org/abs/1803.09337)), 
albeit from a 2021 dump of Wikpeida, and split at paragraph boundaries instead of the sentence level.

## Usage
Preferred usage is through `transformers.pipeline`:

```python
from transformers import pipeline
pipe = pipeline("text-classification", model="dennlinger/bert-wiki-paragraphs")

pipe("{First paragraph} [SEP] {Second paragraph}")
```
A predicted "1" means that paragraphs belong to the same topic, a "0" indicates a disconnect.

## Training Setup
The model was trained for 3 epochs from `bert-base-uncased` on paragraph pairs (limited to 512 subwork with the `longest_first` truncation strategy).
We use a batch size of 24 wit 2 iterations gradient accumulation (effective batch size of 48), and a learning rate of 1e-4, with gradient clipping at 5.
Training was performed on a single Titan RTX GPU over the duration of 3 weeks.