Update README.md

README.md

@@ -114,10 +114,55 @@ and context of a document, which is quite an improvement.
  'semantics' 'statistics' 'text analysis' 'transformers']
 ```
 
+### 🛑 Limitations
+* The model performs very well on abstracts of scientific papers. Please be aware that this model very domain-specific.
+* Only works in English.
+
 ## [📚 Training Dataset](https://huggingface.co/datasets/midas/inspec)
 ## 👷‍♂️ Training procedure
+The model is fine-tuned as a token classification problem where the text is labeled using the BIO scheme.
+- B => Begin of a keyphrase
+- I => Inside of a keyphrase
+- O => Ouside of a keyphrase
+
+For more information, you can take a look at the training notebook.
 
 ### Preprocessing
+```python
+def preprocess_fuction(all_samples_per_split):
+    tokenized_samples = tokenizer.batch_encode_plus(
+        all_samples_per_split[dataset_document_column],
+        padding="max_length",
+        truncation=True,
+        is_split_into_words=True,
+        max_length=max_length,
+    )
+    total_adjusted_labels = []
+    for k in range(0, len(tokenized_samples["input_ids"])):
+        prev_wid = -1
+        word_ids_list = tokenized_samples.word_ids(batch_index=k)
+        existing_label_ids = all_samples_per_split[dataset_biotags_column][k]
+        i = -1
+        adjusted_label_ids = []
+
+        for wid in word_ids_list:
+            if wid is None:
+                adjusted_label_ids.append(lbl2idx["O"])
+            elif wid != prev_wid:
+                i = i + 1
+                adjusted_label_ids.append(lbl2idx[existing_label_ids[i]])
+                prev_wid = wid
+            else:
+                adjusted_label_ids.append(
+                    lbl2idx[
+                        f"{'I' if existing_label_ids[i] == 'B' else existing_label_ids[i]}"
+                    ]
+                )
+
+        total_adjusted_labels.append(adjusted_label_ids)
+    tokenized_samples["labels"] = total_adjusted_labels
+    return tokenized_samples
+```
 ## 📝Evaluation results
 
 The model achieves the following results on the Inspec test set:
@@ -126,7 +171,8 @@ The model achieves the following results on the Inspec test set:
 |:-----------------:|:----:|:----:|:----:|:----:|:----:|:-----:|:----:|:----:|:----:|
 | Inspec Test Set   | 0.53 | 0.47 | 0.46 | 0.36 | 0.58 | 0.41  | 0.58 | 0.60 | 0.56 |
 
-### BibTeX entry and citation info
+For more information on the evaluation process, you can take a look at the keyphrase extraction evaluation notebook.
+### Bibliography
 Debanjan Mahata, Navneet Agarwal, Dibya Gautam, Amardeep Kumar, Sagar Dhiman, Anish Acharya, & Rajiv Ratn Shah. (2021). LDkp Dataset [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5501744
 
 Kulkarni, Mayank, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. "Learning Rich Representation of Keyphrases from Text." arXiv preprint arXiv:2112.08547 (2021).