Update README.md

README.md

@@ -1,3 +1,129 @@
 ---
+language:
+- sv
+- 'no'
+- da
+- en
 license: mit
+tags:
+- bert
+- roberta
+pipeline_tag: fill-mask
+widget:
+- text: Huvudstaden i Sverige är <mask>.
+  example_title: Swedish
+- text: Hovedstaden i Norge er <mask>.
+  example_title: Norwegian
+- text: Danmarks hovedstad er <mask>.
+  example_title: Danish
 ---
+
+# roberta-large-1160k
+
+## Intended uses
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='AI-Sweden-Models/roberta-large-550k')
+>>> unmasker("Huvudstaden i Sverige är <mask>.")
+[{'score': 0.5841221213340759,
+  'token': 1945,
+  'token_str': ' Stockholm',
+  'sequence': 'Huvudstaden i Sverige är Stockholm.'},
+ {'score': 0.06775698810815811,
+  'token': 5007,
+  'token_str': ' Göteborg',
+  'sequence': 'Huvudstaden i Sverige är Göteborg.'},
+ {'score': 0.05057400465011597,
+  'token': 5761,
+  'token_str': ' Malmö',
+  'sequence': 'Huvudstaden i Sverige är Malmö.'},
+ {'score': 0.021936343982815742,
+  'token': 21449,
+  'token_str': ' Norrköping',
+  'sequence': 'Huvudstaden i Sverige är Norrköping.'},
+ {'score': 0.017798304557800293,
+  'token': 5658,
+  'token_str': ' Uppsala',
+  'sequence': 'Huvudstaden i Sverige är Uppsala.'}]
+```
+```python
+>>> unmasker("Hovedstaden i Norge er <mask>.")
+[{'score': 0.6792309284210205,
+  'token': 5158,
+  'token_str': ' Oslo',
+  'sequence': 'Hovedstaden i Norge er Oslo.'},
+ {'score': 0.09379775077104568,
+  'token': 15456,
+  'token_str': ' Trondheim',
+  'sequence': 'Hovedstaden i Norge er Trondheim.'},
+ {'score': 0.052535850554704666,
+  'token': 11370,
+  'token_str': ' Bergen',
+  'sequence': 'Hovedstaden i Norge er Bergen.'},
+ {'score': 0.03465486690402031,
+  'token': 29407,
+  'token_str': ' hovedstaden',
+  'sequence': 'Hovedstaden i Norge er hovedstaden.'},
+ {'score': 0.03017985075712204,
+  'token': 33311,
+  'token_str': ' Kristiansand',
+  'sequence': 'Hovedstaden i Norge er Kristiansand.'}]
+```
+```python
+>>> unmasker("Danmarks hovedstad er <mask>.")
+[{'score': 0.11624140292406082,
+  'token': 4794,
+  'token_str': ' København',
+  'sequence': 'Danmarks hovedstad er København.'},
+ {'score': 0.045051511377096176,
+  'token': 7680,
+  'token_str': ' død',
+  'sequence': 'Danmarks hovedstad er død.'},
+ {'score': 0.02936543896794319,
+  'token': 10795,
+  'token_str': ' lukket',
+  'sequence': 'Danmarks hovedstad er lukket.'},
+ {'score': 0.026030730456113815,
+  'token': 13580,
+  'token_str': ' Odense',
+  'sequence': 'Danmarks hovedstad er Odense.'},
+ {'score': 0.02130937948822975,
+  'token': 16347,
+  'token_str': ' Roskilde',
+  'sequence': 'Danmarks hovedstad er Roskilde.'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import RobertaTokenizer, RobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('AI-Sweden-Models/roberta-large-550k')
+model = RobertaModel.from_pretrained('AI-Sweden-Models/roberta-large-550k')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+## Training data
+The Scandinavian subset of the Nordic Pile (Swedish, Norwegian, Danish), consisting of 414 962 688 text samples.
+
+## Training procedure
+
+The model was trained with the [optimum-habana](https://github.com/huggingface/optimum-habana) framework. Utilizing 8X Intel® Gaudi® 2 AI accelerators, managed by Intel Sweden AB.
+
+The weights from https://huggingface.co/FacebookAI/roberta-large are used as initialization, and the tokenizer is trained from scratch.
+
+This model is a checkpoint (1 160 000 / 1 350 790). The final run is 5 epochs.
+
+A batch size of 1536 was used.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results: