## Model description
This model is based on [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://arxiv.org/abs/2210.05529). Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. arXiv:2210.05529 (Preprint).

Initial weights were taken from [google/bert_uncased_L-8_H-256_A-4](https://huggingface.co/google/bert_uncased_L-8_H-256_A-4).
Model was additionally pretrained for 20_000 steps on 5m lines of text from english version of [OpenSubtitles](http://www.opensubtitles.org/) dataset. 

Maximum input length is 512 tokens that is enoungh to encode dialog with few previous utterances (average sentence length per utterance in SWDA, MAPTASK, MRDA, BT_OASIS, FRAMES, AMI, DSTC3 is less than 11 tokens).
{"language": "en", "license": "cc-by-nc-sa-4.0", "library_name": "transformers", "datasets": ["OpenSubtitles"], "pipeline_tag": "fill-mask"}
transformers pytorch tensorboard hierarchical-transformer fill-mask custom_code en dataset:OpenSubtitles arxiv:2210.05529 license:cc-by-nc-sa-4.0 autotrain_compatible region:us
TAGS
#transformers #pytorch #tensorboard #hierarchical-transformer #fill-mask #custom_code #en #dataset-OpenSubtitles #arxiv-2210.05529 #license-cc-by-nc-sa-4.0 #autotrain_compatible #region-us 

## Model description
This model is based on An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. arXiv:2210.05529 (Preprint).

Initial weights were taken from google/bert_uncased_L-8_H-256_A-4.
Model was additionally pretrained for 20_000 steps on 5m lines of text from english version of OpenSubtitles dataset. 

Maximum input length is 512 tokens that is enoungh to encode dialog with few previous utterances (average sentence length per utterance in SWDA, MAPTASK, MRDA, BT_OASIS, FRAMES, AMI, DSTC3 is less than 11 tokens).
## Model description
This model is based on An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. arXiv:2210.05529 (Preprint).

Initial weights were taken from google/bert_uncased_L-8_H-256_A-4.
Model was additionally pretrained for 20_000 steps on 5m lines of text from english version of OpenSubtitles dataset. 

Maximum input length is 512 tokens that is enoungh to encode dialog with few previous utterances (average sentence length per utterance in SWDA, MAPTASK, MRDA, BT_OASIS, FRAMES, AMI, DSTC3 is less than 11 tokens).
TAGS
#transformers #pytorch #tensorboard #hierarchical-transformer #fill-mask #custom_code #en #dataset-OpenSubtitles #arxiv-2210.05529 #license-cc-by-nc-sa-4.0 #autotrain_compatible #region-us 
 ## Model description
This model is based on An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. arXiv:2210.05529 (Preprint).

Initial weights were taken from google/bert_uncased_L-8_H-256_A-4.
Model was additionally pretrained for 20_000 steps on 5m lines of text from english version of OpenSubtitles dataset. 

Maximum input length is 512 tokens that is enoungh to encode dialog with few previous utterances (average sentence length per utterance in SWDA, MAPTASK, MRDA, BT_OASIS, FRAMES, AMI, DSTC3 is less than 11 tokens).