How to train BERT model with next sentence prediction?
{'login': 'jnishi', 'id': 836541, 'node_id': 'MDQ6VXNlcjgzNjU0MQ==', 'avatar_url': 'https://avatars.githubusercontent.com/u/836541?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/jnishi', 'html_url': 'https://github.com/jnishi', 'followers_url': 'https://api.github.com/users/jnishi/followers', 'following_url': 'https://api.github.com/users/jnishi/following{/other_user}', 'gists_url': 'https://api.github.com/users/jnishi/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/jnishi/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/jnishi/subscriptions', 'organizations_url': 'https://api.github.com/users/jnishi/orgs', 'repos_url': 'https://api.github.com/users/jnishi/repos', 'events_url': 'https://api.github.com/users/jnishi/events{/privacy}', 'received_events_url': 'https://api.github.com/users/jnishi/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi !
We're not using `TextDatasetForNextSentencePrediction` in `datasets`.
Although you can probably use the `TextDatasetForNextSentencePrediction.create_examples_from_document` on a dataset to prepare it for next sentence prediction. Thanks.

Do you mean that `TextDatasetForNextSentencePrediction.create_exapmles_from_document` can be applied to dataset object other than `TextDatasetForNextSentencePrediction`  e.g. a `Dataset` object which is  loaded by  `datasets.load_dataset`? It would probably require a bit of tweaking, but you can apply it to a dataset, yes.
This should give you a new dataset with sentence pairs you can train a model on.

You can find the documentation about dataset processing here:
https://huggingface.co/docs/datasets/processing.html#processing-data-with-map Thank you for detail information.

I'll try to apply `create_examples_from_document` to `Dataset` object.

Hello.

I'm trying to pretrain the BERT model with next sentence prediction. Is there any function that supports next sentence prediction 
like ` TextDatasetForNextSentencePrediction` of `huggingface/transformers` ?

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2129/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}