Help understanding how to build a dataset for language modeling as with the old TextDataset
{'login': 'danieldiezmallo', 'id': 46021411, 'node_id': 'MDQ6VXNlcjQ2MDIxNDEx', 'avatar_url': 'https://avatars.githubusercontent.com/u/46021411?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/danieldiezmallo', 'html_url': 'https://github.com/danieldiezmallo', 'followers_url': 'https://api.github.com/users/danieldiezmallo/followers', 'following_url': 'https://api.github.com/users/danieldiezmallo/following{/other_user}', 'gists_url': 'https://api.github.com/users/danieldiezmallo/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/danieldiezmallo/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/danieldiezmallo/subscriptions', 'organizations_url': 'https://api.github.com/users/danieldiezmallo/orgs', 'repos_url': 'https://api.github.com/users/danieldiezmallo/repos', 'events_url': 'https://api.github.com/users/danieldiezmallo/events{/privacy}', 'received_events_url': 'https://api.github.com/users/danieldiezmallo/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['\r\nI received an answer for this question on the HuggingFace Datasets forum by @lhoestq\r\n\r\nHi !\r\n\r\nIf you want to tokenize line by line, you can use this:\r\n\r\n```\r\nmax_seq_length = 512\r\nnum_proc = 4\r\n\r\ndef tokenize_function(examples):\r\n# Remove empty lines\r\nexamples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]\r\nreturn tokenizer(\r\n    examples["text"],\r\n    truncation=True,\r\n    max_length=max_seq_length,\r\n)\r\n\r\ntokenized_dataset = dataset.map(\r\ntokenize_function,\r\nbatched=True,\r\nnum_proc=num_proc,\r\nremove_columns=["text"],\r\n)\r\n```\r\n\r\nThough the TextDataset was doing a different processing by concatenating all the texts and building blocks of size 512. If you need this behavior, then you must apply an additional map function after the tokenization:\r\n\r\n```\r\n# Main data processing function that will concatenate all texts from\r\n# our dataset and generate chunks of max_seq_length.\r\ndef group_texts(examples):\r\n# Concatenate all texts.\r\nconcatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\r\ntotal_length = len(concatenated_examples[list(examples.keys())[0]])\r\n# We drop the small remainder, we could add padding if the model supported it instead of this drop,\r\n# you can customize this part to your needs.\r\ntotal_length = (total_length // max_seq_length) * max_seq_length\r\n# Split by chunks of max_len.\r\nresult = {\r\n    k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]\r\n    for k, t in concatenated_examples.items()\r\n}\r\nreturn result\r\n\r\n# Note that with `batched=True`, this map processes 1,000 texts together,\r\n# so group_texts throws away a remainder for each of those groups of 1,000 texts.\r\n# You can adjust that batch_size here but a higher value might be slower to preprocess.\r\n\r\ntokenized_dataset = tokenized_dataset.map(\r\ngroup_texts,\r\nbatched=True,\r\nnum_proc=num_proc,\r\n)\r\n```\r\n\r\nThis code comes from the processing of the run_mlm.py example script of transformers\r\n\r\n'
 'Resolved']
Hello,

I am trying to load a custom dataset that I will then use for language modeling. The dataset consists of a text file that has a whole document in each line, meaning that each line overpasses the normal 512 tokens limit of most tokenizers.

I would like to understand what is the process to build a text dataset that tokenizes each line, having previously split the documents in the dataset into lines of a "tokenizable" size, as the old TextDataset class would do, where you only had to do the following, and a tokenized dataset without text loss would be available to pass to a DataCollator:

```
model_checkpoint = 'distilbert-base-uncased'

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="path/to/text_file.txt",
    block_size=512,
)
```

For now, what I have is the following, which, of course, throws an error because each line is longer than the maximum block size in the tokenizer:

```
import datasets
dataset = datasets.load_dataset('path/to/text_file.txt')

model_checkpoint = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def tokenize_function(examples):
    return tokenizer(examples["text"])

tokenized_datasets = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])

tokenized_datasets
```

So what would be the "standard" way of creating a dataset in the way it was done before?

Thank you very much for the help :))
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2285/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}