pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
{'login': 'LuCeHe', 'id': 9610770, 'node_id': 'MDQ6VXNlcjk2MTA3NzA=', 'avatar_url': 'https://avatars.githubusercontent.com/u/9610770?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/LuCeHe', 'html_url': 'https://github.com/LuCeHe', 'followers_url': 'https://api.github.com/users/LuCeHe/followers', 'following_url': 'https://api.github.com/users/LuCeHe/following{/other_user}', 'gists_url': 'https://api.github.com/users/LuCeHe/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/LuCeHe/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/LuCeHe/subscriptions', 'organizations_url': 'https://api.github.com/users/LuCeHe/orgs', 'repos_url': 'https://api.github.com/users/LuCeHe/repos', 'events_url': 'https://api.github.com/users/LuCeHe/events{/privacy}', 'received_events_url': 'https://api.github.com/users/LuCeHe/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
``` You saved my life.
I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1817/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}