Why is dataset after tokenization far more larger than the orginal one ?
{'login': 'richarddwang', 'id': 17963619, 'node_id': 'MDQ6VXNlcjE3OTYzNjE5', 'avatar_url': 'https://avatars.githubusercontent.com/u/17963619?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/richarddwang', 'html_url': 'https://github.com/richarddwang', 'followers_url': 'https://api.github.com/users/richarddwang/followers', 'following_url': 'https://api.github.com/users/richarddwang/following{/other_user}', 'gists_url': 'https://api.github.com/users/richarddwang/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/richarddwang/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/richarddwang/subscriptions', 'organizations_url': 'https://api.github.com/users/richarddwang/orgs', 'repos_url': 'https://api.github.com/users/richarddwang/repos', 'events_url': 'https://api.github.com/users/richarddwang/events{/privacy}', 'received_events_url': 'https://api.github.com/users/richarddwang/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map` Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔 Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings. Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/258/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}