larger dataset, larger GPU memory in the training phase? Is that correct?
{'login': 'debby1103', 'id': 38886373, 'node_id': 'MDQ6VXNlcjM4ODg2Mzcz', 'avatar_url': 'https://avatars.githubusercontent.com/u/38886373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/debby1103', 'html_url': 'https://github.com/debby1103', 'followers_url': 'https://api.github.com/users/debby1103/followers', 'following_url': 'https://api.github.com/users/debby1103/following{/other_user}', 'gists_url': 'https://api.github.com/users/debby1103/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/debby1103/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/debby1103/subscriptions', 'organizations_url': 'https://api.github.com/users/debby1103/orgs', 'repos_url': 'https://api.github.com/users/debby1103/repos', 'events_url': 'https://api.github.com/users/debby1103/events{/privacy}', 'received_events_url': 'https://api.github.com/users/debby1103/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
does the trainer save it in GPU? sooo curious... how to fix it It's my bad. didn't limit the input length
    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1


   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4944/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}