 add one field  "example_id", but I can't see it in the "comput_loss" function
{'login': 'PatricYan', 'id': 38966558, 'node_id': 'MDQ6VXNlcjM4OTY2NTU4', 'avatar_url': 'https://avatars.githubusercontent.com/u/38966558?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/PatricYan', 'html_url': 'https://github.com/PatricYan', 'followers_url': 'https://api.github.com/users/PatricYan/followers', 'following_url': 'https://api.github.com/users/PatricYan/following{/other_user}', 'gists_url': 'https://api.github.com/users/PatricYan/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/PatricYan/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/PatricYan/subscriptions', 'organizations_url': 'https://api.github.com/users/PatricYan/orgs', 'repos_url': 'https://api.github.com/users/PatricYan/repos', 'events_url': 'https://api.github.com/users/PatricYan/events{/privacy}', 'received_events_url': 'https://api.github.com/users/PatricYan/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Hi ! Your function looks fine, I used to map `squad` locally and it indeed added the `example_id` field correctly.\r\n\r\nHowever I think that in the `compute_loss` method only a subset of the fields are available: the model inputs. Since `example_id` is not a model input (it's not passed as a parameter to the model), the data loader doesn't need to return it by default.\r\n\r\nHowever you can disable this behavior by setting `remove_unused_columns` to `False` to your training arguments. In this case in `compute_loss` you will get the full item with all the fields.\r\n\r\nNote that since the model doesn't take `example_id` as input, you will have to remove it from the inputs when `model(**inputs)` is called"
 'Hi, I have set **args.remove_unused_columns=False** and  **training_args.remove_unused_columns=False**, but the field doesn\'t been contained yet.\r\n```\r\ndef main():\r\n    argp = HfArgumentParser(TrainingArguments)\r\n    # The HfArgumentParser object collects command-line arguments into an object (and provides default values for unspecified arguments).\r\n    # In particular, TrainingArguments has several keys that you\'ll need/want to specify (when you call run.py from the command line):\r\n    # --do_train\r\n    #     When included, this argument tells the script to train a model.\r\n    #     See docstrings for "--task" and "--dataset" for how the training dataset is selected.\r\n    # --do_eval\r\n    #     When included, this argument tells the script to evaluate the trained/loaded model on the validation split of the selected dataset.\r\n    # --per_device_train_batch_size <int, default=8>\r\n    #     This is the training batch size.\r\n    #     If you\'re running on GPU, you should try to make this as large as you can without getting CUDA out-of-memory errors.\r\n    #     For reference, with --max_length=128 and the default ELECTRA-small model, a batch size of 32 should fit in 4gb of GPU memory.\r\n    # --num_train_epochs <float, default=3.0>\r\n    #     How many passes to do through the training data.\r\n    # --output_dir <path>\r\n    #     Where to put the trained model checkpoint(s) and any eval predictions.\r\n    #     *This argument is required*.\r\n\r\n    argp.add_argument(\'--model\', type=str,\r\n                      default=\'google/electra-small-discriminator\',\r\n                      help="""This argument specifies the base model to fine-tune.\r\n        This should either be a HuggingFace model ID (see https://huggingface.co/models)\r\n        or a path to a saved model checkpoint (a folder containing config.json and pytorch_model.bin).""")\r\n    argp.add_argument(\'--task\', type=str, choices=[\'nli\', \'qa\'], required=True,\r\n                      help="""This argument specifies which task to train/evaluate on.\r\n        Pass "nli" for natural language inference or "qa" for question answering.\r\n        By default, "nli" will use the SNLI dataset, and "qa" will use the SQuAD dataset.""")\r\n    argp.add_argument(\'--dataset\', type=str, default=None,\r\n                      help="""This argument overrides the default dataset used for the specified task.""")\r\n    argp.add_argument(\'--max_length\', type=int, default=128,\r\n                      help="""This argument limits the maximum sequence length used during training/evaluation.\r\n        Shorter sequence lengths need less memory and computation time, but some examples may end up getting truncated.""")\r\n    argp.add_argument(\'--max_train_samples\', type=int, default=None,\r\n                      help=\'Limit the number of examples to train on.\')\r\n    argp.add_argument(\'--max_eval_samples\', type=int, default=None,\r\n                      help=\'Limit the number of examples to evaluate on.\')\r\n\r\n    argp.remove_unused_columns = False\r\n    training_args, args = argp.parse_args_into_dataclasses()\r\n    args.remove_unused_columns=False\r\n    training_args.remove_unused_columns=False\r\n```\r\n\r\n\r\n```\r\n**************** train_dataset: Dataset({\r\n    features: [\'id\', \'title\', \'context\', \'question\', \'answers\'],\r\n    num_rows: 87599\r\n})\r\n\r\n\r\n**************** train_dataset_featurized: Dataset({\r\n    features: [\'attention_mask\', \'end_positions\', \'input_ids\', \'start_positions\', \'token_type_ids\'],\r\n    num_rows: 87714\r\n})\r\n```'
 "Hi, I print the value, all are set to False, but don't work.\r\n```\r\n********************* training_args: TrainingArguments(\r\n_n_gpu=1,\r\nadafactor=False,\r\nadam_beta1=0.9,\r\nadam_beta2=0.999,\r\nadam_epsilon=1e-08,\r\ndataloader_drop_last=False,\r\ndataloader_num_workers=0,\r\ndataloader_pin_memory=True,\r\nddp_find_unused_parameters=None,\r\ndebug=[],\r\ndeepspeed=None,\r\ndisable_tqdm=False,\r\ndo_eval=False,\r\ndo_predict=False,\r\ndo_train=True,\r\neval_accumulation_steps=None,\r\neval_steps=None,\r\nevaluation_strategy=IntervalStrategy.NO,\r\nfp16=False,\r\nfp16_backend=auto,\r\nfp16_full_eval=False,\r\nfp16_opt_level=O1,\r\ngradient_accumulation_steps=1,\r\ngreater_is_better=None,\r\ngroup_by_length=False,\r\nignore_data_skip=False,\r\nlabel_names=None,\r\nlabel_smoothing_factor=0.0,\r\nlearning_rate=5e-05,\r\nlength_column_name=length,\r\nload_best_model_at_end=False,\r\nlocal_rank=-1,\r\nlog_level=-1,\r\nlog_level_replica=-1,\r\nlog_on_each_node=True,\r\nlogging_dir=./re_trained_model/runs/Dec01_14-15-08_399b9290604c,\r\nlogging_first_step=False,\r\nlogging_steps=500,\r\nlogging_strategy=IntervalStrategy.STEPS,\r\nlr_scheduler_type=SchedulerType.LINEAR,\r\nmax_grad_norm=1.0,\r\nmax_steps=-1,\r\nmetric_for_best_model=None,\r\nmp_parameters=,\r\nno_cuda=False,\r\nnum_train_epochs=3.0,\r\noutput_dir=./re_trained_model,\r\noverwrite_output_dir=False,\r\npast_index=-1,\r\nper_device_eval_batch_size=8,\r\nper_device_train_batch_size=8,\r\nprediction_loss_only=False,\r\npush_to_hub=False,\r\npush_to_hub_model_id=re_trained_model,\r\npush_to_hub_organization=None,\r\npush_to_hub_token=None,\r\nremove_unused_columns=False,\r\nreport_to=['tensorboard'],\r\nresume_from_checkpoint=None,\r\nrun_name=./re_trained_model,\r\nsave_on_each_node=False,\r\nsave_steps=500,\r\nsave_strategy=IntervalStrategy.STEPS,\r\nsave_total_limit=None,\r\nseed=42,\r\nsharded_ddp=[],\r\nskip_memory_metrics=True,\r\ntpu_metrics_debug=False,\r\ntpu_num_cores=None,\r\nuse_legacy_prediction_loop=False,\r\nwarmup_ratio=0.0,\r\nwarmup_steps=0,\r\nweight_decay=0.0,\r\n)\r\n```\r\n```\r\n********************* args: Namespace(dataset='squad', max_eval_samples=None, max_length=128, max_train_samples=None, model='google/electra-small-discriminator', remove_unused_columns=False, task='qa')\r\n2021-12-01 14:15:10,048 - WARNING - datasets.builder - Reusing dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)\r\nSome weights of the model checkpoint at google/electra-small-discriminator were not used when initializing ElectraForQuestionAnswering: ['discriminator_predictions.dense_prediction.weight', 'discriminator_predictions.dense_prediction.bias', 'discriminator_predictions.dense.weight', 'discriminator_predictions.dense.bias']\r\n- This IS expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\r\n- This IS NOT expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\r\nSome weights of ElectraForQuestionAnswering were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\r\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\r\nPreprocessing data... (this takes a little bit, should only happen once per dataset)\r\n```"
 "Hmmm, it might be because the default data collator removes all the fields with `string` type:\r\n\r\nhttps://github.com/huggingface/transformers/blob/4c0dd199c8305903564c2edeae23d294edd4b321/src/transformers/data/data_collator.py#L107-L112\r\n\r\nI guess you also need a custom data collator that doesn't remove them."
 'can you give a tutorial about how to do this?'
 'I overwrite **get_train_dataloader**, and remove **_remove_unused_columns**, but it doesn\'t work.\r\n\r\n```\r\n    def get_train_dataloader(self) -> DataLoader:\r\n        """\r\n        Returns the training :class:`~torch.utils.data.DataLoader`.\r\n\r\n        Will use no sampler if :obj:`self.train_dataset` does not implement :obj:`__len__`, a random sampler (adapted\r\n        to distributed training if necessary) otherwise.\r\n\r\n        Subclass and override this method if you want to inject some custom behavior.\r\n        """\r\n        if self.train_dataset is None:\r\n            raise ValueError("Trainer: training requires a train_dataset.")\r\n\r\n        train_dataset = self.train_dataset\r\n        # if is_datasets_available() and isinstance(train_dataset, datasets.Dataset):\r\n        #     train_dataset = self._remove_unused_columns(train_dataset, description="training")\r\n\r\n        if isinstance(train_dataset, torch.utils.data.IterableDataset):\r\n            if self.args.world_size > 1:\r\n                train_dataset = IterableDatasetShard(\r\n                    train_dataset,\r\n                    batch_size=self.args.train_batch_size,\r\n                    drop_last=self.args.dataloader_drop_last,\r\n                    num_processes=self.args.world_size,\r\n                    process_index=self.args.process_index,\r\n                )\r\n\r\n            return DataLoader(\r\n                train_dataset,\r\n                batch_size=self.args.train_batch_size,\r\n                collate_fn=self.data_collator,\r\n                num_workers=self.args.dataloader_num_workers,\r\n                pin_memory=self.args.dataloader_pin_memory,\r\n            )\r\n\r\n        train_sampler = self._get_train_sampler()\r\n\r\n        return DataLoader(\r\n            train_dataset,\r\n            batch_size=self.args.train_batch_size,\r\n            sampler=train_sampler,\r\n            collate_fn=self.data_collator,\r\n            drop_last=self.args.dataloader_drop_last,\r\n            num_workers=self.args.dataloader_num_workers,\r\n            pin_memory=self.args.dataloader_pin_memory,\r\n        )\r\n```'
 'Hi, it works now, thank you.\r\n1. **args.remove_unused_columns=False** and **training_args.remove_unused_columns=False**\r\n2. overwrite **get_train_dataloader**, and remove **_remove_unused_columns**\r\n3. add new fields, and can be got in **inputs**. ']
Hi,  I add one field **example_id**, but I can't see it in the **comput_loss** function, how can I do this? below is the information of inputs

```
*********************** inputs: {'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0'), 'end_positions': tensor([ 25,  97,  93,  44,  25, 112, 109, 134], device='cuda:0'), 'input_ids': tensor([[ 101, 2054, 2390,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2106,  ...,    0,    0,    0],
        ...,
        [ 101, 2339, 2001,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2003,  ...,    0,    0,    0]], device='cuda:0'), 'start_positions': tensor([ 20,  90,  89,  41,  25,  96, 106, 132], device='cuda:0'), 'token_type_ids': tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0]], device='cuda:0')} 
```

```
# This function preprocesses a question answering dataset, tokenizing the question and context text
# and finding the right offsets for the answer spans in the tokenized context (to use as labels).
# Adapted from https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py
def prepare_train_dataset_qa(examples, tokenizer, max_seq_length=None):
    questions = [q.lstrip() for q in examples["question"]]
    max_seq_length = tokenizer.model_max_length
    # tokenize both questions and the corresponding context
    # if the context length is longer than max_length, we split it to several
    # chunks of max_length
    tokenized_examples = tokenizer(
        questions,
        examples["context"],
        truncation="only_second",
        max_length=max_seq_length,
        stride=min(max_seq_length // 2, 128),
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length"
    )

    # Since one example might give us several features if it has a long context,
    # we need a map from a feature to its corresponding example.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
    # The offset mappings will give us a map from token to character position
    # in the original context. This will help us compute the start_positions
    # and end_positions to get the final answer string.
    offset_mapping = tokenized_examples.pop("offset_mapping")

    tokenized_examples["start_positions"] = []
    tokenized_examples["end_positions"] = []

    tokenized_examples["example_id"] = []

    for i, offsets in enumerate(offset_mapping):
        input_ids = tokenized_examples["input_ids"][i]
        # We will label features not containing the answer the index of the CLS token.
        cls_index = input_ids.index(tokenizer.cls_token_id)
        sequence_ids = tokenized_examples.sequence_ids(i)
        # from the feature idx to sample idx
        sample_index = sample_mapping[i]
        # get the answer for a feature
        answers = examples["answers"][sample_index]

        tokenized_examples["example_id"].append(examples["id"][sample_index])

        if len(answers["answer_start"]) == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers["answer_start"][0]
            end_char = start_char + len(answers["text"][0])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != 1:
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != 1:
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and
                    offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and \
                        offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(
                    token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)

    return tokenized_examples
```

_Originally posted by @yanllearnn in https://github.com/huggingface/datasets/issues/3333#issuecomment-983457161_
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3353/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}