 load JSON files, get the errors
{'login': 'PatricYan', 'id': 38966558, 'node_id': 'MDQ6VXNlcjM4OTY2NTU4', 'avatar_url': 'https://avatars.githubusercontent.com/u/38966558?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/PatricYan', 'html_url': 'https://github.com/PatricYan', 'followers_url': 'https://api.github.com/users/PatricYan/followers', 'following_url': 'https://api.github.com/users/PatricYan/following{/other_user}', 'gists_url': 'https://api.github.com/users/PatricYan/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/PatricYan/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/PatricYan/subscriptions', 'organizations_url': 'https://api.github.com/users/PatricYan/orgs', 'repos_url': 'https://api.github.com/users/PatricYan/repos', 'events_url': 'https://api.github.com/users/PatricYan/events{/privacy}', 'received_events_url': 'https://api.github.com/users/PatricYan/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Hi ! The message you're getting is not an error. It simply says that your JSON dataset is being prepared to a location in `/root/.cache/huggingface/datasets`"
 '> \r\n\r\nbut I want to load local JSON file by command\r\n`python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`\r\n\r\n**squad-retrain-data/train-v2.0.json** is the local JSON file, how to load it and map it to a special structure?'
 "You can load it with `dataset = datasets.load_dataset('json', data_files=args.dataset)` as you said.\r\nThen if you need to apply additional processing to map it to a special structure, you can use rename columns or use `dataset.map`. For more information, you can check the documentation here: https://huggingface.co/docs/datasets/process.html\r\n\r\nAlso feel free to share your `run.py` code so we can take a look"
 '```\r\n# Dataset selection\r\n    if args.dataset.endswith(\'.json\') or args.dataset.endswith(\'.jsonl\'):\r\n        dataset_id = None\r\n        # Load from local json/jsonl file\r\n        dataset = datasets.load_dataset(\'json\', data_files=args.dataset)\r\n        # By default, the "json" dataset loader places all examples in the train split,\r\n        # so if we want to use a jsonl file for evaluation we need to get the "train" split\r\n        # from the loaded dataset\r\n        eval_split = \'train\'\r\n    else:\r\n        default_datasets = {\'qa\': (\'squad\',), \'nli\': (\'snli\',)}\r\n        dataset_id = tuple(args.dataset.split(\':\')) if args.dataset is not None else \\\r\n            default_datasets[args.task]\r\n        # MNLI has two validation splits (one with matched domains and one with mismatched domains). Most datasets just have one "validation" split\r\n        eval_split = \'validation_matched\' if dataset_id == (\'glue\', \'mnli\') else \'validation\'\r\n        # Load the raw data\r\n        dataset = datasets.load_dataset(*dataset_id)\r\n```\r\n\r\nI want to load JSON squad dataset instead `dataset = datasets.load_dataset(\'squad\')` to retrain the model. \r\n'
 'If your JSON has the same format as the SQuAD dataset, then you need to pass `field="data"` to `load_dataset`, since the SQuAD format is one big JSON object in which the "data" field contains the list of questions and answers.\r\n```python\r\ndataset = datasets.load_dataset(\'json\', data_files=args.dataset, field="data")\r\n```\r\n\r\nLet me know if that helps :)\r\n\r\n'
 'Yes,  code works. but the format is not as expected.\r\n```\r\ndataset = datasets.load_dataset(\'json\', data_files=args.dataset, field="data")\r\n```\r\n```\r\npython3 run.py --do_train --task qa --dataset squad --output_dir ./re_trained_model/\r\n```\r\n************ train_dataset: Dataset({\r\n    features: [\'id\', \'title\', \'context\', \'question\', \'answers\'],\r\n    num_rows: 87599\r\n})\r\n\r\n\r\n```\r\npython3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/\r\n```\r\n************ train_dataset: Dataset({\r\n    features: [\'title\', \'paragraphs\'],\r\n    num_rows: 442\r\n})\r\n\r\nI want the JSON to have the same format as before features. https://github.com/huggingface/datasets/blob/master/datasets/squad_v2/squad_v2.py is the script dealing with **squad** but how can I apply it by using JSON? '
 'Ok I see, you have the paragraphs so you just need to process them to extract the questions and answers. I think you can process the SQuAD-like data this way:\r\n```python\r\ndef process_squad(articles):\r\n    out = {\r\n        "title": [],\r\n        "context": [],\r\n        "question": [],\r\n        "id": [],\r\n        "answers": [],\r\n    }\r\n    for title, paragraphs in zip(articles["title"], articles["paragraphs"]):\r\n        for paragraph in paragraphs:\r\n            for qa in paragraph["qas"]:\r\n                out["title"].append(title)\r\n                out["context"].append(paragraph["context"])\r\n                out["question"].append(qa["question"])\r\n                out["id"].append(qa["id"])\r\n                out["answers"].append({\r\n                    "answer_start": [answer["answer_start"] for answer in qa["answers"]],\r\n                    "text": [answer["text"] for answer in qa["answers"]],\r\n                })\r\n    return out\r\n\r\ndataset = dataset.map(process_squad, batched=True, remove_columns=["paragraphs"])\r\n```\r\n\r\nI adapted the code from [squad.py](https://github.com/huggingface/datasets/blob/master/datasets/squad/squad.py). The code takes as input a batch of articles (title + paragraphs) and gets all the questions and answers from the JSON structure.\r\n\r\nThe output is a dataset with `features: [\'answers\', \'context\', \'id\', \'question\', \'title\']`\r\n\r\nLet me know if that helps !\r\n'
 'Yes, this works. But how to get the training output during training the squad by **Trainer** \r\nfor example https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/trainer_qa.py \r\nI want the training inputs, labels, outputs for every epoch and step to produce the training dynamic graph'
 'I think you may need to implement your own Trainer, from the `QuestionAnsweringTrainer` for example.\r\nThis way you can have the flexibility of saving all the inputs/output used at each step'
 'does there have any function to be overwritten to do this?'
 '> does there have any function to be overwritten to do this?\r\n\r\nok, I overwrote the compute_loss, thank you.'
 'Hi,  I add one field **example_id**, but I can\'t see it in the **comput_loss** function, how can I do this? below is the information of inputs\r\n\r\n```\r\n*********************** inputs: {\'attention_mask\': tensor([[1, 1, 1,  ..., 0, 0, 0],\r\n        [1, 1, 1,  ..., 0, 0, 0],\r\n        [1, 1, 1,  ..., 0, 0, 0],\r\n        ...,\r\n        [1, 1, 1,  ..., 0, 0, 0],\r\n        [1, 1, 1,  ..., 0, 0, 0],\r\n        [1, 1, 1,  ..., 0, 0, 0]], device=\'cuda:0\'), \'end_positions\': tensor([ 25,  97,  93,  44,  25, 112, 109, 134], device=\'cuda:0\'), \'input_ids\': tensor([[ 101, 2054, 2390,  ...,    0,    0,    0],\r\n        [ 101, 2054, 2515,  ...,    0,    0,    0],\r\n        [ 101, 2054, 2106,  ...,    0,    0,    0],\r\n        ...,\r\n        [ 101, 2339, 2001,  ...,    0,    0,    0],\r\n        [ 101, 2054, 2515,  ...,    0,    0,    0],\r\n        [ 101, 2054, 2003,  ...,    0,    0,    0]], device=\'cuda:0\'), \'start_positions\': tensor([ 20,  90,  89,  41,  25,  96, 106, 132], device=\'cuda:0\'), \'token_type_ids\': tensor([[0, 0, 0,  ..., 0, 0, 0],\r\n        [0, 0, 0,  ..., 0, 0, 0],\r\n        [0, 0, 0,  ..., 0, 0, 0],\r\n        ...,\r\n        [0, 0, 0,  ..., 0, 0, 0],\r\n        [0, 0, 0,  ..., 0, 0, 0],\r\n        [0, 0, 0,  ..., 0, 0, 0]], device=\'cuda:0\')} \r\n```\r\n\r\n```\r\n# This function preprocesses a question answering dataset, tokenizing the question and context text\r\n# and finding the right offsets for the answer spans in the tokenized context (to use as labels).\r\n# Adapted from https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py\r\ndef prepare_train_dataset_qa(examples, tokenizer, max_seq_length=None):\r\n    questions = [q.lstrip() for q in examples["question"]]\r\n    max_seq_length = tokenizer.model_max_length\r\n    # tokenize both questions and the corresponding context\r\n    # if the context length is longer than max_length, we split it to several\r\n    # chunks of max_length\r\n    tokenized_examples = tokenizer(\r\n        questions,\r\n        examples["context"],\r\n        truncation="only_second",\r\n        max_length=max_seq_length,\r\n        stride=min(max_seq_length // 2, 128),\r\n        return_overflowing_tokens=True,\r\n        return_offsets_mapping=True,\r\n        padding="max_length"\r\n    )\r\n\r\n    # Since one example might give us several features if it has a long context,\r\n    # we need a map from a feature to its corresponding example.\r\n    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")\r\n    # The offset mappings will give us a map from token to character position\r\n    # in the original context. This will help us compute the start_positions\r\n    # and end_positions to get the final answer string.\r\n    offset_mapping = tokenized_examples.pop("offset_mapping")\r\n\r\n    tokenized_examples["start_positions"] = []\r\n    tokenized_examples["end_positions"] = []\r\n\r\n    tokenized_examples["example_id"] = []\r\n\r\n    for i, offsets in enumerate(offset_mapping):\r\n        input_ids = tokenized_examples["input_ids"][i]\r\n        # We will label features not containing the answer the index of the CLS token.\r\n        cls_index = input_ids.index(tokenizer.cls_token_id)\r\n        sequence_ids = tokenized_examples.sequence_ids(i)\r\n        # from the feature idx to sample idx\r\n        sample_index = sample_mapping[i]\r\n        # get the answer for a feature\r\n        answers = examples["answers"][sample_index]\r\n\r\n        tokenized_examples["example_id"].append(examples["id"][sample_index])\r\n\r\n        if len(answers["answer_start"]) == 0:\r\n            tokenized_examples["start_positions"].append(cls_index)\r\n            tokenized_examples["end_positions"].append(cls_index)\r\n        else:\r\n            # Start/end character index of the answer in the text.\r\n            start_char = answers["answer_start"][0]\r\n            end_char = start_char + len(answers["text"][0])\r\n\r\n            # Start token index of the current span in the text.\r\n            token_start_index = 0\r\n            while sequence_ids[token_start_index] != 1:\r\n                token_start_index += 1\r\n\r\n            # End token index of the current span in the text.\r\n            token_end_index = len(input_ids) - 1\r\n            while sequence_ids[token_end_index] != 1:\r\n                token_end_index -= 1\r\n\r\n            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\r\n            if not (offsets[token_start_index][0] <= start_char and\r\n                    offsets[token_end_index][1] >= end_char):\r\n                tokenized_examples["start_positions"].append(cls_index)\r\n                tokenized_examples["end_positions"].append(cls_index)\r\n            else:\r\n                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\r\n                # Note: we could go after the last offset if the answer is the last word (edge case).\r\n                while token_start_index < len(offsets) and \\\r\n                        offsets[token_start_index][0] <= start_char:\r\n                    token_start_index += 1\r\n                tokenized_examples["start_positions"].append(\r\n                    token_start_index - 1)\r\n                while offsets[token_end_index][1] >= end_char:\r\n                    token_end_index -= 1\r\n                tokenized_examples["end_positions"].append(token_end_index + 1)\r\n\r\n    return tokenized_examples\r\n```']
Hi, does this bug be fixed? when I load JSON files, I get the same errors by the command 
`!python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`

change the dateset to load json by refering to https://huggingface.co/docs/datasets/loading.html
`dataset = datasets.load_dataset('json', data_files=args.dataset)`

Errors:
`Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-c1e124ad488911b8/0.0.0/45636811569ec4a6630521c18235dfbbab83b7ab572e3393c5ba68ccabe98264...
`

_Originally posted by @yanllearnn in https://github.com/huggingface/datasets/issues/730#issuecomment-981095050_
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3333/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}