Dataset file size on disk is very large with 3D Array
Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D
Hi, 

I have created my own dataset using the provided dataset loading script. It is an image dataset where images are stored as 3D Array with dtype=uint8. 

The actual size on disk is surprisingly large. It takes 520 MB. Here is some info from `dataset_info.json`. 

`{
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "shape": [224, 224, 3],
            "dtype": "uint8",
            "id": null,
            "_type": "Array3D",
        }
    },
    "post_processed": null,
    "supervised_keys": null,
    "builder_name": "shot_type_image_dataset",
    "config_name": "default",
    "version": {
        "version_str": "0.0.0",
        "description": null,
        "major": 0,
        "minor": 0,
        "patch": 0,
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 520803408,
            "num_examples": 1479,
            "dataset_name": "shot_type_image_dataset",
        }
    },
    "download_checksums": {
        "": {
            "num_bytes": 16940447118,
            "checksum": "5854035705efe08b0ed8f3cf3da7b4d29cba9055c2d2d702c79785350d72ee03",
        }
    },
    "download_size": 16940447118,
    "post_processing_size": null,
    "dataset_size": 520803408,
    "size_in_bytes": 17461250526,
}`

I have created the same dataset with tensorflow_dataset and it takes only 125MB on disk.

I am wondering, is it normal behavior ? I understand `Datasets` uses Arrow for serialization wheres tf uses TF Records.

This might be a problem for large dataset. 

Thanks for your help. 

Dataset file size on disk is very large with 3D Array 
 Hi, 

I have created my own dataset using the provided dataset loading script. It is an image dataset where images are stored as 3D Array with dtype=uint8. 

The actual size on disk is surprisingly large. It takes 520 MB. Here is some info from `dataset_info.json`. 

`{
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "shape": [224, 224, 3],
            "dtype": "uint8",
            "id": null,
            "_type": "Array3D",
        }
    },
    "post_processed": null,
    "supervised_keys": null,
    "builder_name": "shot_type_image_dataset",
    "config_name": "default",
    "version": {
        "version_str": "0.0.0",
        "description": null,
        "major": 0,
        "minor": 0,
        "patch": 0,
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 520803408,
            "num_examples": 1479,
            "dataset_name": "shot_type_image_dataset",
        }
    },
    "download_checksums": {
        "": {
            "num_bytes": 16940447118,
            "checksum": "5854035705efe08b0ed8f3cf3da7b4d29cba9055c2d2d702c79785350d72ee03",
        }
    },
    "download_size": 16940447118,
    "post_processing_size": null,
    "dataset_size": 520803408,
    "size_in_bytes": 17461250526,
}`

I have created the same dataset with tensorflow_dataset and it takes only 125MB on disk.

I am wondering, is it normal behavior ? I understand `Datasets` uses Arrow for serialization wheres tf uses TF Records.

This might be a problem for large dataset. 

Thanks for your help. 
 
 Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D