Loading methods

datasets.load_dataset

( path: str name: typing.Optional[str] = None data_dir: typing.Optional[str] = None data_files: typing.Union[str, typing.Sequence[str], typing.Mapping[str, typing.Union[str, typing.Sequence[str]]], NoneType] = None split: typing.Union[str, datasets.splits.Split, NoneType] = None cache_dir: typing.Optional[str] = None features: typing.Optional[datasets.features.features.Features] = None download_config: typing.Optional[datasets.download.download_config.DownloadConfig] = None download_mode: typing.Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None verification_mode: typing.Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None keep_in_memory: typing.Optional[bool] = None save_infos: bool = False revision: typing.Union[str, datasets.utils.version.Version, NoneType] = None token: typing.Union[bool, str, NoneType] = None streaming: bool = False num_proc: typing.Optional[int] = None storage_options: typing.Optional[typing.Dict] = None trust_remote_code: bool = None **config_kwargs ) → Dataset or DatasetDict

Parameters

path (str) — Path or name of the dataset. Depending on path, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.

For local datasets:
- if path is a local directory (containing data files only) -> load a generic dataset builder (csv, json, text etc.) based on the content of the directory e.g. './path/to/directory/with/my/csv/data'.
- if path is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory) -> load the dataset builder from the dataset script e.g. './dataset/squad' or './dataset/squad/squad.py'.
For datasets on the Hugging Face Hub (list all available datasets with huggingface_hub.list_datasets)
- if path is a dataset repository on the HF hub (containing data files only) -> load a generic dataset builder (csv, text etc.) based on the content of the repository e.g. 'username/dataset_name', a dataset repository on the HF hub containing your data files.
- if path is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory) -> load the dataset builder from the dataset script in the dataset repository e.g. glue, squad, 'username/dataset_name', a dataset repository on the HF hub containing a dataset script 'dataset_name.py'.
name (str, optional) — Defining the name of the dataset configuration.
data_dir (str, optional) — Defining the data_dir of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and data_files is None, the behavior is equal to passing os.path.join(data_dir, **) as data_files to reference all the files in a directory.
data_files (str or Sequence or Mapping, optional) — Path(s) to source data file(s).
split (Split or str) — Which split of the data to load. If None, will return a dict with all splits (typically datasets.Split.TRAIN and datasets.Split.TEST). If given, will return a single Dataset. Splits can be combined and specified like in tensorflow-datasets.
cache_dir (str, optional) — Directory to read/write data. Defaults to "~/.cache/huggingface/datasets".
features (Features, optional) — Set the features type to use for this dataset.
download_config (DownloadConfig, optional) — Specific download configuration parameters.
download_mode (DownloadMode or str, defaults to REUSE_DATASET_IF_EXISTS) — Download/generate mode.
verification_mode (VerificationMode or str, defaults to BASIC_CHECKS) — Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/…).

Added in 2.9.1
keep_in_memory (bool, defaults to None) — Whether to copy the dataset in-memory. If None, the dataset will not be copied in-memory unless explicitly enabled by setting datasets.config.IN_MEMORY_MAX_SIZE to nonzero. See more details in the improve performance section.
save_infos (bool, defaults to False) — Save the dataset information (checksums/size/splits/…).
revision (Version or str, optional) — Version of the dataset script to load. As datasets have their own git repository on the Datasets Hub, the default version “main” corresponds to their “main” branch. You can specify a different version than the default “main” by using a commit SHA or a git tag of the dataset repository.
token (str or bool, optional) — Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from "~/.huggingface".
streaming (bool, defaults to False) — If set to True, don’t download the data files. Instead, it streams the data progressively while iterating on the dataset. An IterableDataset or IterableDatasetDict is returned instead in this case.

Note that streaming works for datasets that use data formats that support being iterated over like txt, csv, jsonl for example. Json files may be downloaded completely. Also streaming from remote zip or gzip files is supported but other compressed formats like rar and xz are not yet supported. The tgz format doesn’t allow streaming.
num_proc (int, optional, defaults to None) — Number of processes when downloading and generating the dataset locally. Multiprocessing is disabled by default.

Added in 2.7.0
storage_options (dict, optional, defaults to None) — Experimental. Key/value pairs to be passed on to the dataset file-system backend, if any.

Added in 2.11.0
trust_remote_code (bool, defaults to False) — Whether or not to allow for datasets defined on the Hub using a dataset script. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.

Added in 2.16.0

Changed in 2.20.0

trust_remote_code defaults to False if not specified.
**config_kwargs (additional keyword arguments) — Keyword arguments to be passed to the BuilderConfig and used in the DatasetBuilder.

Returns

Dataset or DatasetDict

if split is not None: the dataset requested,
if split is None, a DatasetDict with each split.

or IterableDataset or IterableDatasetDict: if streaming=True

if split is not None, the dataset is requested
if split is None, a ~datasets.streaming.IterableDatasetDict with each split.

Load a dataset from the Hugging Face Hub, or a local dataset.

You can find the list of datasets on the Hub or with huggingface_hub.list_datasets.

A dataset is a directory that contains:

some data files in generic formats (JSON, CSV, Parquet, text, etc.).
and optionally a dataset script, if it requires some code to read the data files. This is used to load any kind of formats or structures.

Note that dataset scripts can also download and read data files from anywhere - in case your data files already exist online.

This function does the following under the hood:

Download and import in the library the dataset script from path if it’s not already cached inside the library.

If the dataset has no dataset script, then a generic dataset script is imported instead (JSON, CSV, Parquet, text, etc.)

Dataset scripts are small python scripts that define dataset builders. They define the citation, info and format of the dataset, contain the path or URL to the original data files and the code to load examples from the original data files.

You can find the complete list of datasets in the Datasets Hub.
Run the dataset script which will:
- Download the dataset file from the original URL (see the script) if it’s not already available locally or cached.
- Process and cache the dataset in typed Arrow tables for caching.
  
  Arrow table are arbitrarily long, typed tables which can store nested objects and be mapped to numpy/pandas/python generic types. They can be directly accessed from disk, loaded in RAM or even streamed over the web.
Return a dataset built from the requested splits in split (default: all).

It also allows to load a dataset from a local directory or a dataset repository on the Hugging Face Hub without dataset script. In this case, it automatically loads all the data files from the directory or the dataset repository.

Example:

Load a dataset from the Hugging Face Hub:

>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train')

# Map data files to splits
>>> data_files = {'train': 'train.csv', 'test': 'test.csv'}
>>> ds = load_dataset('namespace/your_dataset_name', data_files=data_files)

Load a local dataset:

# Load a CSV file
>>> from datasets import load_dataset
>>> ds = load_dataset('csv', data_files='path/to/local/my_dataset.csv')

# Load a JSON file
>>> from datasets import load_dataset
>>> ds = load_dataset('json', data_files='path/to/local/my_dataset.json')

# Load from a local loading script
>>> from datasets import load_dataset
>>> ds = load_dataset('path/to/local/loading_script/loading_script.py', split='train')

Load an IterableDataset:

>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train', streaming=True)

Load an image dataset with the ImageFolder dataset builder:

>>> from datasets import load_dataset
>>> ds = load_dataset('imagefolder', data_dir='/path/to/images', split='train')

Datasets

Loading methods

Datasets

datasets.load_dataset

datasets.load_from_disk

datasets.load_dataset_builder

datasets.get_dataset_config_names

datasets.get_dataset_infos

datasets.get_dataset_split_names

From files

Text

class datasets.packaged_modules.text.TextConfig

class datasets.packaged_modules.text.Text

CSV

class datasets.packaged_modules.csv.CsvConfig

class datasets.packaged_modules.csv.Csv

JSON

class datasets.packaged_modules.json.JsonConfig

class datasets.packaged_modules.json.Json

XML

class datasets.packaged_modules.xml.XmlConfig

class datasets.packaged_modules.xml.Xml

Parquet

class datasets.packaged_modules.parquet.ParquetConfig

class datasets.packaged_modules.parquet.Parquet

Arrow

class datasets.packaged_modules.arrow.ArrowConfig

class datasets.packaged_modules.arrow.Arrow

SQL

class datasets.packaged_modules.sql.SqlConfig

class datasets.packaged_modules.sql.Sql

Images

class datasets.packaged_modules.imagefolder.ImageFolderConfig

class datasets.packaged_modules.imagefolder.ImageFolder

Audio

class datasets.packaged_modules.audiofolder.AudioFolderConfig

class datasets.packaged_modules.audiofolder.AudioFolder

Videos

class datasets.packaged_modules.videofolder.VideoFolderConfig

class datasets.packaged_modules.videofolder.VideoFolder

WebDataset

class datasets.packaged_modules.webdataset.WebDataset