<commit_before># Adding a dataset

Follow this guide to add a dataset to TFDS.

See our [list of datasets](datasets.md) to see if the dataset you want isn't
already added.

* [Overview](#overview)
* [Writing `my_dataset.py`](#writing-my-datasetpy)
* [Specifying `DatasetInfo`](#specifying-datasetinfo)
  * [`FeatureConnector`s](#featureconnectors)
* [Downloading and extracting source data](#downloading-and-extracting-source-data)
  * [Manual download and extraction](#manual-download-and-extraction)
* [Specifying dataset splits](#specifying-dataset-splits)
* [Writing an example generator](#writing-an-example-generator)
  * [File access and `tf.io.gfile`](#file-access-and-tfiogfile)
  * [Extra dependencies](#extra-dependencies)
* [Dataset configuration](#dataset-configuration)
* [Create your own `FeatureConnector`](#create-your-own-featureconnector)
* [Adding the dataset to `tensorflow/datasets`](#adding-the-dataset-to-tensorflowdatasets)
* [Large datasets and distributed generation](#large-datasets-and-distributed-generation)
* [Testing `MyDataset`](#testing-mydataset)

## Overview

Datasets are distributed in all kinds of formats and in all kinds of places,
and they're not always stored in a format that's ready to feed into a machine
learning pipeline. Enter TFDS.

TFDS provides a way to transform all those datasets into a standard format,
do the preprocessing necessary to make them ready for a machine learning
pipeline, and provides a standard input pipeline using `tf.data`.

To enable this, each dataset implements a subclass of `DatasetBuilder`, which
specifies:

* Where the data is coming from (i.e. its URL);
* What the dataset looks like (i.e. its features);
* How the data should be split (e.g. `TRAIN` and `TEST`);
* and the individual records in the dataset.

The first time a dataset is used, the dataset is downloaded, prepared, and
written to disk in a standard format. Subsequent access will read from those
pre-processed files directly.

**Note**: Currently we do not support datasets that take longer than 1 day to
generate on a single machine. See the
[section below on large datasets](#large-datasets-and-distributed-generation).

## Writing `my_dataset.py`

### Use the default template

To help you get started to add a new dataset inside `tfds`, you can cloned the
repository and use the default template by running the following command:

```
python tensorflow_datasets/scripts/create_new_dataset.py \
  --name my_dataset \
  --type image  # text, audio, translation,...
```


It will create the minimum python files that you're requiring to get started.

### `DatasetBuilder`

Each dataset is defined as a subclass of
[`tfds.core.DatasetBuilder`](api_docs/python/tfds/core/DatasetBuilder.md)
implementing the following methods:

* `_info`: builds the
  [`DatasetInfo`](api_docs/python/tfds/core/DatasetInfo.md) object
  describing the dataset
* `_download_and_prepare`: to download and serialize the source data to disk
* `_as_dataset`: to produce a `tf.data.Dataset` from the serialized data<commit_msg>Updated dataset name flag<commit_after># Adding a dataset

Follow this guide to add a dataset to TFDS.

See our [list of datasets](datasets.md) to see if the dataset you want isn't
already added.

* [Overview](#overview)
* [Writing `my_dataset.py`](#writing-my-datasetpy)
* [Specifying `DatasetInfo`](#specifying-datasetinfo)
  * [`FeatureConnector`s](#featureconnectors)
* [Downloading and extracting source data](#downloading-and-extracting-source-data)
  * [Manual download and extraction](#manual-download-and-extraction)
* [Specifying dataset splits](#specifying-dataset-splits)
* [Writing an example generator](#writing-an-example-generator)
  * [File access and `tf.io.gfile`](#file-access-and-tfiogfile)
  * [Extra dependencies](#extra-dependencies)
* [Dataset configuration](#dataset-configuration)
* [Create your own `FeatureConnector`](#create-your-own-featureconnector)
* [Adding the dataset to `tensorflow/datasets`](#adding-the-dataset-to-tensorflowdatasets)
* [Large datasets and distributed generation](#large-datasets-and-distributed-generation)
* [Testing `MyDataset`](#testing-mydataset)

## Overview

Datasets are distributed in all kinds of formats and in all kinds of places,
and they're not always stored in a format that's ready to feed into a machine
learning pipeline. Enter TFDS.

TFDS provides a way to transform all those datasets into a standard format,
do the preprocessing necessary to make them ready for a machine learning
pipeline, and provides a standard input pipeline using `tf.data`.

To enable this, each dataset implements a subclass of `DatasetBuilder`, which
specifies:

* Where the data is coming from (i.e. its URL);
* What the dataset looks like (i.e. its features);
* How the data should be split (e.g. `TRAIN` and `TEST`);
* and the individual records in the dataset.

The first time a dataset is used, the dataset is downloaded, prepared, and
written to disk in a standard format. Subsequent access will read from those
pre-processed files directly.

**Note**: Currently we do not support datasets that take longer than 1 day to
generate on a single machine. See the
[section below on large datasets](#large-datasets-and-distributed-generation).

## Writing `my_dataset.py`

### Use the default template

To help you get started to add a new dataset inside `tfds`, you can cloned the
repository and use the default template by running the following command:

```
python tensorflow_datasets/scripts/create_new_dataset.py \
  --dataset my_dataset \
  --type image  # text, audio, translation,...
```


It will create the minimum python files that you're requiring to get started.

### `DatasetBuilder`

Each dataset is defined as a subclass of
[`tfds.core.DatasetBuilder`](api_docs/python/tfds/core/DatasetBuilder.md)
implementing the following methods:

* `_info`: builds the
  [`DatasetInfo`](api_docs/python/tfds/core/DatasetInfo.md) object
  describing the dataset
* `_download_and_prepare`: to download and serialize the source data to disk
* `_as_dataset`: to produce a `tf.data.Dataset` from the serialized data
{'fraction_non_alphanumeric': 0.09746524938675388, 'fraction_numerical': 0.00032706459525756336, 'mean_word_length': 5.190283400809717, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2534052', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1679, 'n_words': 680}