<jupyter_start><jupyter_text>How to create a simple pipeline to process a dataset by batches<jupyter_code>import sys
import warnings
warnings.filterwarnings("ignore")

import PIL
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

# the following line is not required if Dataset is installed as a python package.
sys.path.append("../..")
from dataset import Dataset, DatasetIndex, R, P, V, C
from dataset.opensets import MNIST
BATCH_SIZE = 10<jupyter_output><empty_output><jupyter_text>Create a dataset [MNIST](http://yann.lecun.com/exdb/mnist/) is a dataset of handwritten digits frequently used as a baseline for machine learning tasks. Let's download the data and create the dataset (it might take a few minutes to complete)<jupyter_code>dataset = MNIST()<jupyter_output>Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Extracting /tmp/t10k-labels-idx1-ubyte.gz
Extracting /tmp/train-labels-idx1-ubyte.gz
Extracting /tmp/train-images-idx3-ubyte.gz
Extracting /tmp/t10k-images-idx3-ubyte.gz<jupyter_text>MNIST dataset will create instances of [ImagesBatch](https://analysiscenter.github.io/dataset/intro/images_batch.html). You can also use [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html):    from dataset.opensets import CIFAR10        dataset = CIFAR10()It takes much more time to download, though.For CIFAR examples see [the image augmentation tutorial](./07_image_augmentation_examples.ipynb). Let's get a batch and look at the dataset content.<jupyter_code>def show_images(batch):
    shape = batch.image_shape
    total_width = len(batch) * shape[0]
    img = PIL.Image.new('L' if shape[-1]==1 else 'RGB', (total_width, shape[1]))
    for image, offset in zip(batch.images, np.arange(0, len(batch)*shape[0], shape[0])):
        img.paste(image, (offset,0))
    fig, ax = plt.subplots(1, figsize=(10, 4))
    ax.axis('off')
    ax.imshow(img, cmap="gray" if shape[-1]==1 else None)
    plt.show()
batch = dataset.train.next_batch(BATCH_SIZE)
show_images(batch)<jupyter_output><empty_output><jupyter_text>Execute the cell above several time to see different batches.  Define a pipeline A pipeline represents a sequence of actions applied to a dataset.These actions might come from [Pipeline API](https://analysiscenter.github.io/dataset/intro/pipeline.html) or a batch class action-methods (e.g. [ImagesBatch](https://analysiscenter.github.io/dataset/intro/images_batch.html))Just write them one after another.<jupyter_code>simple_pipeline = (dataset.train.p
                     .scale(p=.5, factor=1.5, preserve_shape=True)
                     .rotate(p=.5, angle=60)
                     .salt(p=.5, color=255, p_noise=.05)
                     .elastic_transform(p=.5, alpha=20, sigma=1.8)
)
batch = simple_pipeline.next_batch(BATCH_SIZE)
show_images(batch)<jupyter_output><empty_output><jupyter_text>Read [the documentation](https://analysiscenter.github.io/dataset/intro/pipeline.html) for advanced pipeline techniques.  Use pipeline variables<jupyter_code>simple_pipeline = (dataset.train.p
                     .init_variable('angle', 60)
                     .init_variable('factor', 1.5)
                     .init_variable('salt_color', 255)
                     .init_variable('proba', .5)
                     .scale(p=V('proba'), factor=V('factor'), preserve_shape=True)
                     .rotate(p=V('proba'), angle=V('angle'))
                     .salt(p=V('proba'), color=V('salt_color'), p_noise=.05)
                     .elastic_transform(p=V('proba'), alpha=20, sigma=1.8)
)
batch = simple_pipeline.next_batch(BATCH_SIZE)
show_images(batch)<jupyter_output><empty_output><jupyter_text>Read [more about variables and how to use them](https://analysiscenter.github.io/dataset/intro/pipeline.htmlpipeline-variables).  Pipeline configuration Same result can be achieved with a pipeline config.<jupyter_code>config = dict(angle=60, factor=1.5, salt_color=255, proba=.5)

simple_pipeline = (dataset.train.pipeline(config)
                     .scale(p=C('proba'), factor=C('factor'), preserve_shape=True)
                     .rotate(p=C('proba'), angle=C('angle'))
                     .salt(p=C('proba'), color=C('salt_color'), p_noise=.05)
                     .elastic_transform(p=C('proba'), alpha=20, sigma=1.8)
)
batch = simple_pipeline.next_batch(BATCH_SIZE)
show_images(batch)<jupyter_output><empty_output><jupyter_text>Sampling random values Sometimes you might want random values instead of hard-coded constants.`R` and `P` [named expressions](https://analysiscenter.github.io/dataset/intro/named_expr.html) might come in handy here.<jupyter_code>config = dict(angle=R('uniform', -45, 45), factor=P(R('normal', 1.5, .2)), salt_color=255, proba=.5)

simple_pipeline = (dataset.train.pipeline(config)
                     .scale(p=C('proba'), factor=C('factor'), preserve_shape=True)
                     .rotate(p=C('proba'), angle=C('angle'))
                     .salt(p=C('proba'), color=C('salt_color'), p_noise=.05)
                     .elastic_transform(p=C('proba'), alpha=20, sigma=1.8)
)
batch = simple_pipeline.next_batch(BATCH_SIZE)
show_images(batch)<jupyter_output><empty_output><jupyter_text>The difference between `R(...)` and `P(R(...))` is that the former gives a single random value for **all** batch items, while the latter gives a random value for **each** batch item.  Running pipelines See [the documentation](https://analysiscenter.github.io/dataset/intro/pipeline.htmlrunning-pipelines) or [batch operations tutorial](./01_batch_operations.ipynb) for a the detailed description.  next_batch<jupyter_code>for i in range(5):
    batch = simple_pipeline.next_batch(BATCH_SIZE, n_epochs=1)
    show_images(batch)<jupyter_output><empty_output><jupyter_text>See [batch operations tutorial](./01_batch_operations.ipynb) for more info about `next_batch` / `gen_batch` and their parameters (`shuffle`, `drop_last`, `n_epochs`, etc)  gen_batch While `next_batch` is an ordinary method returning processed batches, `gen_batch` is a generator<jupyter_code>for batch in simple_pipeline.gen_batch(BATCH_SIZE, n_epochs=1):
    # do whatever you want with the batch
    pass<jupyter_output><empty_output><jupyter_text>Executing this cell might take **a lot of time**, depending on your hardware, pipeline content and the dataset size. If you want to use large batches with heavy actions (or with I/O operations) then consider using [target](https://analysiscenter.github.io/dataset/intro/parallel.html?highlight=targettarget-threadsIncreasing)='threads'. It might gain considerable boost for multi-CPU platforms.  run It is just a concise form of __`for batch in pipeline.gen_batch(...)`__<jupyter_code>BATCH_SIZE=100
simple_pipeline.run(BATCH_SIZE, n_epochs=1, shuffle=True, drop_last=True, bar=True, prefetch=2)<jupyter_output>100%|█████████▉| 599/600 [01:35<00:00,  6.28it/s]
{'fraction_non_alphanumeric': 0.12951467268623024, 'fraction_numerical': 0.018623024830699775, 'mean_word_length': 4.354229607250756, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 39, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '134545', 'license': 'Apache-2.0', 'n_tokens_mistral': 2396, 'n_tokens_neox': 2268, 'n_words': 521, 'path': 'examples/tutorials/02_pipeline_operations.ipynb', 'repo_name': 'mikhailkin/dataset', 'size': '7088'}