<jupyter_start><jupyter_text>Writing Filters and Offline AugmentationWebdataset can be used for filters and offline augmentation of datasets. Here is a complete example that pre-augments a shard and extracts class labels.<jupyter_code>def extract_class(data):
    # mock implementation
    return 0

def augment(a):
    a += torch.randn_like(a) * 0.01
    return a

def augment_wds(url, output, maxcount=999999999):
    src = (
        wds.WebDataset(url)
        .decode("torchrgb")
        .to_tuple("__key__", "jpg;png", "json")
        .map_tuple(lambda x: x, augment)
    )
    with wds.TarWriter(output) as dst:
        for key, image, data in islice(src, 0, maxcount):
            print(key)
            image = image.numpy().transpose(1, 2, 0)
            image -= amin(image)
            image /= amax(image)
            sample = {
                "__key__": key,
                "png": image,
                "cls": extract_class(data)
            }
            dst.write(sample)<jupyter_output><empty_output><jupyter_text>Now run the augmentation pipeline:<jupyter_code>url = "http://storage.googleapis.com/nvdata-openimages/openimages-train-000000.tar"
url = f"pipe:curl -L -s {url} || true"
augment_wds(url, "_temp.tar", maxcount=5)<jupyter_output>e39871fd9fd74f55
f18b91585c4d3f3e
ede6e66b2fb59aab
ed600d57fcee4f94
ff47e649b23f446d<jupyter_text>To verify that things worked correctly, let's look at the output file:<jupyter_code>%%bash
tar tf _temp.tar<jupyter_output>e39871fd9fd74f55.cls
e39871fd9fd74f55.png
f18b91585c4d3f3e.cls
f18b91585c4d3f3e.png
ede6e66b2fb59aab.cls
ede6e66b2fb59aab.png
ed600d57fcee4f94.cls
ed600d57fcee4f94.png
ff47e649b23f446d.cls
ff47e649b23f446d.png
{'fraction_non_alphanumeric': 0.10112359550561797, 'fraction_numerical': 0.09225310467179183, 'mean_word_length': 3.4526315789473685, 'pattern_counts': {'":': 3, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '439361', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 732, 'n_tokens_neox': 637, 'n_words': 124, 'path': 'notebooks/writing.ipynb', 'repo_name': 'kaushikb11/webdataset', 'size': '1691'}