<jupyter_start><jupyter_text>The following code snipped was used to generate and save (locally) a csv file containing "Scrapped Atributes" of 10000 ready to send annotations.select SCRAPED_ATTRIBUTES from PUBLIC.PRODUCTVARIANTS pvwhere STATUS = 'ready_to_send'limit 10000<jupyter_code>df = pd.read_csv("datasets/catalog/result_10000.csv")
annotations_list = [json.loads(row[1].item()) for row in df.iterrows()]<jupyter_output><empty_output><jupyter_text>The following snippet can be used to download images associated with one of the annotations in the above created list.<jupyter_code># ann = annotations_list[0]
# args = (ann["image_links"], ann["id"], image_downloader(longer_resize=resize))
# output = _image_worker(args)
# print("{}, Broken link fraction: {:1.3f}".format(ix, 1.-(sum(output)/len(output))))<jupyter_output><empty_output><jupyter_text>The following snippet can be used to iterate over all items in the annotations list and save the images locally in the "dataset/catalog" folder, with subfolders to distinguish between items with different ids.<jupyter_code>resize = 512
for ix, ann in enumerate(annotations_list[10:20]):
    args = (ann["image_links"], ann["id"], image_downloader(longer_resize=resize))
    output = _image_worker(args)
    print("{}, Broken link fraction: {:1.3f}".format(ix, 1.-(sum(output)/len(output))))
annotations_list[13]
folder = os.path.join("datasets", "catalog")
ann = annotations_list[13]
ix = 0
os.path.join(folder, ann["id"], f"{ix}.png")
annotations_list[755]
annotations_list[13]["id"]
annotations_list[13]["image_links"]
# download_imgs(annotations_list[20:30], resize=512, workers=4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1330137807070102, 'fraction_numerical': 0.029958058717795086, 'mean_word_length': 7.697916666666667, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '660980', 'license': 'MIT', 'n_tokens_mistral': 583, 'n_tokens_neox': 532, 'n_words': 137, 'path': 'testbed.ipynb', 'repo_name': 'auag92/redcaps-downloader', 'size': '1669'}