<jupyter_start><jupyter_text>Example code for exploring `dataset/meta_data/full-candidate-subsets.pkl` Understanding `dataset/meta_data/full-candidate-subsets.pkl`The metadata file `dataset/meta_data/full-candidate-subsets.pkl` is the most important piece of metadata of MetaShift, which provides the full subset information of MetaShift. To facilitate understanding, we have provided a notebook `dataset/understanding_full-candidate-subsets-pkl.ipynb` to show how to extract information from it. Basically, the pickle file stores a `collections.defaultdict(set)` object, which contains *17,938* keys. Each key is a string of the subset name like `dog(frisbee)`, and the corresponding value is a list of the IDs of the images that belong to this subset. The image IDs can be used to retrieve the image files from the Visual Genome dataset that you just downloaded. In our current version, *13,543* out of *17,938* subsets have more than 25 valid images. In addition, `dataset/meta_data/full-candidate-subsets.pkl` is drived from the [scene graph annotation](https://nlp.stanford.edu/data/gqa/sceneGraphs.zip), so check it out if your project need additional information about each image.<jupyter_code>import pickle
import numpy as np
from collections import Counter, defaultdict
import pprint
from PIL import Image
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import shutil # for copy files
import networkx as nx # graph vis
import pandas as pd

# Compare with Visaul Genome based MetaDataset
def load_candidate_subsets():
    pkl_save_path = "./meta_data/full-candidate-subsets.pkl" 
    with open(pkl_save_path, "rb") as pkl_f:
        load_data = pickle.load( pkl_f )
        print('pickle load', len(load_data), pkl_save_path)
    return load_data

VG_node_name_to_img_id = load_candidate_subsets()
assert type(VG_node_name_to_img_id)==defaultdict
print('full-candidate-subsets.pkl is a ', type(VG_node_name_to_img_id) )
print('full-candidate-subsets.pkl contains', len(VG_node_name_to_img_id), 'keys (or, subsets)')
img_IDs = sorted(VG_node_name_to_img_id['dog(frisbee)'])
img_IDs[:10]
IMGAGE_SUBSET_SIZE_THRESHOULD = 25 
counter_large_enough_subsets = 0
node_name_to_img_id = VG_node_name_to_img_id
for node_name in node_name_to_img_id.keys():
    ##################################
    # Apply a threshould: e.g., 100
    ##################################
    imageID_set = node_name_to_img_id[node_name]
    imageID_set = imageID_set
    node_name_to_img_id[node_name] = imageID_set
    if len(imageID_set) >= IMGAGE_SUBSET_SIZE_THRESHOULD:
        counter_large_enough_subsets += 1
        
print('Number of large enough subsets (i.e., with >={} images):'.format(IMGAGE_SUBSET_SIZE_THRESHOULD), counter_large_enough_subsets )
VG_node_name_to_img_id.keys()<jupyter_output><empty_output><jupyter_text>Bonus: Access General ContextsHere we show how to access some general contexts subsets. E.g. 'dog(ocean)', 'dog(street)'. Some subsets might not be present, since we have a threshold for the minimum number of images within a subset. If interested, one can further utilize the `location` and `weather` fields in the scene graph annotation to construct more subsets. Ref: https://cs.stanford.edu/people/dorarad/gqa/download.html:~:text=weather<jupyter_code>img_IDs = sorted(VG_node_name_to_img_id['ocean(dog)'])
img_IDs[:10]
img_IDs = sorted(VG_node_name_to_img_id['street(dog)'])
img_IDs[:10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10343839541547278, 'fraction_numerical': 0.008595988538681949, 'mean_word_length': 5.912871287128713, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '133785', 'license': 'MIT', 'n_tokens_mistral': 1179, 'n_tokens_neox': 1129, 'n_words': 324, 'path': 'dataset/understanding_full-candidate-subsets-pkl.ipynb', 'repo_name': 'Weixin-Liang/MetaShift', 'size': '3490'}