<jupyter_start><jupyter_text>Split into train/dev/test1. Split Coco `train2017` into train and validation sets    1. Split into **train** and **dev** sets using 50/50 from each of the disjoint categories    2. Select `TRAIN_IMGS_PER_CLASS` at random from each of the categories of interest for `train1`    3. Select `DEV_IMGS_PER_CLASS` at ramdom from each of the categories of interest for `dev1`2. Use Coco `val2017` as test setSave image IDs into `.txt` files**Note,** we use balanced `train1` and `dev1` to remove the class imbalance present in **train** and **dev** sets. This is done so that we can solve one problem at a time.  Imports<jupyter_code>%matplotlib inline
from pycocotools.coco import COCO

import json
import numpy as np
import pandas as pd
import skimage.io as io
import matplotlib.pyplot as plt
import pylab
pylab.rcParams['figure.figsize'] = (8.0, 10.0)
%cd ../utils/
from config import CATEGORIES_OF_INTEREST, TRAIN_IMGS_PER_CLASS, DEV_IMGS_PER_CLASS
print(CATEGORIES_OF_INTEREST, TRAIN_IMGS_PER_CLASS, DEV_IMGS_PER_CLASS)
%cd ../notebooks/<jupyter_output>/Users/gbatch/Google Drive/colab_projects/cocoapi/utils
['bird', 'cat', 'dog', 'person'] 128 128
/Users/gbatch/Google Drive/colab_projects/cocoapi/notebooks<jupyter_text>Load `strCatNms_to_imgIds` mapping from `train2017` to do (1.A)<jupyter_code>dataDir='..'
dataType='train2017'
annFile='{}/annotations/instances_{}.json'.format(dataDir, dataType)
# initialize COCO api for instance annotations
coco=COCO(annFile)
with open(f'../my_annotations/strCatNms_to_imgIds_{dataType}.json', 'r') as f:
    strCatNms_to_imgIds = json.load(f)
strCatNms_to_ImgIdsNum = {key: len(value) for (key, value) in strCatNms_to_imgIds.items()}
strCatNms_to_ImgIdsNum
train_ids = []
dev_ids = []


for key in strCatNms_to_imgIds:
    img_ids = strCatNms_to_imgIds[key].copy()
    n = len(img_ids)
    
    # shuffle, but set seed for reproducibility
    np.random.seed(42)
    np.random.shuffle(img_ids)
    
    cutoff = int(n*0.5)
    
    train_ids_in_this_cat = img_ids[:cutoff]
    dev_ids_in_this_cat = img_ids[cutoff:]
    
    print(key, len(train_ids_in_this_cat), len(dev_ids_in_this_cat))
    print('\t', train_ids_in_this_cat[:5], dev_ids_in_this_cat[:5])
    
    train_ids.extend(train_ids_in_this_cat)
    dev_ids.extend(dev_ids_in_this_cat)
len(train_ids), len(dev_ids)
len(train_ids) + len(dev_ids)
len(train_ids) / (len(train_ids) + len(dev_ids))<jupyter_output><empty_output><jupyter_text>For each of the generic categories of interest choose 20 images and put into `train1_ids`<jupyter_code>TRAIN_IMGS_PER_CLASS
train1_ids = []

for catNm in CATEGORIES_OF_INTEREST:
    catIds = coco.getCatIds(catNms=catNm)
    imgIds = coco.getImgIds(catIds=catIds)
    train_ids_in_this_cat = set(imgIds).intersection(set(train_ids))
    print(catNm, catIds, len(imgIds), len(train_ids_in_this_cat))
    
    np.random.seed(42)
    random_ids = np.random.choice(list(train_ids_in_this_cat), TRAIN_IMGS_PER_CLASS)
    
    
    train1_ids.extend(random_ids)
    
print(len(train1_ids))


# add IMGS_PER_CLASS images the rest
rest_ids_all = strCatNms_to_imgIds['']
train_ids_rest = set(rest_ids_all).intersection(set(train_ids))

np.random.seed(42)
random_ids_rest = np.random.choice(list(train_ids_rest), TRAIN_IMGS_PER_CLASS)
print('rest', '[no specific id]', len(rest_ids_all), len(train_ids_rest))
train1_ids.extend(random_ids_rest)


#print(type(train1_ids[0]))
train1_ids = list(set(train1_ids))
train1_ids = np.array(train1_ids).tolist() # make sure to use python native int - makes saving with json possible
#print(type(train1_ids[0]))

len(train1_ids), type(train1_ids)<jupyter_output>bird [16] 3237 1616
cat [17] 4114 2054
dog [18] 4385 2191
person [1] 64115 32055
512
rest [no specific id] 46223 23111<jupyter_text>For each of the generic categories of interest choose 100 images and put into `dev1_ids`<jupyter_code>CATEGORIES_OF_INTEREST
DEV_IMGS_PER_CLASS
dev1_ids = []

for catNm in CATEGORIES_OF_INTEREST:
    catIds = coco.getCatIds(catNms=catNm)
    imgIds = coco.getImgIds(catIds=catIds)
    dev_ids_in_this_cat = set(imgIds).intersection(set(dev_ids))
    print(catNm, catIds, len(imgIds), len(dev_ids_in_this_cat))
    
    np.random.seed(42)
    random_ids = np.random.choice(list(dev_ids_in_this_cat), DEV_IMGS_PER_CLASS)
    
    dev1_ids.extend(random_ids)
    
print(len(dev1_ids))


# add IMGS_PER_CLASS images the rest
rest_ids_all = strCatNms_to_imgIds['']
dev_ids_rest = set(rest_ids_all).intersection(set(dev_ids))

np.random.seed(42)
random_ids_rest = np.random.choice(list(dev_ids_rest), DEV_IMGS_PER_CLASS)
print('rest', '[no specific id]', len(rest_ids_all), len(dev_ids_rest))
dev1_ids.extend(random_ids_rest)


#print(type(train1_ids[0]))
dev1_ids = list(set(dev1_ids))
dev1_ids = np.array(dev1_ids).tolist() # make sure to use python native int - makes saving with json possible
#print(type(dev1_ids[0]))

print(len(dev1_ids))
%ls ../my_splits/<jupyter_output>dev1_ids.txt    dev_ids.txt     test_ids.txt    train1_ids.txt  train_ids.txt<jupyter_text>Get ids from `val2017` and save them as `test_ids`<jupyter_code>dataDir='..'
dataType='val2017'
annFile='{}/annotations/instances_{}.json'.format(dataDir, dataType)
# initialize COCO api for instance annotations
coco=COCO(annFile)
len(coco.getImgIds())
test_ids = coco.getImgIds()<jupyter_output><empty_output><jupyter_text>Save `train`, `dev`, `train1`, and `test` ids<jupyter_code>with open('../my_splits/train_ids.txt', 'w') as f:
    f.write(json.dumps(train_ids))
with open('../my_splits/dev_ids.txt', 'w') as f:
    f.write(json.dumps(dev_ids))
    
with open('../my_splits/train1_ids.txt', 'w') as f:
    f.write(json.dumps(train1_ids))
with open('../my_splits/dev1_ids.txt', 'w') as f:
    f.write(json.dumps(dev1_ids))
    
with open('../my_splits/test_ids.txt', 'w') as f:
    f.write(json.dumps(test_ids))
#Now read the file back into a Python list object
with open('../my_splits/train_ids.txt', 'r') as f:
    train_ids = json.loads(f.read())
with open('../my_splits/dev_ids.txt', 'r') as f:
    dev_ids = json.loads(f.read())

with open('../my_splits/train1_ids.txt', 'r') as f:
    train1_ids = json.loads(f.read())
with open('../my_splits/dev1_ids.txt', 'r') as f:
    dev1_ids = json.loads(f.read())
    
with open('../my_splits/test_ids.txt', 'r') as f:
    test_ids = json.loads(f.read())<jupyter_output><empty_output><jupyter_text>**Note**, all sets have their annotation files in `../my_annotations/`* train and development set in files with`*_train2017*`* test set in files with `*_val2017*`<jupyter_code>len(train1_ids), len(dev1_ids)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12923633077270674, 'fraction_numerical': 0.02590751619219762, 'mean_word_length': 5.509803921568627, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '181498', 'license': 'BSD-2-Clause-FreeBSD', 'n_tokens_mistral': 2701, 'n_tokens_neox': 2595, 'n_words': 519, 'path': 'notebooks/2-splits.ipynb', 'repo_name': 'GeorgeBatch/cocoapi', 'size': '6639'}