<jupyter_start><jupyter_text>awc_202103<jupyter_code>dataset_name = 'awc_202103'

container_root = '/mink_disk_0/camtraps/awc/'
path_prefix = '202103drop/'

path_to_output = f'/home/mink/camtraps/data/megadb_jsons/{dataset_name}.json' 
path_to_output_temp = f'/home/mink/camtraps/data/megadb_jsons/{dataset_name}_temp.json'<jupyter_output><empty_output><jupyter_text>Step 0 - Add an entry to the `datasets` tableDone  Step 1 - Prepare the `sequence` objects to insert into the databaseFolder structure for labeled folders (no location info, guessed sequence):`Reptiles`species - region/IMG_0463.JPG - can guess sequence from when the number in image name breaks, but sometimes there are consecutive sequences that we can't separate.`MZT reptile pics`species / 10_IMG_0779.JPG - same thing for sequence info.Folder structure for unlabeled folders (has location info but no sequence info), quite empty:`Bullo Rocky Cams Nov19-Feb20`location / RCNX0001.JPG`Treatment_R1`FM96 - MG102 (looks like both components are unique - location) / 3.1.20 (date - is different from the one in the header of the image though) / RCNX0001.JPG<jupyter_code>folder = os.path.join(container_root, path_prefix)

paths = path_utils.recursive_file_list(folder)
len(paths)
paths = sorted([p.split(folder)[1] for p in paths if path_utils.is_image_file(p) and not os.path.basename(p).startswith('.')])
len(paths)
paths[10]
paths[-10]<jupyter_output><empty_output><jupyter_text>Labeled folders<jupyter_code>labeled_folders = ('Reptiles', 'MZT reptile pics')

images_labeled = []
species = set()

last_frame_num = 0
seq_count = 0
frame_in_seq = 1

for p in tqdm(paths):
    if not p.startswith(labeled_folders):
        continue
    
    p_parts = p.split('/')
    
    clss = p_parts[1].split('-')[0].strip().lower()
    species.add(clss)
    
    fn = p_parts[-1].split('.')[0]
    fn_num = int(fn.split('_')[-1])
    
    if fn_num > last_frame_num + 1 or fn_num < last_frame_num:
        seq_count += 1  # new sequence
        frame_in_seq = 1
    last_frame_num = fn_num
    
    
    images_labeled.append({
        'dataset': dataset_name,
        'seq_id': f'labeled_{seq_count}',
        'location': 'unknown',
        'class': [clss],
        'file': p,
        'frame_num': frame_in_seq
    })
    frame_in_seq += 1
    
len(images_labeled)
species
images_labeled[10:15]
sequences_labeled = process_sequences(images_labeled, dataset_name)<jupyter_output>100%|██████████| 712/712 [00:00<00:00, 747894.93it/s]<jupyter_text>Unlabeled folders<jupyter_code>sequences_unlabeled = []
locations = set()

for p in tqdm(paths):
    if p.startswith(labeled_folders):
        continue
    
    p_parts = p.split('/')
    
    location = f'{p_parts[0]}_{p_parts[1]}'.replace(' ', '')
    locations.add(location)
    
    # RCNX0007.JPG
    im_id = p_parts[-1].split('.')[0].split('RCNX')[1]
    
    seq_id = f'dummy_{location}_{im_id}_{len(sequences_unlabeled)}'
    
    sequences_unlabeled.append({
        'dataset': dataset_name,
        'seq_id': seq_id,
        'location': location,
        'class': ['__label_unavailable'],
        'images': [{
            'file': p,
            'frame_num': 1
        }]
    })
    
len(locations)
len(sequences_unlabeled)<jupyter_output>100%|██████████| 494332/494332 [00:03<00:00, 147250.66it/s]<jupyter_text>Combined<jupyter_code>sequences = sequences_labeled + sequences_unlabeled
len(sequences)<jupyter_output><empty_output><jupyter_text>Step 2 - Pass the schema check<jupyter_code>%%time

sequences_schema_check.sequences_schema_check(sequences)
with open(path_to_output_temp, 'w', encoding='utf-8') as f:
    json.dump(sequences, f, ensure_ascii=False)<jupyter_output><empty_output><jupyter_text>Step 2b - sample unlabeled sequences and copy to flat folder<jupyter_code>sample_sequences_unlabeled = sample(sequences_unlabeled, 100000)
locations_in_sample = set()
for seq in sample_sequences_unlabeled:
    locations_in_sample.add(seq['location'])
len(locations_in_sample)  # all 156 locations are represented
seq_to_send = sequences_labeled + sample_sequences_unlabeled
len(seq_to_send)

path_pairs = []

for seq in tqdm(seq_to_send):
    seq_id = seq['seq_id']
    
    for im in seq['images']:
        frame = im['frame_num']
    
        src_path = os.path.join(container_root, path_prefix, im['file'])
        dst_path = os.path.join('/mink_disk_0/camtraps/imerit12g', 
                                f'{dataset_name}.seq{seq_id}.frame{frame}.jpg')
        path_pairs.append((src_path, dst_path))
len(path_pairs)
path_pairs[:10]
path_pairs[-100]
%%time

def copy_file(src_path, dst_path):
    return copyfile(src_path, dst_path)

with ThreadPool(12) as pool:
    dst_paths = pool.starmap(copy_file, path_pairs)
len(dst_paths)<jupyter_output><empty_output><jupyter_text>Step 3 - Add any iMerit bbox annotations<jupyter_code>annotation_path = 'path/to/folder_containing_imerit_csvs'  # dataset_name will be the part before `+`
image_filename_to_bboxes = extract_annotations(annotation_path, dataset_name)
sequences, images_updated = add_annotations_to_sequences(sequences, image_filename_to_bboxes)
# sample some sequences to make sure things look good

sample(sequences, 10)<jupyter_output><empty_output><jupyter_text>Step 4 - Save the `sequence` items to a fileYou can now take the resulting JSON file to the .Net application for bulk insertion to the database:<jupyter_code>with open(path_to_output, 'w', encoding='utf-8') as f:
    json.dump(sequences, f, ensure_ascii=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12131621768215513, 'fraction_numerical': 0.030193455071415656, 'mean_word_length': 4.587878787878788, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '222607', 'license': 'MIT', 'n_tokens_mistral': 2125, 'n_tokens_neox': 1928, 'n_words': 386, 'path': 'data_management/megadb/importer_notebooks/megadb_add_awc_202103.ipynb', 'repo_name': 'col-kol/CameraTraps', 'size': '5531'}