<jupyter_start><jupyter_text>Uploading Brain Images in the Octree Format  upload_to_neuroglancer.py and upload_skeletons.py are scripts for uploading entire brain volumes, or uploading specific resolutions onto AWS or a local directory. To use these scripts, simply run each with arguments for a destination directory (usually an s3 bucket) and the source directory (usually titled "20xx-xx-xx") Data must be tif files arranged in folders where the highest level corresponds to a single, low res image. Additionally, the highest level directory should include a folder of consensus swcs and a transform.txt file containing the metadata (origin, size) for the volume  Files should be arranged as octree with 1-8 indicating volume octant ![alt text](ex_dir.png "Title")  IMPORTANT! CloudVolume by default does not accept float resolutions due to bug in its downsampling code. To get around this, pull the changes from the Neurodata fork of cloudvolume here: https://github.com/neurodata/cloud-volume/blob/voxel-res-non-int/cloudvolume/datasource/precomputed/metadata.py Additionally, make sure to include your aws keys in /.cloudvolume/secrets/. Find more info here: https://github.com/seung-lab/cloud-volume  Here are example usages of the functions in the upload scripts and what each is expecting Part 1: Uploading images<jupyter_code>data_dir = "../../../tests/data/"
dest_dir = "./test_precomputed/"

num_res = 2
files, bin_paths, vox_size,tiff_dims = upload.get_volume_info(data_dir, num_res+5, channel = 0)
print
print("Low res files: " + str(files[0]))
print("\nHigh res files: " + str(files[1]))
print("---")
print("Single image binary: " + str(bin_paths[0]))
print("\nMultiple image binaries: " + str(bin_paths[1]))
print("---")
print("Size of individual tif image: " + str(tiff_dims))
print("---")
print("Highest resolution voxel spacing in volume: " + str(vox_size))<jupyter_output>got files and binary representations of paths.
Low res files: [['..', '..', 'tests', 'data', 'default.0.tif']]

High res files: [['..', '..', 'tests', 'data', '4', 'default.0.tif'], ['..', '..', 'tests', 'data', '8', 'default.0.tif'], ['..', '..', 'tests', 'data', '6', 'default.0.tif'], ['..', '..', 'tests', 'data', '1', 'default.0.tif'], ['..', '..', 'tests', 'data', '2', 'default.0.tif'], ['..', '..', 'tests', 'data', '5', 'default.0.tif'], ['..', '..', 'tests', 'data', '3', 'default.0.tif'], ['..', '..', 'tests', 'data', '7', 'default.0.tif']]
---
Single image binary: [[]]

Multiple image binaries: [['011'], ['111'], ['101'], ['000'], ['001'], ['100'], ['010'], ['110']]
---
Size of individual tif image: (528, 400, 208)
---
Highest resolution voxel spacing in volume: [298.0, 304.0, 988.0]<jupyter_text>Cloudvolume image layers are created with the number of resolutions in the original data. These do not hold image data until it is pushed to them Note: To upload to s3: "s3://" is required as before the file path<jupyter_code>vols = upload.create_image_layer("file://" + dest_dir,tiff_dims,vox_size, num_res+5)

print("Number of volumes: " + str(len(vols)))
print("mips: " + str(vols[0].mip) + ' and ' + str(vols[1].mip))
#print("Volumes info: " + str(vols[0].info))
print("---")
print("High res volume info: " + str(vols[0].info['scales'][5]))
print("\nLow res volume info: " + str(vols[1].info['scales'][6]))<jupyter_output>{'num_channels': 1, 'type': 'image', 'data_type': 'uint16', 'scales': [{'encoding': 'raw', 'chunk_sizes': [[132, 100, 52]], 'key': '298.0_304.0_988.0', 'resolution': [298, 304, 988], 'voxel_offset': [0, 0, 0], 'size': [33792, 25600, 13312]}]}
Number of volumes: 7
mips: 6 and 5
---
High res volume info: {'chunk_sizes': [[66, 50, 52]], 'encoding': 'raw', 'key': '9536_9728_31616', 'resolution': [9536, 9728, 31616], 'size': [1056, 800, 416], 'voxel_offset': [0, 0, 0]}

Low res volume info: {'chunk_sizes': [[66, 50, 52]], 'encoding': 'raw', 'key': '19072_19456_63232', 'resolution': [19072, 19456, 63232], 'size': [528, 400, 208], 'voxel_offset': [0, 0, 0]}<jupyter_text>Uploading can be done with either Joblib parallel or non-parrallel sequential if the cpu power isn't there.  Uploading large volumes will take several days for the highest resolutions, make sure your pc does not shut off during this time. If an upload fails for any reason, you can start it again by specifying which mip you wish to upload (0-6 usually) for upload_to_neuroglancer.py<jupyter_code>%%capture
u1=upload.upload_chunks(vols[0], files[0], bin_paths[0], parallel=False) # Low res
u2=upload.upload_chunks(vols[1], files[1], bin_paths[1], parallel=False) # High res

img0 = np.squeeze(vols[0][:,:,:])
img1 = np.squeeze(vols[1][:,:,:])<jupyter_output><empty_output><jupyter_text>Visualize your data with NeuroglancerSession or with cloudvolume by reading in volumes as np arrays<jupyter_code>plot_image_2d(img0[:,:,100])
plot_image_2d(img1[:,:,100])<jupyter_output><empty_output><jupyter_text>Part 2: Uploading Skeletons<jupyter_code>#slight difference from image upload info since an origin is found for the skeletons

origin, vox_size, tiff_dims = upload_skel.get_volume_info(data_dir, num_res+5)
print("Volume origin in space: " + str(origin))
print("---")
print("Size of individual tif image: " + str(tiff_dims))
print("---")
print("Highest resolution voxel spacing in volume: " + str(vox_size))
skel_dest = "./test_skeletons/"
skel_layer = upload_skel.create_skeleton_layer("file://"+skel_dest,vox_size,tiff_dims)
#only highest res is used for any skeleton work
print("Highest res skel info: " + str(skel_layer.info['scales'][0]))
skeletons,segids = upload_skel.create_skel_segids(data_dir+"consensus-swcs/",origin)
print("Skeleton info : " + str(skeletons))
print(("---"))
print('IDs of skeletons to be uploaded:' + str(segids))

for skel in skeletons:
    skel_layer.skeleton.upload(skel)<jupyter_output>converting swcs to neuroglancer format...: 100%|██████████| 2/2 [00:00<00:00, 427.90it/s]
Uploading: 100%|██████████| 1/1 [00:00<00:00, 348.39it/s]
Uploading: 100%|██████████| 1/1 [00:00<00:00, 500.87it/s]
{'fraction_non_alphanumeric': 0.17866622428666223, 'fraction_numerical': 0.051758460517584606, 'mean_word_length': 5.567538126361656, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '492481', 'license': 'Apache-2.0', 'n_tokens_mistral': 2316, 'n_tokens_neox': 2067, 'n_words': 630, 'path': 'docs/archive/uploading_brains.ipynb', 'repo_name': 'alishakodibagkar/brainlit', 'size': '6028'}