<jupyter_start><jupyter_text>Questions1. Could this be a CWL compiler?2. WIll it take a root document and return the whole structure?3. Can I find the dockerRequirement anywhere in the doc?4. Can I find the dockerRequirement using the schema?  1. CWL Docker CompilerWhat does that mean? Abstractly, that it would read an input document, look for all docker requirements and hints, pull the images, and then write a shell script to reload everything  2. Root document and return whole structure?<jupyter_code>workflow = parse('/Users/dcl9/Code/python/mmap-cwl/mmap.cwl')<jupyter_output><empty_output><jupyter_text>Yes, that works<jupyter_code># This function will find dockerImageId anyhwere in the tree
def find_key(d, key, path=[]):
    if isinstance(d, list):
        for i, v in enumerate(d):
            for f in find_key(v, key, path + [str(i)]):
                yield f
    elif isinstance(d, dict):
        if key in d:
            pathstring = '/'.join(path + [key])
            yield pathstring
        for k, v in d.items():
            for f in find_key(v, key, path + [k]):
                yield f
    
# Could adapt to find class: DockerRequirement instead
for x in find_key(workflow, 'dockerImageId'):
    print x, dpath.util.get(workflow, x)
dpath.util.get(workflow, 'steps/0/run/steps/0/run/hints/0')<jupyter_output><empty_output><jupyter_text>extract docker image names<jupyter_code>def image_names(workflow):
    image_ids = []
    for x in find_key(workflow, 'dockerImageId'):
        image_id = dpath.util.get(workflow, x)
        if image_id not in image_ids: image_ids.append(image_id) 
    return image_ids
image_names(workflow)
import docker

def docker_hashes(image_ids):
    for name in image_ids:
        print name

docker_hashes(image_names(workflow))<jupyter_output>dukegcb/xgenovo
dukegcb/glimmer
dukegcb/go-blast
dukegcb/extract-go-terms<jupyter_text>Docker IOQuery docker for the sha of the docker image id<jupyter_code>%%sh

eval $(docker-machine env default)
import docker_io
images = get_image_metadata(client, 'dukegcb/xgenovo')
for img in images:
    write_image(client, img, '/tmp/images')
md<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0953262378528459, 'fraction_numerical': 0.004627487274409995, 'mean_word_length': 3.924829157175399, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '124737', 'license': 'MIT', 'n_tokens_mistral': 738, 'n_tokens_neox': 712, 'n_words': 216, 'path': 'cwl-freezing.ipynb', 'repo_name': 'Duke-GCB/cwl-freezer', 'size': '2161'}