<jupyter_start><jupyter_text>**Title**: Uploading and Editing Metadata with Flywheel SDK**Date**:  July 23rd 2020**Description**:  Topics that we will be covering in this webinars:1. Creating the Project to host our data.2. Creating the hierarchy of Subject/Session/Acquisition matching our data input.3. Uploading the DICOM archive to each Acquisition.4. Showing how to update metadata of a container. DISCLAIMER: We assumed that you have Flywheel 12+ Version. ***  Setup Before Webinar 1. Requirements 2. Download some test data  1. RequirementsBefore getting started, we want to make sure that you have the right permission to create a new project on your instance. Below, we will be calling the `check_user_permission` function to validate whether you meet the `min_reqs` that is defined below.   Install and import dependencies<jupyter_code># Install specific packages required for this notebook
!pip install flywheel-sdk 
from getpass import getpass

import flywheel
from permission import check_user_permission # To check user permission on Flywheel Instance<jupyter_output><empty_output><jupyter_text>Flywheel API Key and Client To find your API key, first click on the circle in the top right corner of Flywheel, then click on `Profile` as shown below: ![profile_location](https://gitlab.com/flywheel-io/public/flywheel-tutorials/-/raw/master/webinars/finding_things_in_fw/assets/profile_location.png)On your profile page, your API key will be located under the `Your API Key` as below:![api_key_location](https://gitlab.com/flywheel-io/public/flywheel-tutorials/-/raw/master/webinars/finding_things_in_fw/assets/api_key_location.png)Copy this key, run the cell below, paste the key into the `Enter API_KEY here:` box and press return. Note: you must include the full cell in `Your API Key`. For example, `ss.ce.flywheel.io:123456ABCDEF789zyxw`.TIP:  While you can initialize your client as fw with fw = flywheel.Client('your-api-key'), it is essential to obscure credentials from your code, especially when sharing with others/commiting to a shared repository.WARNING: Do NOT share your API key with anyone for any reason - it is the same as sharing your password and constitutes a HIPAA violation.<jupyter_code># Password prompt (good security practice)
API_KEY = getpass('Enter API_KEY here: ')

# Initialize the client
fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))

# Clean up the API_KEY
del API_KEY<jupyter_output><empty_output><jupyter_text>Check Permission<jupyter_code># Minimum requirements that you will need to create a new project
min_reqs = {
"site": "user",
"group": "admin"
}<jupyter_output><empty_output><jupyter_text>Next step, please provide a Group ID that you are planning to use for this tutorial. Feel free to refer to the snippet below on how to find Group ID on the Flywheel Instance. ![find-group-id](https://gitlab.com/flywheel-io/public/flywheel-tutorials/-/raw/master/webinars/upload_data_modify_metadata_w_fw/assets/find-group-id.png)<jupyter_code>CHECK_GROUP_ID = input('Please enter the Group ID that you will be using to create the new project: ')<jupyter_output><empty_output><jupyter_text>If you have the right permission on the Site and Group container, the function below will `True` otherwise, a message with a list of compatible Group(s) which you have the right permission for will be displayed.Here is an example of how the message will look like.![compatible-list](https://gitlab.com/flywheel-io/public/flywheel-tutorials/-/raw/master/webinars/upload_data_modify_metadata_w_fw/assets/expected-output-compatible-list.png)<jupyter_code>check_user_permission(fw, min_reqs, group = CHECK_GROUP_ID)<jupyter_output><empty_output><jupyter_text>2. Download some test data First, we will be uploading images to a Flywheel Instance.  To get started, your first need to download the test dataset that will be used throughout this webinars. On mybinder.org or any Mac/Linux system, the following commands will download a zip archive and unzip the data into a folder called `data-upload-notebook` in your current directory:<jupyter_code>!curl -L -o data.zip "https://drive.google.com/a/umn.edu/uc?export=download&id=1UhGymg0UgoKdigGEmHbN3EcWbG5KGQ1Q"
!unzip -qf data.zip -d data-upload-notebook<jupyter_output><empty_output><jupyter_text>If the previous commands return an errors, download the file directly using the link provided to the `curl` commandabove and extract the archive in the current working directory to a folder named `data-upload-notebook` The file tree of `data-upload-notebook` should like this:```data-upload-notebook├── anx_s1│   └── anx_s1_anx_ses1_protA│       └── T1_high-res_inplane_Ret_knk_0│           └── 6879_3_1_t1.dcm.zip├── anx_s2│   └── anx_s2_anx_ses1_protA│       └── T1\ high-res\ inplane\ FSPGR\ BRAVO_0│           └── 4784_3_1_t1.dcm.zip├── anx_s3│   └── anx_s3_anx_ses1_protA│       ├── T1_high-res_inplane_Ret_knk_0│       │   └── 6879_3_1_t1.dcm.zip│       └── fMRI\ Loc\ Word\ Face\ Obj│           └── 4784_5_1_fmri.dcm.zip├── anx_s4│   └── anx_s4_anx_ses2_protB│       └── T1_high-res_inplane_Ret_knk_1│           └── 8403_4_1_t1.dcm.zip├── anx_s5│   └── anx_s5_anx_ses1_protA│       └── T1_high-res_inplane_Ret_knk_1│           └── 8403_4_1_t1.dcm.zip└── participants.csv```  SETUP COMPLETE! *** NOTES: Run the Install and Import Dependencies and Flywheel API Key and Client section in the beginning of the webinar.  Install and Import Dependencies<jupyter_code># Install specific packages required for this notebook
!pip install flywheel-sdk pandas
# Import packages
from getpass import getpass # Handle sensitive information securely eg: API key or password
import logging # To write status message 
import os # To interact with the Operating System
from pathlib import Path # Simpler method to interact with files on your local machine
import re # Regex
import time # To deal with time access
import pandas as pd
import pprint

import flywheel
from permission import check_user_permission # To check user permission on Flywheel Instance
# Instantiate a logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
log = logging.getLogger('root')<jupyter_output><empty_output><jupyter_text>Flywheel API Key and Client Get your API_KEY. More on this at in the Flywheel SDK doc [here](https://flywheel-io.gitlab.io/product/backend/sdk/branches/master/python/getting_started.htmlapi-key).<jupyter_code>API_KEY = getpass('Enter API_KEY here: ')<jupyter_output><empty_output><jupyter_text>Instantiate the Flywheel API client<jupyter_code>fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))
# Clean up the API_KEY
del API_KEY<jupyter_output><empty_output><jupyter_text>Show Flywheel logging information<jupyter_code>log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])<jupyter_output><empty_output><jupyter_text>***  *The webinar will begin from here*  Initialize a few valuesNow, we will be uploading data to a Project. The label of the Project will be defined by the `PROJECT_LABEL` variable defined below. Here we set it up to be `AnxietyStudy01` but feel free to change it to something that makes more sense to you.<jupyter_code>PROJECT_LABEL = 'AnxietyStudy01'<jupyter_output><empty_output><jupyter_text>In Flywheel each project belongs to a Group. The label of the Group that will be used to create the Project is defined by the `GROUP_ID` variable below.Specify the Group you have Admin permission on and where the Project will be created:<jupyter_code>GROUP_ID = input("Enter the Group ID here: ")<jupyter_output><empty_output><jupyter_text>We also define a variable that pointed to the root directory where the data got downloaded.If you have followed the steps above to download your data, you should have all the data in a folder called `data-upload-notebook`. If that's not the case, edit the below variable accordingly.<jupyter_code># Define Path to your data with Path()
PATH_TO_DATA = Path('data-upload-notebook')<jupyter_output><empty_output><jupyter_text>Add a New ProjectIn this section, we will be creating a new project with label `PROJECT_LABEL` in the Group's `GROUP_ID`. First, we will be getting the Group container using the `fw.lookup()` method.<jupyter_code>my_group = fw.lookup(GROUP_ID)<jupyter_output><empty_output><jupyter_text>Before creating a new project, it is a good practice to check if the Project you are trying to create exists in the Flywheel instance or not. We can do this by checking if a Project with label=PROJECT_LABEL exists in the Group you have specified:<jupyter_code>project = my_group.projects.find_first(f'label={PROJECT_LABEL}')
while project:
    log.info(f'Project {GROUP_ID}/{PROJECT_LABEL} already exists. Please update your PROJECT_LABEL variable.')
    PROJECT_LABEL = input('Please enter a new label for your new project: ')
    project = my_group.projects.find_first(f'label={PROJECT_LABEL}')

    
log.info(f'Project {GROUP_ID}/{PROJECT_LABEL} does not exist. Looking all good.')<jupyter_output><empty_output><jupyter_text>If the Project does not exist, it will return False and we can create it.<jupyter_code>if not project:
    project = my_group.add_project(label=PROJECT_LABEL)
    log.info(f'Project {PROJECT_LABEL} has successfully added to the group {GROUP_ID}.')<jupyter_output><empty_output><jupyter_text>***  Create Subjects, Sessions and Acquisitions and upload filesNow that we have a Project, we can create all the containers that are required to host our dataset. What's the plan?Following the Flywheel Hierarchy, we will loop through each subject folders and create the subject containers. We will be doing the same step to create the Session and Acquisition containers. Once we get down to the Acquisition container, we will upload the corresponding DICOM archive to it  Processing In this notebook we will parse the Subject, Session and Acquisition labels from the folders and subfolder path directly. If we wanted to do more, we could use regular expression (aka REGEX) on the path. Example on how to use regular expression in Python:```python To match a few date stringsregex = r"[a-zA-Z]+ \d+"matches = re.findall(regex, "June 24, August 9, Dec 12")for match in matches:     This will print:       June 24       August 9       Dec 12    print("Full match: %s" % (match))``` Tip: Use Regex101, an online regex tester and debugger, to write and test on example inputs before putting it in your code . Tip: Path.glob(pattern) is a Python built in module that returns any matching files with the given pattern parameters. For more information, you can visit the documentations here to learn more. We are now ready to walk our folders, create the containers accordingly and upload the DICOM zip archive to the Acquisition container.<jupyter_code>log.info('Starting upload...')

# Get Subjects that have label starts with `anx`
for subj in PATH_TO_DATA.glob('anx*'):
    log.info('Processing subject %s', str(subj))
    subject = project.add_subject(label=subj.name)
    
    # Get Sessions folder that starts with `anx`
    for ses in subj.glob('anx*'):
        log.info('Processing session %s', str(ses))
        session = subject.add_session(label=ses.name)
        
        # Get Acquisition folder that starts with `T1`
        for acq in ses.glob('T1*'):            
            log.info('Processing acquisition %s', str(acq))            
            acquisition = session.add_acquisition(label=acq.name)
            # Upload file into each Acquisition container
            for file in acq.glob('*.dcm.zip'):
                acquisition.upload_file(file)

log.info('DONE')<jupyter_output><empty_output><jupyter_text>Once the upload is done, you should have all your data available in your Flywheel Project, which should look like this:    Update Subject Metadata OverviewFor sake of example, let's demonstarate how we can update the metadata for Subject `anx_s1`.Let's first see how the subject container looks like on the Flywheel Instance.  Getting Started As you can see from the snippet above, we have 6 basic metadata for Human subject: `First Name`, `Last Name`, `Sex`, `Cohort`, `Race` and `Ethinicity`.This list will vary depending on which subject type is selected. Now, we will find that specific Subject by calling `flywheel.finder.find_first()`.<jupyter_code>anx_s1 = project.subjects.find_first('label=anx_s1').reload()<jupyter_output><empty_output><jupyter_text>Tip: Using reload() is nessecary to load the entire container. We are going to update the firstname, lastname and the sex of this Subject in this tutorial. Let's check what we have currently:<jupyter_code>print(f'Subject anx_s1 sex is: {anx_s1.sex}, first name is: {anx_s1.firstname}, last name is: {anx_s1.lastname}')<jupyter_output><empty_output><jupyter_text>We can update it with the `update` method of the container:<jupyter_code>anx_s1.update(
            firstname='John',
            lastname='Doe',
            sex='male',
)<jupyter_output><empty_output><jupyter_text>Let's reload the subject from the database to make sure the update went through:<jupyter_code>anx_s1 = project.subjects.find_first('label=anx_s1').reload()
print(f'Subject anx_s1 sex is: {anx_s1.sex}, first name is: {anx_s1.firstname}, last name is: {anx_s1.lastname}')<jupyter_output><empty_output><jupyter_text>Custom Information OverviewEach container also contains a field called `info` (aka `Custom Information` on the Flywheel Instance) which can be used to stored unstructured information in a dictionary.There is 5 different data types that you can use for the `info` section. Here is how it looks like on the Flywheel Instance:How they look like on Python: To demonstrate, we will be using the `object` data type.<jupyter_code>complicated_nested_dict = {'a_complicated_nested_dict': {'key1': [1, 2, 3, 4], 
                                                        'key2': [{'an': 'other', 'list': 'with'}, 
                                                                {'dictionaries': ['in', 'it']}]
                                                        }
                            }
anx_s1.update_info(complicated_nested_dict)
anx_s1 = project.subjects.find_first('label=anx_s1').reload()
pprint.pprint(f'Info field: {anx_s1.info}')<jupyter_output><empty_output><jupyter_text>What to expect on the Flywheel Instance:  Update all Subject Metadata with a CSV fileUpdating Subject Metadata/Info can be made by parsing CSV file or TSV file. By using this method, you can modify metadata for each Subject all at once. In this example, you will need to access the `participants.csv` file which can be found in the .zip folder you downloaded earlier.  First, you will need read the csv file with `pandas` (which imported as `pd`). INFO: pandas is a Python library that takes data (like a CSV or TSC file, or a SQL database) and creates a Pyton object with rowls and columns called data frame that look very similar to table in Excel or SPSS and R. It is useful for data manipulation and data analysis.<jupyter_code>metadata = pd.read_csv(PATH_TO_DATA/'participants.csv')
# View the data in the csv file 
display(metadata)<jupyter_output><empty_output><jupyter_text>We are going to loop through each Subjects in the Flywheel instance and check if there is any metadata stored in the `metadata` dataframe.INFO: any() function looks through the elements in an iterable and return True if any item in the iterable is true, else it returns false.If the Subject is in the `metadata` dataframe, we will add the `age` and `treatment` information into the Subject container and update the `sex` metadata for each Subject.  INFO: dataframe.loc[] method is being used to retrieve the row for the specific subject from the dataframe. While dataframe.iloc[] is used for integer-location based indexing/selection by position.<jupyter_code># Iterate through each subjects 
for subj in project.subjects.iter():
    
    # Check if there is any matches for subj.label in the participant_id columns
    if (metadata["participant_id"] == subj.label).any():
        # Get data of the subject from the `metadata`
        tmp_info = metadata.loc[(metadata["participant_id"] == subj.label)]
        # Get the age and treatment for the subject
        # Convert the information to a dictionary with the value being stored in a list
        other_metadata = tmp_info[['age', 'treatment']].to_dict('l')
        # Update the metadata contains in the subject container
        sex = tmp_info.iloc[0]['sex']
        subj.update(type='human', sex = sex)
        subj.update_info(other_metadata)
        
    else:
        print(subj.label + ' does not have metadata stored in the CSV file.')<jupyter_output><empty_output><jupyter_text>View the updated metadata in the Subject container<jupyter_code>for subj in project.subjects.iter():
    subj = subj.reload()
    print(f'Subject Label: {subj.label}, Sex: {subj.sex}, Info: {subj.info}')<jupyter_output><empty_output><jupyter_text>*** Appendix Below is a few helpful functions that you can include in your script to streamline the process of getting or creating the subject/sessions/acquisition containers.   Helpful Functions<jupyter_code>def get_or_create_subject(project, label, update=True, **kwargs):
    """Get the Subject container if it exists, else create a new Subject container.
    
    Args:
        project (flywheel.Project): A Flywheel Project.
        label (str): The subject label.
        update (bool): If true, update container with key/value passed as kwargs.
        kwargs (dict): Any key/value properties of subject you would like to update.

    Returns:
        (flywheel.Subject): A Flywheel Subject container.
    """
    
    if not label:
        raise ValueError(f'label is required (currently {label})')
        
    subject = project.subjects.find_first(f'label={label}')
    if not subject:
        subject = project.add_subject(label=label)
        
    if update and kwargs:
        subject.update(**kwargs)

    if subject:
        subject = subject.reload()

    return subject
def get_or_create_session(subject, label, update=True, **kwargs):
    """Get the Session container if it exists, else create a new Session container.
    
    Args:
        subject (flywheel.Subject): A Flywheel Subject.
        label (str): The session label.
        update (bool): If true, update container with key/value passed as kwargs.        
        kwargs (dict): Any key/value properties of Session you would like to update.

    Returns:
        (flywheel.Session): A flywheel Session container.
    """
    
    if not label:
        raise ValueError(f'label is required (currently {label})')
        
    session = subject.sessions.find_first(f'label={label}')
    if not session:
        session = subject.add_session(label=label)
        
    if update and kwargs:
        session.update(**kwargs)

    if session:
        session = session.reload()

    return session
def get_or_create_acquisition(session, label, update=True, **kwargs):
    """Get the Acquisition container if it exists, else create a new Acquisition container.
    
    Args:
        session (flywheel.Session): A Flywheel Session.
        label (str): The Acquisition label.
        update (bool): If true, update container with key/value passed as kwargs.        
        kwargs (dict): Any key/value properties of Acquisition you would like to update.

    Returns:
        (flywheel.Acquisition): A Flywheel Acquisition container.
    """
    
    if not label:
        raise ValueError(f'label is required (currently {label})')
        
    acquisition = session.acquisitions.find_first(f'label={label}')
    if not acquisition:
        acquisition = session.add_acquisition(label=label)
        
    if update and kwargs:
        acquisition.update(**kwargs)

    if acquisition:
        acquisition = acquisition.reload()

    return acquisition
def upload_file_to_acquistion(acquistion, fp, update=True, **kwargs):
    """Upload file to Acquisition container and update info if `update=True`
    
    Args:
        acquisition (flywheel.Acquisition): A Flywheel Acquisition
        fp (Path-like): Path to file to upload
        update (bool): If true, update container with key/value passed as kwargs.        
        kwargs (dict): Any key/value properties of Acquisition you would like to update.        
    """
    basename = os.path.basename(fp)
    if not os.path.isfile(fp):
        raise ValueError(f'{fp} is not file.')
        
    if acquistion.get_file(basename):
        log.info(f'File {basename} already exists in container. Skipping.')
        return
    else:
        log.info(f'Uploading {fp} to acquisition {acquistion.id}')
        acquistion.upload_file(fp)
        while not acquistion.get_file(basename):   # to make sure the file is available before performing an update
            acquistion = acquistion.reload()
            time.sleep(1)
            
    if update and kwargs:
        f = acquisition.get_file(basename)
        f.update(**kwargs)<jupyter_output><empty_output><jupyter_text>Following the Flywheel Hierarchy, you can loop through each subject folders and either get the Subject if it exists in the Project already or create it if not ( this can be done by using the `get_or_create_subject` function above). You can do the same to get/create the Session and Acquisition containers. Once you get down to the Acqusition container, you can upload the corresponding DICOM archive to it by using the `upload_file_to_acquistion` function above. Below is an example on how you can utilize the functions above to upload data into your project container.*Feel free to modify the code below that correspond to your dataset*<jupyter_code>log.info('Starting upload...')

# Get Subjects that have label starts with `anx`
for subj in PATH_TO_DATA.glob('anx*'):
    log.info('Processing subject %s', str(subj))
    subject = get_or_create_subject(project, subj.name, update=True, type='human', sex='female')  
    # Get Sessions folder that starts with `anx`
    for ses in subj.glob('anx*'):
        log.info('Processing session %s', str(ses))
        session = get_or_create_session(subject, ses.name)
        # Get Acquisition folder that starts with `T1` only
        for acq in ses.glob('T1*'):            
            log.info('Processing acquisition %s', str(acq))            
            acquisition = get_or_create_acquisition(session, acq.name)
            # Get DICOM File 
            for file in acq.glob('*.dcm.zip'):
                upload_file_to_acquistion(acquisition, file)
log.info('DONE')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08451945533865068, 'fraction_numerical': 0.006389635570440224, 'mean_word_length': 4.061106155218555, 'pattern_counts': {'":': 2, '<': 109, '<?xml version=': 0, '>': 109, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '439195', 'license': 'MIT', 'n_tokens_mistral': 6712, 'n_tokens_neox': 6526, 'n_words': 2487, 'path': 'webinars/upload_data_modify_metadata_w_fw/upload-data-modify-metadata-notebook.ipynb', 'repo_name': 'flywheel-apps/flywheel-tutorials', 'size': '22693'}