<jupyter_start><jupyter_text>Making sure your data is BIDS-compliant on FlywheelBIDS compliance is absolutely necessary for running many pre-processing pipelines on your data. Having correct metadata ensures that the pre-processing correctly handles your data. Incorrect BIDS naming can result in difficult-to-debug errors.For these reasons, we cover in-depth how to make sure your data is BIDS-compliant.First, let's look at an example subject. These red flags mean that there are errors converting the images to a BIDS-compliant name. Here you have two options  1. Click the "Info" button on the scan and make sure the BIDS section is completed correctly  2. Use the Flywheel API to change the BIDS properties The Flywheel data modelFlywheel stores zip archives of the dicom data and nifti files. The names of the nifti files are usually determined by `dcm2niix` and that name sticks with the file for the rest of its life. I believe these files are managed using MongoDB's GridFS framework, so there are some non-intuitive things that can happen.   - More than one file can exist with the exact same name. The files are indexed by a BSON id, which is    unique for each file. The field containing the file name is not necessarily unique.  - Unless the data was uploaded in BIDS format, the actual BIDS-named data does not exist in Flywheel.    Instead, when a job needs requires BIDS input, the job uses the Flywheel API to download and rename    the data to be BIDS compliant.    - The renaming and JSON Sidecar creation uses information stored in Flywheel's MongoDB. Therefore to      change the BIDS filename, you simply update the files Document/BSON and next time a gear is run,      this new metadata will be used to create the BIDS directory.The metadata associated with each file can be edited through the Flywheel web interface or through the API. The web interface is not complete and still a little buggy (i.e. not all UI's contain BIDS fields). The SDK is fairly complete and documented, so that will be the focus of this notebook.         Editing BIDS info from the web interfaceYou can change the information on your scans by clicking the info icon in the file's row.The fields can then be edited manually. This is the initial interface presented:where basic options about the contents of the file are editable. This will always be available when clicking the Info button. If the `BIDS Curation` gear has been run, you can scroll down and see the BIDS fields:It is unclear to me if the `BIDS Curation` gear needs to be re-run for these changes to be committed to MongoDB. Editing BIDS Info Using the APIThe remainder of this document is a deep dive into how to access the BIDS data using the Flywheel SDK, which can be installed from PyPI using```bashpip install flywheel-sdk```The documentation for this package can be found [here](https://flywheel-io.github.io/core/branches/master/python/index.html).<jupyter_code>import flywheel

# Create client
fw = flywheel.Client()<jupyter_output><empty_output><jupyter_text>Logging In to FlywheelNote the documentation is out of date and the constructor for `flywheel.Client` does not take any arguments. Instead you have to login with your API key using the commandline `fw login`. Verify that your login worked and you can retrieve data from Flywheel:<jupyter_code>fw.get_current_user()['email']<jupyter_output><empty_output><jupyter_text>The Basic Data ModelEverything in flywheel is a *container*. >>>When you access a container, you can either do so as a method attribute, or a function. With a method, you implement a flywheel finder which indexes the data but doesn't quite return anything to you yet. As a function, the returned object usually is a dictionary (or list of dictionaries).For example, we can see the currently available project containers with the following command:<jupyter_code># the function
fw.projects()
# the indexer
fw.projects<jupyter_output><empty_output><jupyter_text>Searching the DataThe indexer itself has some common search methods such as `find()` and `find_first()`.<jupyter_code>project = fw.projects.find_first('label=Reward2018')
print(project)<jupyter_output>{'analyses': None,
 'created': datetime.datetime(2018, 12, 19, 15, 21, 10, 217000, tzinfo=tzutc()),
 'description': None,
 'files': [{'classification': {},
            'created': datetime.datetime(2019, 1, 9, 19, 48, 52, 13000, tzinfo=tzutc()),
            'hash': 'v0-sha384-1c77469675dad45012de0cbf1da5053e27918713ad822de64502a711d0238e56bc7b3f94946ba1e8d1921d1c8d68d1d4',
            'id': '66463c4e-<PASSWORD>',
            'info': {u'BIDS': {u'Filename': u'',
                               u'Folder': u'',
                               u'Path': u'',
                               u'error_message': u"Filename u'' is too short",
                               u'ignore': False,
                               u'template': u'project_file',
                               u'valid': False}},
            'info_exists': True,
            'mimetype': 'text/plain',
            'modality': None,
            'modified': datetime.datetime(2019, 1, 9, 22, 38, 19, 579000, tzinfo=tzutc[...]<jupyter_text>Once you can find a container by label, use the BSON id to extract its other key-value pairs. You'll notice that there is an `'_id'` field in all of these documents. This is the BSON id that is used to identify files in flywheel. Everything - files, subjects, sessions, analyses - has one of these that can be used for direct access. Confusingly, you can't access it directly with `project['id']`: you'll need to use `project['_id']`.<jupyter_code>project['_id']<jupyter_output><empty_output><jupyter_text>In MongoDB, containers don't have to be accessed in order. For example, we can pull the subjects out of the project here:<jupyter_code>for subject in project.subjects():
        print('%s: %s' % (subject.id, subject.label))<jupyter_output>5c352b991de80b0024480dda: 10180
5c352a081de80b0024480dcc: 102102
5c3535641de80b0024480e41: 104059
5c3527fe1de80b0024480db7: 10410
5c3541301de80b0024480ebc: 105168
5c353ab01de80b00198acafb: 105272
5c3542de1de80b00198acb34: 105490
5c35186f1de80b00198ac9f9: 105634
5c3539ea1de80b001c0da9c8: 106573
5c353e151de80b0024480ea0: 107055
5c3545391de80b00198acb42: 109741
5c3511cd1de80b0024480c9f: 11010
5c3532511de80b0024480e26: 11176
5c351c2f1de80b00156d3d76: 11186
5c1a8b619011bd0011369953: 11242
5c1a8b249011bd001436aa02: 11305
5c1a8acd9011bd0015369b48: 113220
5c1a89509011bd001436a68d: 11399
5c1a88959011bd0013369c49: 11419
5c1a876b9011bd0015369675: 11569
5c1a87089011bd0013369978: 11588
5c1a85f79011bd0014369b3b: 11599
5c1a8b2d9011bd0015369ba6: 11647
5c1a8a939011bd0013369f79: 116531
5c1a8a2c9011bd0011369848: 11706
5c1a89b69011bd001436a741: 117256
5c1a88cf9011bd001436a556: 11762
5c1a87df9011bd001436a2f3: 11801
5c1a86b89011bd0015369557: 11866
5c1a863c9011bd00113693cc: 118990
5c1a85ae9011bd0014369914: 1[...]<jupyter_text>And the sessions from a project as shown below. Sessions are available as child nodes of the project container. You can access a child node by using the `.` attribute on a container. Here, we use `find_first()`.<jupyter_code>sessions = project.sessions.find_first()
sessions<jupyter_output><empty_output><jupyter_text>Notice that the use of `()` on a container returns an iteratable list, but without the function parentheses, the container is an in-built attribute that indexes the flywheel database:<jupyter_code>type(project.subjects())
type(project.subjects)<jupyter_output><empty_output><jupyter_text>The `get()` function can be used to go directly to a container:<jupyter_code>subject = fw.get("5c1a82ca9011bd0015368ee6")<jupyter_output><empty_output><jupyter_text>And while containers are not quite Russian nesting dolls, they can still be treated as such (project > subject > session > aqsuisition)<jupyter_code>for ses in subject.sessions():
    print('%s: %s' % (ses.id, ses.label))
ses1 = fw.get("5c1a82ca9011bd0015368ee7")<jupyter_output><empty_output><jupyter_text>File are accessed via the `files` key, but it's easier to use `get()` on the BSON id.<jupyter_code>f1 = fw.get("5c1a82ca9011bd0011368ed1")
f1.label<jupyter_output><empty_output><jupyter_text>Viewing the data as TabularUsing `View()`, you can create a tabular index of the data you want to search through. From there, use `pandas` to present it as a dataframe:<jupyter_code>import pandas as pd

view = fw.View(columns='subject')
df = fw.read_view_dataframe(view,project.id)
df.head()<jupyter_output><empty_output><jupyter_text>In this implementation, there is a clear hierarchy to be followed. Basically, you specify the columns of the minor container you want to return, as members of some major container you have already assigned in the environment. So for example, if I wanted to see the name and size of all of the files of a session from a participant, I'd search the acquisition container, and execute it within the session container (`ses1` that was assigned earlier):<jupyter_code>view = fw.View(columns=["file.size", "file.name"], container = "acquisition", filename = "*")
df = fw.read_view_dataframe(view, ses1.id)
df.head()<jupyter_output><empty_output><jupyter_text>Note that to index and search through files, you must use the `container` and `filename` keywords in the View() function. The following function lists all of the available columns that can be used to tabularise flywheel containers:<jupyter_code>fw.print_view_columns()<jupyter_output>project (group): All column aliases belonging to project
project.id (string): The project id
project.label (string): The project label
project.info (object): The freeform project metadata
subject (group): All column aliases belonging to subject
subject.id (string): The subject id
subject.label (string): The subject label or code
subject.firstname (string): The subject first name
subject.lastname (string): The subject last name
subject.sex (string): The subject sex (one of female|male|other|unknown)
subject.race (string): The subject race (one of American Indian or Alaska Native|Asian|Native Hawaiian or Other Pacific Islander|Black or African American|White|More Than One Race|Unknown or Not Reported)
subject.ethnicity (string): The subject ethnicity (one of Not Hispanic or Latino|Hispanic or Latino|Unknown or Not Reported)
subject.cohort (string): The subject cohort (one of Control|Study|Training|Test|Validation)
subject.type (string): The subject type (one of human|animal|phantom)
subj[...]<jupyter_text>Manipulating Data ValuesWe can manipulate field values by passing a key-value pair to the `update()` function. Then, to update the object in the python environment, simply reassign it with `reload()`:<jupyter_code>newname = {"label":"foo"}
f1.update(newname)
f1 = f1.reload()
f1.label<jupyter_output><empty_output><jupyter_text>Checking for BIDS ValidityFlywheel automatically tries to classify and organise your data into BIDS. The files of an acquisition have an `info` dict that can be used to check if the associated files were BIDS valid:<jupyter_code>for acq in ses1.acquisitions():
    
    print "----------------"
    print "acquisition: " + acq.label
    print acq.info
    print
    for f in acq.files:
        print "file:"
        print f.name
        print f.classification
        print f.info
        print<jupyter_output>----------------
acquisition: B0map_onesizefitsall_v4
{u'BIDS': {u'ignore': False, u'template': u'acquisition'}}

file:
B0map_onesizefitsall_v4.dicom.zip
{}
{u'BIDS': u'NA'}

----------------
acquisition: MPRAGE_TI1100_ipat2
{u'BIDS': {u'ignore': False, u'template': u'acquisition'}}

file:
MPRAGE_TI1100_ipat2.dicom.zip
{u'Intent': [u'Structural'], u'Measurement': [u'T1']}
{u'BIDS': {u'error_message': u'', u'Filename': u'MPRAGE_TI1100_ipat2.dicom.zip', u'ignore': False, u'valid': True, u'template': u'dicom_file', u'Path': u'sourcedata/sub-19830/ses-neff2', u'Folder': u'sourcedata'}}

file:
MPRAGE_TI1100_ipat2_2.nii.gz
{u'Intent': [u'Structural'], u'Measurement': [u'T1']}
{u'BIDS': {u'Run': u'', u'error_message': u'', u'Ce': u'', u'Filename': u'sub-19830_ses-neff2_T1w.nii.gz', u'ignore': False, u'Acq': u'', u'valid': True, u'template': u'anat_file', u'Path': u'sub-19830/ses-neff2/anat', u'Rec': u'', u'Folder': u'anat', u'Modality': u'T1w', u'Mod': u''}}

file:
MPRAGE_TI1100_ipat2_2_mriqc[...]<jupyter_text>We can create a viewer of these files and use the file info object to assess whether or not the flywheel gear successfully classified the files in valid BIDS:<jupyter_code>#build viewer
view = fw.View(columns=["file.info"], container = "acquisition", filename = "*")
df = fw.read_view_dataframe(view, ses1.id)
df<jupyter_output><empty_output><jupyter_text>Using the acquisition ID, we can dig into the file information (which we looped over just above) and assess whether a nifti image was created, whether the BIDS key was created, and pull any info that was processed:<jupyter_code># eg get the ep2d from the above table
ep2d = fw.get("5c1a82ca9011bd0013369089")
# pull out only nifti files
niftis = [x for x in ep2d.files if x['type'] == 'nifti']
# check if the nifti file 'info' has bids
print([x['info'].keys() for x in niftis])
def ExtractBidsValidity(acquisitionID):
        
    '''
    A helper function to dig into the file.info object (a dictionary of dictionaries) and extract the BIDS validity value
    '''
    
    #create the acquisition object and pull the niftis
    acq = fw.get(acquisitionID)
    niftis = [x for x in acq.files if x['type'] == 'nifti']
    
    # if there are no niftis, return
    if (len(niftis) < 1):
        return None
    else:
        df = []
        
        #for each nifti, if the info has a BIDS dict:
        for nii in niftis:
            info = nii['info']
            if 'BIDS' in info.keys() and isinstance(info['BIDS'], dict):
                
                #also add the acquisition idto the dict for joining purposes
                nii['info']['BIDS']['acquisition.id'] = str(acquisitionID)
                
                #pull out the bids info
                df.append(nii['info']['BIDS'])
            else:
                return None

        return(df)
ExtractBidsValidity(df.loc[2,'acquisition.id'])<jupyter_output><empty_output><jupyter_text>Here we loop over the rows to get all of the nifti bids info:<jupyter_code>nifti_info = []

for ind, row in df.iterrows():
    temp_info = ExtractBidsValidity(row["acquisition.id"])
    if temp_info is not None:
        
        nifti_info.extend(temp_info)<jupyter_output><empty_output><jupyter_text>And then we join merge the nifti info to the original query data frame:<jupyter_code>niftis = pd.DataFrame(nifti_info)
df_with_niftis = pd.merge(df[['acquisition.id', 'acquisition.label', 'project.label', 'session.label', 'subject.label']], niftis, on="acquisition.id")<jupyter_output><empty_output><jupyter_text>We can thus examine the exact nifti files that are not BIDS compliant:<jupyter_code>df_with_niftis<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09275806881835157, 'fraction_numerical': 0.06428380901573753, 'mean_word_length': 4.612649700598802, 'pattern_counts': {'":': 1, '<': 83, '<?xml version=': 0, '>': 88, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '90338', 'license': 'MIT', 'n_tokens_mistral': 5379, 'n_tokens_neox': 4720, 'n_words': 1673, 'path': 'notebooks/Checking_BIDS_Data_in_Python.ipynb', 'repo_name': 'TinasheMTapera/bids-on-flywheel', 'size': '15013'}