<jupyter_start><jupyter_text>Using Nipype with Amazon Web Services (AWS)Several groups have been successfully using Nipype on AWS. This procedureinvolves setting a temporary cluster using StarCluster and potentiallytransferring files to/from S3. The latter is supported by Nipype through`DataSink` and `S3DataGrabber`.  Using DataSink with S3The `DataSink` class now supports sending output data directly to an AWS S3bucket. It does this through the introduction of several input attributes to the`DataSink` interface and by parsing the `base_directory` attribute. This classuses the [boto3](https://boto3.readthedocs.org/en/latest/) and[botocore](https://botocore.readthedocs.org/en/latest/) Python packages tointeract with AWS. To configure the `DataSink` to write data to S3, the user mustset the ``base_directory`` property to an S3-style filepath.For example:<jupyter_code>from nipype.interfaces.io import DataSink
ds = DataSink()
ds.inputs.base_directory = 's3://mybucket/path/to/output/dir'<jupyter_output><empty_output><jupyter_text>With the `"s3://"` prefix in the path, the `DataSink` knows that the outputdirectory to send files is on S3 in the bucket `"mybucket"`. `"path/to/output/dir"`is the relative directory path within the bucket `"mybucket"` where output datawill be uploaded to (***Note***: if the relative path specified contains folders thatdon’t exist in the bucket, the `DataSink` will create them). The `DataSink` treatsthe S3 base directory exactly as it would a local directory, maintaining supportfor containers, substitutions, subfolders, `"."` notation, etc. to route outputdata appropriately.There are four new attributes introduced with S3-compatibility: ``creds_path``,``encrypt_bucket_keys``, ``local_copy``, and ``bucket``.<jupyter_code>ds.inputs.creds_path = '/home/neuro/aws_creds/credentials.csv'
ds.inputs.encrypt_bucket_keys = True
ds.local_copy = '/home/neuro/workflow_outputs/local_backup'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09433962264150944, 'fraction_numerical': 0.006629270780214177, 'mean_word_length': 6.266666666666667, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '44887', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 576, 'n_tokens_neox': 572, 'n_words': 204, 'path': 'workshop/nipype_tutorial/notebooks/advanced_aws.ipynb', 'repo_name': 'jooh/workshop_pybrain', 'size': '1961'}