<jupyter_start><jupyter_text>Example Notebook for efsync scp file uploadthis notebooks includes examples for the `CLI` and the `SDK`. This Notebook can be runned in google colab.<jupyter_code>!pip3 install efsync2<jupyter_output>Collecting efsync
  Downloading https://files.pythonhosted.org/packages/dc/6a/d2076c2778a56df0743e5361a9183423b1820d5230d19d43265456c8e972/efsync-0.4.0-py3-none-any.whl
Collecting paramiko
[?25l  Downloading https://files.pythonhosted.org/packages/95/19/124e9287b43e6ff3ebb9cdea3e5e8e88475a873c05ccdf8b7e20d2c4201e/paramiko-2.7.2-py2.py3-none-any.whl (206kB)
[K     |████████████████████████████████| 215kB 5.4MB/s 
[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from efsync) (1.14.59)
Collecting scp
  Downloading https://files.pythonhosted.org/packages/4d/7a/3d76dc5ad8deea79642f50a572e1c057cb27e8b427f83781a2c05ce4e5b6/scp-0.13.2-py2.py3-none-any.whl
Collecting pyaml
  Downloading https://files.pythonhosted.org/packages/15/c4/1310a054d33abc318426a956e7d6df0df76a6ddfa9c66f6310274fb75d42/pyaml-20.4.0-py2.py3-none-any.whl
Collecting pynacl>=1.0.1
[?25l  Downloading https://files.pythonhosted.org/packages/9d/57/2f5e6226a674b2bcb6db531e8b383079b678df5b10[...]<jupyter_text>Check installed version<jupyter_code>!pip freeze | grep efsync2
!efsync2 -h
from efsync2 import efsync<jupyter_output><empty_output><jupyter_text>---  Example with `efsync.yaml`<jupyter_code>%%writefile ./efsync.yaml

#standard configuration
efs_filesystem_id: fs-2226b27a # aws efs filesystem id (moint point)
subnet_Id: subnet-17f97a7d # subnet of which the efs is running in
ec2_key_name: <KEY> # required key name for starting the ec2 instance
clean_efs: all # Defines if the EFS should be cleaned up before. values: `'all'`,`'pip'`,`'file'` uploading
# aws profile configuration
aws_profile: schueler-vz # aws iam profile with required permission configured in .aws/credentials
aws_region: eu-central-1 # the aws region where the efs is running

# upload files with scp to efs
file_dir: scp_data # extra local directory for file upload like ML models
file_dir_on_ec2: ml # name of the directory where your file from <file_dir> will be uploaded
from efsync2 import efsync

efsync('./efsync.yaml')<jupyter_output>2020-10-28 08:34:59,142 - efsync - starting....
2020-10-28 08:34:59,143 - efsync - loading config
2020-10-28 08:34:59,161 - efsync - creating security group
2020-10-28 08:34:59,746 - efsync - loading default security group
2020-10-28 08:34:59,931 - efsync - creating ssh key for scp in memory
2020-10-28 08:35:00,073 - efsync - starting ec2 instance with security group sg-0544f2ee5fdbc55a8 and subnet_Id subnet-17f97a7d
2020-10-28 08:37:43,332 - efsync - coping files from scp_data to /home/ec2-user/efs/ml 
2020-10-28 08:37:44,876 - efsync - stopping ec2 instance with instance id i-05632fd939f12b7d6
2020-10-28 08:38:30,359 - efsync - deleting iam profile
2020-10-28 08:38:31,965 - efsync - deleting ssh key
2020-10-28 08:38:32,119 - efsync - deleting security group
2020-10-28 08:38:32,351 - efsync - #################### finished after 3.5535 minutes ####################<jupyter_text>Example with pyton `dict`<jupyter_code>config = {
  'efs_filesystem_id': 'fs-2226b27a', # aws efs filesystem id (moint point)
  'subnet_Id': 'subnet-17f97a7d', # subnet of which the efs is running in
  'ec2_key_name':'<KEY>',  # required key name for starting the ec2 instance
  'clean_efs': 'all', # Defines if the EFS should be cleaned up before. values: `'all'`,`'pip'`,`'file'` uploading
  'aws_profile': 'schueler-vz', # aws iam profile with required permission configured in .aws/credentials
  'aws_region': 'eu-central-1', # the aws region where the efs is running
  'file_dir': 'scp_data', # extra local directory for file upload like ML models
  'file_dir_on_ec2': 'ml' # name of the directory where your file from <file_dir> will be uploaded
}
from efsync2 import efsync


efsync(config)<jupyter_output>2020-10-28 08:38:34,221 - efsync - starting....
2020-10-28 08:38:34,222 - efsync - loading config
2020-10-28 08:38:34,234 - efsync - creating security group
2020-10-28 08:38:34,641 - efsync - loading default security group
2020-10-28 08:38:34,818 - efsync - creating ssh key for scp in memory
2020-10-28 08:38:35,023 - efsync - starting ec2 instance with security group sg-0da96cdf3911cd70f and subnet_Id subnet-17f97a7d
2020-10-28 08:41:03,401 - efsync - coping files from scp_data to /home/ec2-user/efs/ml 
2020-10-28 08:41:04,857 - efsync - stopping ec2 instance with instance id i-0031e6c9c29c7a2d7
2020-10-28 08:41:35,337 - efsync - deleting iam profile
2020-10-28 08:41:36,495 - efsync - deleting ssh key
2020-10-28 08:41:36,664 - efsync - deleting security group
2020-10-28 08:41:36,895 - efsync - #################### finished after 3.0444999999999998 minutes ####################<jupyter_text>Example with cli and `efsync.yaml`<jupyter_code>!efsync2 -cf efsync.yaml<jupyter_output>2020-10-28 08:41:37,372 - efsync - starting....
2020-10-28 08:41:37,372 - efsync - loading config
2020-10-28 08:41:37,384 - efsync - creating security group
2020-10-28 08:41:37,760 - efsync - loading default security group
2020-10-28 08:41:37,931 - efsync - creating ssh key for scp in memory
2020-10-28 08:41:38,048 - efsync - starting ec2 instance with security group sg-0f8bfe18062a2bf9c and subnet_Id subnet-17f97a7d
2020-10-28 08:44:05,837 - efsync - coping files from scp_data to /home/ec2-user/efs/ml 
2020-10-28 08:44:07,397 - efsync - stopping ec2 instance with instance id i-03a609557a033b18d
2020-10-28 08:44:37,833 - efsync - deleting iam profile
2020-10-28 08:44:38,904 - efsync - deleting ssh key
2020-10-28 08:44:39,035 - efsync - deleting security group
2020-10-28 08:44:39,273 - efsync - #################### finished after 3.0316666666666667 minutes ####################<jupyter_text>Example with cli and parameters<jupyter_code>!efsync2 --efs_filesystem_id  fs-2226b27a \
        --subnet_Id subnet-17f97a7d \
        --ec2_key_name <KEY> \
        --clean_efs all \
        --aws_profile schueler-vz \
        --aws_region eu-central-1 \
        --file_dir scp_data \
        --file_dir_on_ec2 ml<jupyter_output>2020-10-28 08:44:39,700 - efsync - starting....
2020-10-28 08:44:39,700 - efsync - loading config
2020-10-28 08:44:39,711 - efsync - creating security group
2020-10-28 08:44:40,142 - efsync - loading default security group
2020-10-28 08:44:40,323 - efsync - creating ssh key for scp in memory
2020-10-28 08:44:40,440 - efsync - starting ec2 instance with security group sg-0623da6ebd41d7766 and subnet_Id subnet-17f97a7d
2020-10-28 08:55:26,897 - efsync - coping files from scp_data to /home/ec2-user/efs/ml 
2020-10-28 08:55:28,370 - efsync - stopping ec2 instance with instance id i-067c3e0703c72e808
2020-10-28 08:56:13,879 - efsync - deleting iam profile
2020-10-28 08:56:14,888 - efsync - deleting ssh key
2020-10-28 08:56:15,040 - efsync - deleting security group
2020-10-28 08:56:15,283 - efsync - #################### finished after 11.593 minutes ####################
{'fraction_non_alphanumeric': 0.1337579617834395, 'fraction_numerical': 0.18018400566171267, 'mean_word_length': 4.528951486697966, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '160786', 'license': 'MIT', 'n_tokens_mistral': 3500, 'n_tokens_neox': 2716, 'n_words': 655, 'path': 'examples/efsync_scp_files.ipynb', 'repo_name': 'sean-bailey/efsync2', 'size': '7104'}