<jupyter_start><jupyter_text><jupyter_code>from google.colab import drive
drive.mount('/content/drive')
%cd /content/drive/MyDrive/4BAI/CAP-CV
!pip install awscli<jupyter_output>Requirement already satisfied: awscli in /usr/local/lib/python3.7/dist-packages (1.22.6)
Requirement already satisfied: PyYAML<5.5,>=3.10 in /usr/local/lib/python3.7/dist-packages (from awscli) (3.13)
Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.7/dist-packages (from awscli) (0.15.2)
Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /usr/local/lib/python3.7/dist-packages (from awscli) (0.5.0)
Requirement already satisfied: rsa<4.8,>=3.1.2 in /usr/local/lib/python3.7/dist-packages (from awscli) (4.7.2)
Requirement already satisfied: colorama<0.4.4,>=0.2.5 in /usr/local/lib/python3.7/dist-packages (from awscli) (0.4.3)
Requirement already satisfied: botocore==1.23.6 in /usr/local/lib/python3.7/dist-packages (from awscli) (1.23.6)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from botocore==1.23.6->awscli) (0.10.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.7/di[...]<jupyter_text>Create the directory /content/drive/MyDrive/configRun the following cell once<jupyter_code>text = '''
[default]
aws_access_key_id = <your access key id> 
aws_secret_access_key = <your secret access key>
region = <your region>
'''
path = "/content/drive/MyDrive/config/awscli.ini"
with open(path, 'w') as f:
   f.write(text)
!cat /content/drive/MyDrive/config/awscli.ini
import os
!export AWS_SHARED_CREDENTIALS_FILE=/content/drive/MyDrive/config/awscli.ini
path = "/content/drive/My Drive/config/awscli.ini"
os.environ['AWS_SHARED_CREDENTIALS_FILE'] = path
print(os.environ['AWS_SHARED_CREDENTIALS_FILE'])
!aws s3 ls s3://razaja-capstone/sample_data/sample_training_data_v3/ --human-readable --summarize
!aws s3 sync s3://razaja-capstone/sample_data/sample_training_data_v3/ . > out.log &
more out.log<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15581854043392504, 'fraction_numerical': 0.05128205128205128, 'mean_word_length': 5.281733746130031, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '574748', 'license': 'MIT', 'n_tokens_mistral': 812, 'n_tokens_neox': 799, 'n_words': 139, 'path': 'notebooks/Transfer_Sample_DataSet_Files_S3_to_Gdrive.ipynb', 'repo_name': 'RahulSinghalChicago/fb-mle-capstone', 'size': '2028'}