<jupyter_start><jupyter_text>AWS. S3 Buckets> 'Working with AWS S3 buckets'- toc:true- branch: master- badges: false- comments: false- author: <NAME>- categories: [aws, s3-buckets, cloud-computing, data-storage, data-engineering, data-storage, boto3]  Overview In this notebook, we are going to have a brief view on AWS S3 storage. Concretely, we are going to discuss the following:  - How to create an AWS S3 Bucket- How to upload and download items  - How to do multi-part file transfer- How to generate pre-signed URLS.- How to set up bucket policies Moreover, we will work with AWS S3 buckets using the Boto3 Python package.  S3 Buckets AWS S3 is an object storage system. S3 stands for Simple Storage Service. By design, S3  has 11 9's of durability and stores data for millions of applications. S3 files are referred to as objects. You can find more information about S3 here and here. We will use the Boto3 Python package to interact with AWS S3; that is to create a bucket, upload and download files in the created bucket.<jupyter_code>import logging
import boto3
from botocore.exceptions import ClientError<jupyter_output><empty_output><jupyter_text>Create S3 Bucket<jupyter_code># credentials to be used
AWS_ACCESS_KEY_ID = 'Use your own credentials'
AWS_SECRET_ACCESS_KEY = 'Use your own credentials'
# create a client for the resource we will use
s3_client = boto3.client('s3', region_name='us-west-2',
                       aws_access_key_id=AWS_ACCESS_KEY_ID, 
                       aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
location = {'LocationConstraint': 'us-west-2'}
s3_client.create_bucket(Bucket='coursera-s3-bucket',
                       CreateBucketConfiguration=location)<jupyter_output><empty_output><jupyter_text>The response of the function all above is shown below: ```{'ResponseMetadata': {'RequestId': '355VX5QNYSQBTSCM',  'HostId': '7jXN853VP175Fw/il1Zvx8UXkfRsdQRXH3VrAFOcCYZl4y2ZTF6zNPp6tXvwnpBGlmAKTCP9RFA=',  'HTTPStatusCode': 200,  'HTTPHeaders': {'x-amz-id-2': '7jXN853VP175Fw/il1Zvx8UXkfRsdQRXH3VrAFOcCYZl4y2ZTF6zNPp6tXvwnpBGlmAKTCP9RFA=',   'x-amz-request-id': '355VX5QNYSQBTSCM',   'date': 'Thu, 11 Nov 2021 11:01:14 GMT',   'location': 'http://coursera-s3-bucket.s3.amazonaws.com/',   'server': 'AmazonS3',   'content-length': '0'},  'RetryAttempts': 0}, 'Location': 'http://coursera-s3-bucket.s3.amazonaws.com/'}```  Upload and object to a bucket  Bucket policies  Retrieve the policies attached to a bucket<jupyter_code>result = 3_client.get_bucket_policy(Bucket='bucket-name')<jupyter_output><empty_output><jupyter_text>The call above fails because by default there are no policies set. A bucket's policy can be set by calling the ```put_bucket_policy``` method. Moreover, a policy is defined in the same JSON format as an IAM policy.   The **Sid (statement ID)** is an optional identifier that you provide for the policy statement. You can assign a Sid value to each statement in a statement array.The **Effect** element is required and specifies whether the statement results in an allow or an explicit deny. Valid values for Effect are Allow and Deny.By default, access to resources is denied. Use the **Principal** element in a policy to specify the principal that is allowed or denied access to a resource.You can specify any of the following principals in a policy:- AWS account and root user- IAM users- Federated users (using web identity or SAML federation)- IAM roles- Assumed-role sessions- AWS services- Anonymous usersThe **Action** element describes the specific action or actions that will be allowed or denied. We specify a value using a service namespace as an action prefix (iam, ec2, sqs, sns, s3, etc.) followed by the name of the action to allow or deny.The **Resource** element specifies the object or objects that the statement covers. We specify a resource using an ARN. Amazon Resource Names (ARNs) uniquely identify AWS resources.  CORS Configuration<jupyter_code>response = s3_client.get_bucket_cors(Bucket=bucket_name)
print(response['CORSRules'])
cors_configuration = {
    'CORSRules':[{'AllowHeaders':['Authorization'],
                'AllowedMethods':['GET', 'PUT'],
                'AllowedOrigins':['*'],
                'ExposeHeaders':['GET', 'PUT'],
                 'MaxAgeSeconds':3000}
                ]
}

response = s3_client.put_bucket_cors(Bucket=bucket_name, CORSConfiguration=cors_configuration)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08817544652950486, 'fraction_numerical': 0.022156907076644812, 'mean_word_length': 4.693693693693693, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '2963', 'license': 'Apache-2.0', 'n_tokens_mistral': 1387, 'n_tokens_neox': 1318, 'n_words': 493, 'path': '_notebooks/2021-11-11-aws-s3-buckets.ipynb', 'repo_name': 'pockerman/qubit_opus', 'size': '4436'}