Software Installations: March 2019

Wednesday, March 6, 2019

Using AWS SDK for Python to store files on S3 (Simple Storage Service)

MacOS 10.14.3

Python 3.7.2

Boto3 AWS SDK for Python

Goal:

Create a Bucket in AWS S3, list the Buckets, upload, list and delete files stored in the Bucket.

Info:

Amazon S3 buckets, which are similar to file folders, store objects, which consist of data and its descriptive metadata.
There is no limit to the amount of objects an IT professional can store in a bucket, though buckets cannot exist inside of other buckets.
The S3 API to list files in the bucket (ListObjects) is paginated, returning up to 1000 keys (filenames) at a time, it also includes information like size, modified date, and ETag (md5 hash).

Install:

Using the Terminal:

pip3 install boto3

Create the credential file. By default, its location is at ~/.aws/credentials:

nano ~/.aws/credentials

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Tests (using python3):

List of Buckets:

import boto3
s3c = boto3.client('s3')
response = s3c.list_buckets()
buckets = [bucket['Name'] for bucket in response['Buckets']]
print(buckets)

Create a new Bucket:

import boto3
s3c = boto3.client('s3')
s3c.create_bucket(Bucket='my-bucket')

Check if a bucket exist:

import boto3
s3r = boto3.resource('s3')
try:

s3r.meta.client.head_bucket(Bucket='my-bucket')

except Exception as e:

print('Bucket does not exist')
print(str(e))

Delete a Bucket:

import boto3
s3r = boto3.resource('s3')
bucket = s3r.Bucket('my-bucket')
# To completely remove the bucket itself (must be empty?):
bucket.delete()

List files (keys) in the Bucket:

import boto3
s3c = boto3.client('s3')
resp = s3c.list_objects_v2(Bucket='my-bucket')
try:

for obj in resp['Contents']:

print(obj['Key'], obj['LastModified'], obj['Size'], obj['ETag'])

except:

print('no files in the bucket')

Note: If there aren’t any files (keys), the ListObjects API doesn’t include a “Contents” key in the response.

Upload a file to the Bucket:

import boto3
s3r = boto3.resource('s3')
s3r.meta.client.upload_file('/tmp/linux.jpg', 'my-bucket', 'linux.jpg')

Download a file from the Bucket:

import boto3
s3r = boto3.resource('s3')
try:

s3r.Bucket('my-bucket',).download_file('linux.jpg', 'my_local_linux.jpg')

except:

print('file does not exist')

Delete a file from the Bucket:

import boto3
s3r = boto3.resource('s3')
obj = s3r.Object('my-bucket', 'linux.jpg')
obj.delete()

Delete all files from the Bucket:

import boto3
s3r = boto3.resource('s3')
bucket = s3r.Bucket('my-bucket')
bucket.objects.all().delete()

References:

Subscribe to: Posts (Atom)