How To Upload Folder To S3 Using Boto3

Raz

Posted on Mar 1, 2020

How to copy files to s3 using boto3

This commodity was also posted on razcodes.dev.

Boto is a the AWS SDK for Python. Information technology's a library that allows you to interact with the different AWS services. You can use it either on a computer/server to run all sorts of automation or to write lambda functions in your AWS account.

In this article I will become over a simple case scenario of copying a file from the local automobile to an s3 bucket. This can be easily adapted into creating automated backups in the deject.

AWS offers a gratuitous tier and if yous are interested in their 12 calendar month free you lot can create an account.

Creating the bucket

You tin create the bucket either from the console or using the AWS CLI. I used the console for this one, so I logged into my AWS account. Under Services chose S3, and then 'Create Bucket'.

Buckets in S3 must take unique names across AWS so you will but be able to requite information technology a proper noun that does not already exists. The interface will tell you that. I named mine razbackupbucket , then chose a name, and then click Create.

Provisioning credentials

In guild for u.s.a. to use the AWS services we volition need to setup admission credentials. To exercise that, under Services again go to IAM.

Here under Users click on Add User. For the username I chose 'svc-s3' (the name is more for you than annihilation). I also selected Programmatic admission only, as the user will not need admission to the console. Click Next: Permissions.

For permissions I selected 'Attach existing policies direct', then when you type s3 in the filter, you can select AmazonS3FullAccess. Side by side: Tags. Next: Review. Create user.

It is very important on this screen to either download the .csv file or re-create the Access key ID and Secret access primal because one time you printing Close, you volition never be able to run into the Cloak-and-dagger access primal always over again.

Setup Python environment

On the reckoner now, in your work folder, create a new directory where everything will be created. I like to have virtual environments for every project and to proceed things separated, so we will create one. Subsequently that we will install boto3 as well every bit python-dotenv to store out credentials properly as environment variables.

                          mkdir              s3-projection              cd              s3-projection python3              -m              venv venv              source              ./venv/bin/activate pip              install              boto3 pip              install              python-dotenv

We will now create ii files, one called 'app.py', that will take our code, and one called '.env' where we can shop the credentials.

I also copied a new file in that folder called s3.png that I will use to copy to my bucket.

The .env file looks similar this. Make sure you supplant the values with the ones y'all got from the previous step.

                          AWS_ACCESS_KEY_ID              =your-access-key-id              AWS_SECRET_ACCESS_KEY              =your-surreptitious-admission-key

And finally here is the code in app.py that will accept the image file and upload it to the S3 saucepan.

                          import              boto3              import              os              from              dotenv              import              load_dotenv              load_dotenv              ()              # this loads the .env file with our credentials                            file_name              =              's3.png'              # name of the file to upload                            bucket_name              =              'razbackupbucket'              # name of the saucepan                            s3_client              =              boto3              .              client              (              's3'              ,              aws_access_key_id              =              os              .              getenv              (              'AWS_ACCESS_KEY_ID'              ),              aws_secret_access_key              =              os              .              getenv              (              'AWS_SECRET_ACCESS_KEY'              )              )              response              =              s3_client              .              upload_file              (              file_name              ,              bucket_name              ,              file_name              )

Observe that in the last line, nosotros take the filename referenced 2 times. That is because the first 1, nosotros refer to the bodily file on the deejay and the 2nd time, we chose what the name will exist once it'due south uploaded to s3.

You can now run your program with the virtual environment yet activated.

If you lot become back and check your s3 saucepan and refresh, yous will meet the new file in there. If non, this is a adept time to track back and see what did non go according to plan.

More information on boto and S3 can be found here.

Thank you for reading this and please let me know if yous have whatever questions.