How to configure s3 bucket on CentOS/RHEL 5/6/7

s3cmd is a command line utility used to transfer your information (photographs, features, archives and so on.), you first make a bucket in one of the AWS Amazon. S3cmd is composed in Python. It’s an open source venture accessible under GNU Public License v2 (GPLv2) and is free for both business and private use. You will just need to pay Amazon for utilizing their storage.

This article will help you to how to utilize introduce s3cmd on CentOS and RHEL Systems and manage s3 bucketvia commad line in simple steps.

On CentOS / RHEL:
All the users of RHEL and CentOS with all other derived distributions. Create a s3tools.repo file in /etc/yum.repos.d directory and add following content

# vim /etc/yum.repos.d/s3tools.repo

[s3tools]
name=Tools for managing Amazon S3 - Simple Storage Service (RHEL_6)
type=rpm-md
baseurl=http://s3tools.org/repo/RHEL_6/
gpgcheck=1
gpgkey=http://s3tools.org/repo/RHEL_6/repodata/repomd.xml.key
enabled=1

Install s3cmd Package
After adding s3tools repository we can simply install s3cmd. You can install it using simply executing following commands on your system.
On CentOS/RHEL:

# yum install s3cmd

Configure s3cmd Environment:

In order to configure s3cmd we would required two keys – Access Key and Secret Key of your Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.

After getting key files, use below command to configure s3cmd om CentOS Systems.

# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Encryption password: xxxxxxxxxx
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
Uses of s3cmd Command Line

Once configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.

1.Creating New Bucket
You can create a new bucket in Amazon s3 by using below command. It will create bucket named techoism_code in your account.

# s3cmd mb s3://techoism_code

Sample Output:

Bucket 's3://techoism_code/' created

2. Uploading file in Bucket
Using Below command we will upload any file to s3 bucket using s3cmd command.

# s3cmd put tech.txt s3://techoism_code/

Sample Output:

tech.txt -> s3://techoism_code/tech.txt  [1 of 1]
 190216 of 190216   100% in    0s  1668.35 kB/s  done

3. Uploading Directory in Bucket
Also We can upload entire directory in s3 Account. We use -r to upload it recursively like below.

# s3cmd put -r backup s3://techoism_code/

Sample Output:

backup/tech1.txt -> s3://techoism_code/backup/tech1.txt  [1 of 2]
 9984 of 9984   100% in    0s    18.78 kB/s  done
backup/tech2.txt -> s3://techoism_code/backup/tech2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Verify you are not including slash in transfer directory named backup (eg: backup/), else it will transfer only content to backup directory only.

# s3cmd put -r backup/ s3://techoism_code/

Sample Output:

backup/tech1.txt -> s3://techoism_code/tech1.txt  [1 of 2]
 9984 of 9984   100% in    0s    21.78 kB/s  done
backup/tech2.txt -> s3://techoism_code/tech2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

4. List Data of S3 Bucket
If we have to list the objects of s3 bucket use ls switch with s3cmd.

# s3cmd ls s3://techoism_code/

Sample Output:

                       DIR   s3://techoism_code/backup/
2013-09-03 10:58    190100   s3://techoism_code/tech.txt

4. List Data of directory in S3 Bucket
If we have to list the objects of any directory in s3 bucket use -r & ls switch with s3cmd.

# s3cmd ls -r s3://techoism_code/

Sample Output:

2013-09-03 10:58    190318   s3://techoism_code/backup/tech1.txt
2013-09-03 10:58    197168   s3://techoism_code/backup/tech2.txt
2013-09-03 10:58    190100   s3://techoism_code/tech.txt

6. Download Files from Bucket
Some times if we need to download files from s3 bucket, Use following commands to download it.

# s3cmd get s3://techoism_code/tech.txt

Sample Output:

s3://techoism_code/tech.txt -> ./tech.txt  [1 of 1]
 4 of 4   100% in    0s    10.84 B/s  done

7. Remove Data of S3 Bucket
If we have to remove files & folder from s3 bucket, Use following commands to delete it.

Removing file from s3 bucket
# s3cmd del s3://techoism_code/tech.txt

Sample Output:

File s3://techoism_code/tech.txt deleted

Removing directory from s3 bucket
# s3cmd del s3://techoism_code/backup

Sample Output:

File s3://techoism_code/backup deleted

8. Remove S3 Bucket
If we have to remove s3 bucket, we can simply remove it using bellow command Before removing bucket verify its empty.

# s3cmd rb s3://techoism_code

Sample Output:

ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

Above command failed because of s3 bucket was not empty

To remove bucket first remove all files & directory inside bucket and then use command again.

# s3cmd rb s3://techoism_code

Sample Output:

Bucket 's3://techoism_code/' removed

9. List All S3 Bucket
Use following command to list all s3 buckets

# s3cmd ls

Sample Output:

2015-04-29 12:38  s3://techoism_code

Some more command to use s3 Bucket on CentOS/RHEL