How to move files from amazon ec2 to s3 bucket using command line

Question

In my amazon EC2 instance, I have a folder named uploads. In this folder I have 1000 images. Now I want to copy all images to my new S3 bucket. How can I do this?

Answer 1

There are multiples ways to achieve this:

1. Uisng sm3cmd

s3cmd get s3://AWS_S3_Bucket/dir/file

if you are on linux, run this on the command line:

sudo apt-get install s3cmd

or Centos, Fedore.

yum install s3cmd

2. Using Cli from amazon

Use sync instead of cp

Syntax:

aws s3 sync <source> <target> [--options]

Hope this helps!

Enroll now to become AWS DevOps Engineer.

Thanks!

Answer 2

Hey, 3 ways you can do this:

To Download using AWS S3 CLI :

aws s3 cp s3://WholeBucket LocalFolder --recursive
aws s3 cp s3://Bucket/Folder LocalFolder --recursive

To Download using Code, Use AWS SDK .

To Download using GUI, Use Cyberduck .

Hope it Helps.. 

Comments

How to use and run this code and where? every body talking about it but no one shows exactly whee to use it in the internet browser or in the CMD?

---

Hi@Mike,

As you can see these commands are AWS CLI command. So you need to run from your CLI. But for that, you need to configure AWS CLI in your system.

Answer 3

I've used a few different methods to copy Amazon S3 data to a local machine, including s3cmd, and by far the easiest is Cyberduck. All you need to do is enter your Amazon credentials and use the simple interface to download / upload/sync any of your buckets/ folders/files.

Answer 4

Here is some stuff to download all buckets, list them, list their contents.

    //connection string
    private static void dBConnection() {
    app.setAwsCredentials(CONST.getAccessKey(), CONST.getSecretKey());
    conn = new AmazonS3Client(app.getAwsCredentials());
    app.setListOfBuckets(conn.listBuckets());
    System.out.println(CONST.getConnectionSuccessfullMessage());
    }

    private static void downloadBucket() {

    do {
        for (S3ObjectSummary objectSummary : app.getS3Object().getObjectSummaries()) {
            app.setBucketKey(objectSummary.getKey());
            app.setBucketName(objectSummary.getBucketName());
            if(objectSummary.getKey().contains(CONST.getDesiredKey())){
                //DOWNLOAD
                try 
                {
                    s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
                    s3Client.getObject(
                            new GetObjectRequest(app.getBucketName(),app.getBucketKey()),
                            new File(app.getDownloadedBucket())
                            );
                } catch (IOException e) {
                    e.printStackTrace();
                }

                do
                {
                     if(app.getBackUpExist() == true){
                        System.out.println("Converting back up file");
                        app.setCurrentPacsId(objectSummary.getKey());
                        passIn = app.getDataBaseFile();
                        CONVERT= new DataConversion(passIn);
                        System.out.println(CONST.getFileDownloadedMessage());

Answer 5

AWS Command Line Tool works much like boto and can be installed using sudo easy_install awscli or sudo pip install awscli

Once installed, you can then simply run:

Command:

aws s3 sync s3://mybucket .

Output:

download: s3://mybucket/test.txt to test.txt 

download: s3://mybucket/test2.txt to test2.txt

This will download all of your files (one-way sync). It will not delete any existing files in your current directory (unless you specify --delete), and it won't change or delete any files on S3.

You can also do S3 bucket to S3 bucket, or local to S3 bucket sync.

Check out the documentation and other examples:

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Answer 6

I can recommend using such a safe and reliable tool - Commander One. Besides being a decent alternative to Cyberduck for Mac, Commander one is an advanced file manager with a two-pane interface and a set of handy features to simplify your workflow and can be considered as the most convenient OS X Finder alternative.
Expand your device access capabilities with this Cyberduck alternative Mac solution. Commander One is one spot to work with different devices no matter if it is an iOS/Android or even MTP device.