Transfer data using AWS CLI from one s3 bucket to another

Question

I am trying to transfer 25 tb of data stored in s3 from one aws account to another s3 bucket in another AWS account (Both where in different regions) using AWS CLI, can anyone suggest me which EC2 instance is better to use and process for data transfer with CLI and mainly how much time it may take to complete the transfer.

Comments

Hi Anjali,

I have a similar situation. Were you able to complete the process? And how much time did it take?

Answer 1

Copying is the easy part! Use the AWS Command-Line Interface (CLI):

aws s3 sync s3://source-bucket s3://destination-bucket

Permissions that you will need to check.

What you will need to consider is how to permit access to copy the files. Let's say you have:

• Account A with Bucket A
• Account B with Bucket B
• You wish to copy from Bucket A to Bucket B

You should run the sync command from a user ("User B") in Account B that has permissions to write to Bucket B.

You will also need to add a Bucket Policy to Bucket A that specifically permits access by User B. The policy would look something like:

{
  "Id": "Policy1",
  "Version": "2019-01-15",
  "Statement": [
    {
      "Sid": "ReadOnlyAccess",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-bucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/user-b"
        ]
      }
    }
  ]
}