Copying existing files in a s3 Bucket to another s3 bucket

0 votes

I have a existing s3 bucket which contains large amount of files. I want to run a lambda function every 1 minute and copy those files to another destination s3 bucket.

My function is:

s3 = boto3.resource('s3')
clientname=boto3.client('s3')
def lambda_handler(event, context):
    bucket = 'test-bucket-for-transfer-check'
    try:
        response = clientname.list_objects(
            Bucket=bucket,
            MaxKeys=5
        )

        for record in response['Contents']:
            key = record['Key']
            copy_source = {
                'Bucket': bucket,
                'Key': key
            }
            try:
                destbucket = s3.Bucket('serverless-demo-s3-bucket')
                destbucket.copy(copy_source, key)
                print('{} transferred to destination bucket'.format(key))

            except Exception as e:
                print(e)
                print('Error getting object {} from bucket {}. '.format(key, bucket))
                raise e
    except Exception as e:
        print(e)
        raise e

Now how can I make sure the function is copying new files each time it runs??

Sep 14, 2018 in AWS by bug_seeker
• 14,970 points
35 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Suppose the two buckets in question are Bucket-A and Bucket-B
and task to be done is copy files from Bucket-A --> Bucket-B

  1. Lamba-1 (code in the question) reads all files from Bucket-A and copy them one-by-one to Bucket-B in a loop.
  2. In the same loop it also puts one entry in DynamoDb table say "Copy_Logs" with column File_key and Flag. Where File_Key is the object key of the file and Flag is set to false telling the state of copy operation
  3. Now configure events on Bucket-B to invoke a Lambda-2 on every put and multi-part upload
  4. Now Lambda-2 will read the object key from S3 notification payload and updates its respective record in DynamoDb table with flag set to true.

Now you have all records in DynamoDb table "Copy-Logs" which files were copied successfully and which were not.

answered Sep 14, 2018 by Priyaj
• 56,120 points

Related Questions In AWS

0 votes
2 answers

How to access files in S3 bucket from R?

You can take a look at the ...READ MORE

answered Aug 10, 2018 in AWS by Deepthi
• 300 points
372 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 3,770 points
1,561 views
+1 vote
1 answer

Llimit to number of objects in a S3 bucket

If you are referring to the number of objects ...READ MORE

answered Oct 12, 2018 in AWS by Archana
• 3,770 points
16 views
0 votes
1 answer

How to appoint a sub-domain in a S3 bucket?

You need to rename your bucket to ...READ MORE

answered Oct 12, 2018 in AWS by Archana
• 3,770 points
22 views
0 votes
1 answer

How do I use AWS sdk definitions for TypeScript?

Hey, slight modification with what you have ...READ MORE

answered Dec 3, 2018 in AWS by Archana
• 5,360 points
206 views
0 votes
1 answer
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.