Copying existing files in a s3 Bucket to another s3 bucket

Question

I have a existing s3 bucket which contains large amount of files. I want to run a lambda function every 1 minute and copy those files to another destination s3 bucket.

My function is:

s3 = boto3.resource('s3')
clientname=boto3.client('s3')
def lambda_handler(event, context):
    bucket = 'test-bucket-for-transfer-check'
    try:
        response = clientname.list_objects(
            Bucket=bucket,
            MaxKeys=5
        )

        for record in response['Contents']:
            key = record['Key']
            copy_source = {
                'Bucket': bucket,
                'Key': key
            }
            try:
                destbucket = s3.Bucket('serverless-demo-s3-bucket')
                destbucket.copy(copy_source, key)
                print('{} transferred to destination bucket'.format(key))

            except Exception as e:
                print(e)
                print('Error getting object {} from bucket {}. '.format(key, bucket))
                raise e
    except Exception as e:
        print(e)
        raise e

Now how can I make sure the function is copying new files each time it runs??

Answer 1

Suppose the two buckets in question are Bucket-A and Bucket-B
and task to be done is copy files from Bucket-A --> Bucket-B

1. Lamba-1 (code in the question) reads all files from Bucket-A and copy them one-by-one to Bucket-B in a loop.
2. In the same loop it also puts one entry in DynamoDb table say "Copy_Logs" with column File_key and Flag. Where File_Key is the object key of the file and Flag is set to false telling the state of copy operation
3. Now configure events on Bucket-B to invoke a Lambda-2 on every put and multi-part upload
4. Now Lambda-2 will read the object key from S3 notification payload and updates its respective record in DynamoDb table with flag set to true.

Now you have all records in DynamoDb table "Copy-Logs" which files were copied successfully and which were not.