Delete data from RDS on the onSuccess of a copyActivity in AWS Data Pipeline?

0 votes
I am trying to backup data from RDS(postgres) to s3 incrementally. For this I'm using AWS Data Pipeline. I am able to copy the data, it all works.

The issue I'm facing is that I'm not able to find out a way to delete the already copied data in RDS. I can see there is an onSuccess action, but it only allows me to post a message to SNS or terminate the activity.

Is there a way I can delete data on successful copy of data from RDS to S3 in AWS Data Pipeline?
Oct 1, 2018 in AWS by bug_seeker
• 14,970 points
70 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

onSuccess/onFailure are primarily for status reporting. DPL doesn't really support the idea of branching or error handling. You set up your graph, and it runs through step by step. If something fails, it either pauses or aborts.

I would create a new activity that would run after your copy, using the dependsOn property:

{
  "id": "DeleteDataStep",
  ...
  "dependsOn": { "ref": "CopyRdsToS3Step" },
  ...
}
answered Oct 1, 2018 by Priyaj
• 56,120 points

Related Questions In AWS

0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 56,120 points
85 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,120 points
182 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,520 points
100 views
0 votes
2 answers
0 votes
1 answer

In Amazon Data Pipeline, how to make sure only once instance of a pipeline is running at any time?

On the CopyTablesActivity, you could set a lateAfterTimeout attribute ...READ MORE

answered Sep 19, 2018 in AWS by Priyaj
• 56,120 points
136 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.