Incremental data copy using azure datafactory

0 votes
I've created a pipeline to copy data from one blob storage to a different blob storage.  I would like to use incremental copy if it's possible, but haven't found how to specify it. the reason is i would like to run this on a schedule and only copy any new data since last run.
Jan 2, 2020 in Azure by Karan
• 19,610 points
1,141 views

1 answer to this question.

0 votes
  • If your blob is well named with a timestamp, you could follow this doc to copy partitioned data. You could use copy data tool to setup the pipeline. You could select tumbling window and then in file path filed input {year}/{month}/{day}/fileName and choose the right pattern. It will help you construct the parameters.

  • In case If your blob is not well named with timestamp, you could use get metadata activity to check the last modified time. 

Event trigger is just one way to control when the pipeline should run. You could also use tumbling window trigger or schedule trigger in your scenarios. 

answered Jan 2, 2020 by Sirajul
• 59,230 points

Related Questions In Azure

+1 vote
2 answers

How do I establish an connection to Azure IoT Hub? Say a connection like MQTT/AMQP using Python.

Azure IoT Hub comes with 3 SDK's: Service SDK Device ...READ MORE

answered Apr 13, 2018 in Azure by null_void
• 3,220 points
2,167 views
0 votes
1 answer

How to add an App Settings to existing Azure Web Application using Azure Power Shell?

The example is for slot-specific settings, if ...READ MORE

answered Jun 13, 2018 in Azure by club_seesharp
• 3,450 points
1,415 views
0 votes
2 answers

How to process a file using Azure Function?

You don't really need a http trigger ...READ MORE

answered Jun 18, 2018 in Azure by club_seesharp
• 3,450 points
7,257 views
+1 vote
1 answer

How to copy Azure SQL database to a local development server?

Actually, there are multiple ways to do ...READ MORE

answered Jul 9, 2018 in Azure by null_void
• 3,220 points
2,594 views
0 votes
1 answer

Azure Data factory : Pipeline data flow execution fails.

This error usually occurs when dataset contains ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,230 points
2,745 views
0 votes
1 answer

How can I schedule a pipeline to run in azure datafactory?

You can use the scheduler trigger or ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,230 points
915 views
0 votes
2 answers

What is integration runtime in azure data factory?

IR is like act as brige between ...READ MORE

answered Feb 22, 2020 in Azure by yahya sharif
6,262 views
0 votes
1 answer

What is the limit on the number of integration runtimes?

There is no hard limit on the ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,230 points
2,729 views
0 votes
1 answer

Can I provision an Azure service without using the portal?

Yes, It is possible to provision the ...READ MORE

answered Dec 27, 2019 in Azure by Sirajul
• 59,230 points
670 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP