Incremental data copy using azure datafactory

0 votes
I've created a pipeline to copy data from one blob storage to a different blob storage.  I would like to use incremental copy if it's possible, but haven't found how to specify it. the reason is i would like to run this on a schedule and only copy any new data since last run.
Jan 2, 2020 in Azure by Karan
• 19,590 points
350 views

1 answer to this question.

0 votes
  • If your blob is well named with a timestamp, you could follow this doc to copy partitioned data. You could use copy data tool to setup the pipeline. You could select tumbling window and then in file path filed input {year}/{month}/{day}/fileName and choose the right pattern. It will help you construct the parameters.

  • In case If your blob is not well named with timestamp, you could use get metadata activity to check the last modified time. 

Event trigger is just one way to control when the pipeline should run. You could also use tumbling window trigger or schedule trigger in your scenarios. 

answered Jan 2, 2020 by Sirajul
• 59,030 points

Related Questions In Azure

+1 vote
2 answers

How do I establish an connection to Azure IoT Hub? Say a connection like MQTT/AMQP using Python.

Azure IoT Hub comes with 3 SDK's: Service SDK Device ...READ MORE

answered Apr 13, 2018 in Azure by null_void
• 3,220 points
1,133 views
0 votes
1 answer

How to add an App Settings to existing Azure Web Application using Azure Power Shell?

The example is for slot-specific settings, if ...READ MORE

answered Jun 13, 2018 in Azure by club_seesharp
• 3,450 points
594 views
0 votes
2 answers

How to process a file using Azure Function?

You don't really need a http trigger ...READ MORE

answered Jun 18, 2018 in Azure by club_seesharp
• 3,450 points
4,777 views
+1 vote
1 answer

How to copy Azure SQL database to a local development server?

Actually, there are multiple ways to do ...READ MORE

answered Jul 9, 2018 in Azure by null_void
• 3,220 points
1,188 views
0 votes
1 answer

Azure Data factory : Pipeline data flow execution fails.

This error usually occurs when dataset contains ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,030 points
964 views
0 votes
1 answer

How can I schedule a pipeline to run in azure datafactory?

You can use the scheduler trigger or ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,030 points
164 views
0 votes
2 answers

What is integration runtime in azure data factory?

IR is like act as brige between ...READ MORE

answered Feb 21, 2020 in Azure by yahya sharif
3,100 views
0 votes
1 answer

What is the limit on the number of integration runtimes?

There is no hard limit on the ...READ MORE

answered Jan 2, 2020 in Azure by Sirajul
• 59,030 points
573 views
0 votes
1 answer