Incremental data copy using azure datafactory.

0 votes
I've created a pipeline to copy data from one blob storage to a different blob storage.  I would like to use incremental copy if it's possible, but haven't found how to specify it. the reason is i would like to run this on a schedule and only copy any new data since last run.
Jan 2 in Azure by Karan
• 8,200 points
22 views

1 answer to this question.

0 votes
  • If your blob is well named with a timestamp, you could follow this doc to copy partitioned data. You could use copy data tool to setup the pipeline. You could select tumbling window and then in file path filed input {year}/{month}/{day}/fileName and choose the right pattern. It will help you construct the parameters.

  • In case If your blob is not well named with timestamp, you could use get metadata activity to check the last modified time. 

Event trigger is just one way to control when the pipeline should run. You could also use tumbling window trigger or schedule trigger in your scenarios. 

answered Jan 2 by Sirajul
• 46,080 points

Related Questions In Azure

+1 vote
2 answers

How do I establish an connection to Azure IoT Hub? Say a connection like MQTT/AMQP using Python.

Azure IoT Hub comes with 3 SDK's: Service SDK Device ...READ MORE

answered Apr 13, 2018 in Azure by null_void
• 3,220 points
691 views
0 votes
1 answer

How to add an App Settings to existing Azure Web Application using Azure Power Shell?

The example is for slot-specific settings, if ...READ MORE

answered Jun 13, 2018 in Azure by club_seesharp
• 3,450 points
259 views
0 votes
2 answers

How to process a file using Azure Function?

You don't really need a http trigger ...READ MORE

answered Jun 18, 2018 in Azure by club_seesharp
• 3,450 points
1,955 views
+1 vote
1 answer

How to copy Azure SQL database to a local development server?

Actually, there are multiple ways to do ...READ MORE

answered Jul 9, 2018 in Azure by null_void
• 3,220 points
336 views
0 votes
1 answer

Azure Data factory : Pipeline data flow execution fails.

This error usually occurs when dataset contains ...READ MORE

answered Jan 2 in Azure by Sirajul
• 46,080 points
36 views
0 votes
1 answer
0 votes
1 answer

What is integration runtime in azure data factory?

The integration runtime is the compute infrastructure ...READ MORE

answered Jan 2 in Azure by anonymous
• 46,080 points
64 views
0 votes
1 answer
0 votes
1 answer