Slow transfer rate in Talend Job

Question

I have a job where I need to update data from sql rds to redshift on a daily basis.

Below is my Job flow:

And below is the query I am using to retrieve the data:  

SELECT * FROM abc.xyz ​
WHERE id > (SELECT COALESCE(max(id), 0) as id
FROM abc.demo)

Here:

abc→ myrds Table

Demo → redshift table

The job executes file but the rate of data transfer is very slow.

Can someone help me in speeding up my Job?

Answer 1

You can try using tRedshiftOutputBulkExec component for speedy processing.

This component receives data from the preceding component, then generates a single delimited/CSV file and uploads the file to Amazon S3, finally it loads the data from Amazon S3 to Redshift. It enables you to write 20000 rows/s.

Also for other hindrances like EC2 instances and all you can try installing the Talend on EC2 itself.

Hope this helps!

To know more about Talend, join Talend for big data certification training today.

Thanks.