Slow transfer rate in Talend Job

0 votes

I have a job where I need to update data from sql rds to redshift on a daily basis.

Below is my Job flow:

image

And below is the query I am using to retrieve the data:  

SELECT * FROM abc.xyzWHERE id > (SELECT COALESCE(max(id), 0) as id
FROM abc.demo)

Here:

abc→ myrds Table

Demo → redshift table

The job executes file but the rate of data transfer is very slow.

Can someone help me in speeding up my Job?

Apr 24, 2018 in Talend by code.reaper12
• 3,450 points

edited Apr 24, 2018 by code.reaper12 171 views

1 answer to this question.

0 votes

You can try using tRedshiftOutputBulkExec component for speedy processing.

This component receives data from the preceding component, then generates a single delimited/CSV file and uploads the file to Amazon S3, finally it loads the data from Amazon S3 to Redshift. It enables you to write 20000 rows/s.

Also for other hindrances like EC2 instances and all you can try installing the Talend on EC2 itself.

answered Apr 24, 2018 by geek.erkami
• 2,320 points

Related Questions In Talend

0 votes
1 answer

Looping through all the schemas in Talend

It is really simple to make tOracleInput ...READ MORE

answered Apr 10, 2018 in Talend by code.reaper12
• 3,450 points
90 views
0 votes
1 answer

Parsing JSON in Talend

While working with JSON files, one thing ...READ MORE

answered Apr 2, 2018 in Talend by code.reaper12
• 3,450 points
476 views
0 votes
1 answer
0 votes
2 answers
0 votes
1 answer

Converting Columns into Rows in Talend

Hi, You can try using tNormalize component along ...READ MORE

answered Apr 10, 2018 in Talend by code.reaper12
• 3,450 points

edited Apr 10, 2018 by code.reaper12 670 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

tWaitForFile component insufficiency in Talend

Regarding your first issue, tWaitForFile component provides ...READ MORE

answered Apr 10, 2018 in Talend by geek.erkami
• 2,320 points
283 views
0 votes
1 answer

Implement Common Logging in Talend

Well, you can try using following 2 ...READ MORE

answered Apr 11, 2018 in Talend by geek.erkami
• 2,320 points
122 views