Slow transfer rate in Talend Job

0 votes

I have a job where I need to update data from sql rds to redshift on a daily basis.

Below is my Job flow:

image

And below is the query I am using to retrieve the data:  

SELECT * FROM abc.xyzWHERE id > (SELECT COALESCE(max(id), 0) as id
FROM abc.demo)

Here:

abc→ myrds Table

Demo → redshift table

The job executes file but the rate of data transfer is very slow.

Can someone help me in speeding up my Job?

Apr 24, 2018 in Talend by code.reaper12
• 3,500 points

edited Apr 24, 2018 by code.reaper12 1,681 views

1 answer to this question.

0 votes

You can try using tRedshiftOutputBulkExec component for speedy processing.

This component receives data from the preceding component, then generates a single delimited/CSV file and uploads the file to Amazon S3, finally it loads the data from Amazon S3 to Redshift. It enables you to write 20000 rows/s.

Also for other hindrances like EC2 instances and all you can try installing the Talend on EC2 itself.

Hope this helps!

To know more about Talend, join Talend for big data certification training today.

Thanks.

answered Apr 25, 2018 by geek.erkami
• 2,680 points

Related Questions In Talend

0 votes
1 answer

Looping through all the schemas in Talend

It is really simple to make tOracleInput ...READ MORE

answered Apr 11, 2018 in Talend by code.reaper12
• 3,500 points
1,491 views
0 votes
1 answer

Parsing JSON in Talend

While working with JSON files, one thing ...READ MORE

answered Apr 3, 2018 in Talend by code.reaper12
• 3,500 points
3,576 views
0 votes
1 answer
0 votes
2 answers
0 votes
1 answer

Converting Columns into Rows in Talend

Hi, You can try using tNormalize component along ...READ MORE

answered Apr 11, 2018 in Talend by code.reaper12
• 3,500 points

edited Apr 11, 2018 by code.reaper12 5,923 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

tWaitForFile component insufficiency in Talend

Regarding your first issue, tWaitForFile component provides ...READ MORE

answered Apr 11, 2018 in Talend by geek.erkami
• 2,680 points
2,148 views
0 votes
1 answer

Implement Common Logging in Talend

Well, you can try using following 2 ...READ MORE

answered Apr 12, 2018 in Talend by geek.erkami
• 2,680 points
1,636 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP