Memory Issues in Talend while working with large files

0 votes

I have a job which uses a 3GB input file using Talend Open Studio for Big Data. Below is the screenshot of my job.

image

But on the first execution of this job, it gave me an error related to heap memory issue.

On the second run, it gave me an error related to garbage collector error.

And now, it's not showing error, but simply executing and doing nothing and finally stopping.

Can someone explain why I am getting this kind of error?

May 3, 2018 in Talend by v.liyyah
• 1,300 points
2,287 views

1 answer to this question.

0 votes

I think the problem lies in the huge amount of data that is being returned by the tSampleRow component. Try using a where clause in your tMysqlInput component in order to filter and fetch less number of rows. Also, in the advanced settings select the “stream” option in order to stream the rows.

image

Hope this was what you were looking for. 

Get your Talend certification today to become certified professional.

Thanks.

answered May 3, 2018 by code.reaper12
• 3,500 points

Related Questions In Talend

+1 vote
1 answer
0 votes
0 answers

how to read .dbf files in talend 2020 ?

hi , i am new in talend ...READ MORE

Dec 2, 2020 in Talend by aminegac
• 570 points
1,426 views
0 votes
1 answer

Parsing JSON in Talend

While working with JSON files, one thing ...READ MORE

answered Apr 3, 2018 in Talend by code.reaper12
• 3,500 points
3,577 views
0 votes
1 answer
0 votes
2 answers
0 votes
1 answer

tWaitForFile component insufficiency in Talend

Regarding your first issue, tWaitForFile component provides ...READ MORE

answered Apr 11, 2018 in Talend by geek.erkami
• 2,680 points
2,149 views
0 votes
1 answer

How to load a set of files into database using Talend

Using tFileList component,  you can read all the files present ...READ MORE

answered Apr 17, 2018 in Talend by code.reaper12
• 3,500 points
5,669 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Looping through all the schemas in Talend

It is really simple to make tOracleInput ...READ MORE

answered Apr 11, 2018 in Talend by code.reaper12
• 3,500 points
1,491 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP