Memory Issues in Talend while working with large files

0 votes

I have a job which uses a 3GB input file using Talend Open Studio for Big Data. Below is the screenshot of my job.

image

But on the first execution of this job, it gave me an error related to heap memory issue.

On the second run, it gave me an error related to garbage collector error.

And now, it's not showing error, but simply executing and doing nothing and finally stopping.

Can someone explain why I am getting this kind of error?

May 2, 2018 in Talend by v.liyyah
• 1,300 points
1,179 views

1 answer to this question.

0 votes

I think the problem lies in the huge amount of data that is being returned by the tSampleRow component. Try using a where clause in your tMysqlInput component in order to filter and fetch less number of rows. Also, in the advanced settings select the “stream” option in order to stream the rows.

image

answered May 2, 2018 by code.reaper12
• 3,500 points

Related Questions In Talend

+1 vote
1 answer
0 votes
0 answers

how to read .dbf files in talend 2020 ?

hi , i am new in talend ...READ MORE

Dec 1, 2020 in Talend by aminegac
• 560 points
184 views
0 votes
1 answer

Parsing JSON in Talend

While working with JSON files, one thing ...READ MORE

answered Apr 2, 2018 in Talend by code.reaper12
• 3,500 points
1,675 views
0 votes
1 answer
0 votes
2 answers
0 votes
1 answer

tWaitForFile component insufficiency in Talend

Regarding your first issue, tWaitForFile component provides ...READ MORE

answered Apr 10, 2018 in Talend by geek.erkami
• 2,680 points
957 views
0 votes
1 answer

How to load a set of files into database using Talend

Using tFileList component,  you can read all the files present ...READ MORE

answered Apr 16, 2018 in Talend by code.reaper12
• 3,500 points
2,376 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Looping through all the schemas in Talend

It is really simple to make tOracleInput ...READ MORE

answered Apr 10, 2018 in Talend by code.reaper12
• 3,500 points
677 views