Memory Issues in Talend while working with large files

0 votes

I have a job which uses a 3GB input file using Talend Open Studio for Big Data. Below is the screenshot of my job.

image

But on the first execution of this job, it gave me an error related to heap memory issue.

On the second run, it gave me an error related to garbage collector error.

And now, it's not showing error, but simply executing and doing nothing and finally stopping.

Can someone explain why I am getting this kind of error?

May 2, 2018 in Talend by v.liyyah
• 1,300 points
302 views

1 answer to this question.

0 votes

I think the problem lies in the huge amount of data that is being returned by the tSampleRow component. Try using a where clause in your tMysqlInput component in order to filter and fetch less number of rows. Also, in the advanced settings select the “stream” option in order to stream the rows.

image

answered May 2, 2018 by code.reaper12
• 3,450 points

Related Questions In Talend

0 votes
1 answer

Parsing JSON in Talend

While working with JSON files, one thing ...READ MORE

answered Apr 2, 2018 in Talend by code.reaper12
• 3,450 points
475 views
0 votes
1 answer
0 votes
1 answer

Special issues while using WSO2

To choose an ESB for integration, you ...READ MORE

answered Apr 9, 2018 in Talend by anto.trigg4
• 3,420 points
73 views
0 votes
1 answer

Converting Columns into Rows in Talend

Hi, You can try using tNormalize component along ...READ MORE

answered Apr 10, 2018 in Talend by code.reaper12
• 3,450 points

edited Apr 10, 2018 by code.reaper12 668 views
0 votes
2 answers
0 votes
1 answer

tWaitForFile component insufficiency in Talend

Regarding your first issue, tWaitForFile component provides ...READ MORE

answered Apr 10, 2018 in Talend by geek.erkami
• 2,320 points
282 views
0 votes
1 answer

How to load a set of files into database using Talend

Using tFileList component,  you can read all the files present ...READ MORE

answered Apr 16, 2018 in Talend by code.reaper12
• 3,450 points
543 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Looping through all the schemas in Talend

It is really simple to make tOracleInput ...READ MORE

answered Apr 10, 2018 in Talend by code.reaper12
• 3,450 points
90 views