Getting memory leak error on AWS EMR - Saving Dataframe into hive external table on S3 -

0 votes

Hello i am writing spark using python and tring to write the dataframe into table and table is hive external and stored on AWS S3

below is the command : sqlContext.sql(selectQuery).write.mode("overwrite").format(trgFormat).option("compression", trgCompression).save(trgDataFileBase)

Below is the error

ERROR ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

spark sumit: spark-submit 

--master yarn 

--queue default 

--deploy-mode client 

--num-executors 10 

--executor-memory 12g 

--executor-cores 2 

--conf spark.debug.maxToStringFields=100 

--conf spark.yarn.executor.memoryOverhead=2048

Aug 9, 2018 in AWS by bug_seeker
• 14,970 points
82 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

You can start by creating  a temp table, say trgDataFileBasetmp, then using the same definition create the table on s3. You will need all the parameters in the definition like SERDEPROPERTIES, TBLPROPERTIES. Here the difference I have is saveAsTable:

sqlContext.sql(selectQuery).write.mode("overwrite").format(trgFormat).option("compression", trgCompression).saveAsTable(trgDataFileBase)

If this does not work then you can start with:

sqlContext.sql(selectQuery).write.mode("overwrite").saveAsTable(trgDataFileBase)

Hope one out of the two would help.

answered Aug 9, 2018 by Priyaj
• 56,120 points

Related Questions In AWS

0 votes
1 answer

A strange spark ERROR on AWS EMR

Those warning messages can be suppressed by ...READ MORE

answered Jul 13, 2018 in AWS by Flying geek
• 3,130 points
27 views
0 votes
1 answer

AWS S3 CLI : error while trying to copy files locally using terminal

For the first error you should add ...READ MORE

answered Aug 3, 2018 in AWS by Archana
• 3,770 points
627 views
0 votes
1 answer

AWS s3 -trigger on object created, function gets invoked continuously

This is not supposed to happen. Please ...READ MORE

answered Feb 6 in AWS by Esha
28 views
0 votes
1 answer

Migration on-premise Postgresql into AWS RDS

Actually the solution purposed works. But there's ...READ MORE

answered Aug 24, 2018 in AWS by Priyaj
• 56,120 points
77 views
+13 votes
2 answers
0 votes
1 answer
0 votes
1 answer

AWS EMR script-runner access error

It turns out that a bootstrap action ...READ MORE

answered Aug 6, 2018 in AWS by Priyaj
• 56,120 points
114 views
0 votes
1 answer

AWS Access Key error when uploading to S3

You can try getting creating a new ...READ MORE

answered Feb 6 in AWS by Priyaj
• 56,120 points
601 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.