Getting memory leak error on AWS EMR - Saving Dataframe into hive external table on S3 -

Question

Hello i am writing spark using python and tring to write the dataframe into table and table is hive external and stored on AWS S3

below is the command : sqlContext.sql(selectQuery).write.mode("overwrite").format(trgFormat).option("compression", trgCompression).save(trgDataFileBase)

Below is the error

ERROR ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

spark sumit: spark-submit 

--master yarn 

--queue default 

--deploy-mode client 

--num-executors 10 

--executor-memory 12g 

--executor-cores 2 

--conf spark.debug.maxToStringFields=100 

--conf spark.yarn.executor.memoryOverhead=2048

Answer 1

You can start by creating  a temp table, say trgDataFileBasetmp, then using the same definition create the table on s3. You will need all the parameters in the definition like SERDEPROPERTIES, TBLPROPERTIES. Here the difference I have is saveAsTable:

sqlContext.sql(selectQuery).write.mode("overwrite").format(trgFormat).option("compression", trgCompression).saveAsTable(trgDataFileBase)

If this does not work then you can start with:

sqlContext.sql(selectQuery).write.mode("overwrite").saveAsTable(trgDataFileBase)

Hope one out of the two would help.