How to increase Garbage Collection speed

Question

Please help me with Garbage Collection problem. I have created a Spark application which leaves out a lot of garbage. I know I have to optimize it but for now, I need a quick fix. Please tell me how to fasten the garbage collection interval.

Answer 1

The time interval between Garbage Collection is set to 30min by default. If you want the Garbage Collection to happen sooner, then reduce the time. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.cleaner.periodicGC.interval=20