How to change worker cleanup interval

Question

I have enabled worker cleanup using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.worker.cleanup.enabled=true

It is taking to long to start cleanup. Can someone tell me what is the default interval for cleanup and how it can be changed?

Answer 1

The default interval time is 1800 seconds (30 minutes). You can change the interval similar to how you enabled the cleanup but the only difference is that you to set the interval time to a different property. Something like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.worker.cleanup.interval=900