How to cleanup application work directories faster

Question

Hi. I am running a Spark application on a system that has less disk space. But because of the application work directories, space is getting occupied causing memory problems. How can schedule the cleanup sooner?

Answer 1

By default, the cleanup time is set to 604800 (7 days). You can change it as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.worker.cleanup.appDataTtl=<Time in seconds>