How to change the location of Spark event logs

Question

Hi. By default, the event logs are being stored at file:///tmp/spark-events but I want to change it to another location. I am trying to automate something and I need all the logs to be stored in a directory I have created. So, how I can override the default?

Answer 1

You can change the location where you want to store the logs. I wouldn't suggest you change the whole configuration os Spark. I'd rather suggest you update the location only for the job you want to store the logs in a different location. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.eventLog.dir=<path/to/custom/location>