How to enable Spark event logging

Question

Hi. I am running a Spark application and I want to build a Web UI for it after it is completed. And for this, I need the event logs of the Spark applications. I have read that by default Spark does not store event logs. How can I make Spark to store the logs?

Answer 1

To make Spark store the event logs, you will have to enable this feature. Now that your application is running, I would suggest you to dynamically do it. Refer to the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.eventLog.enabled=true

The logs will be stored at file:///tmp/spark-events