How to monitor interrupted Spark tasks

Question

I am running a Spark application in which some tasks are either getting killed or interrupted due to which I am losing track of it. How can I monitor a task even if it is interrupted?

Answer 1

To enable monitoring interrupted tasks, run the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.task.reaper.enabled=true

Now the task will be monitored until the task actually finishes execution.