Increasing retry before blacklisting executor

Question

Right now, the executor gets blocked after 1 try after the task has failed. I want to increase it to 2. How can I do it?

Answer 1

You can do it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.blacklist.task.maxTaskAttemptsPerExecutor=2