Increasing retry before blacklisting a node

Question

I want to increase the number of retries after which a node can be blacklisted due to task failures. Please tell me how to do this dynamically.

Answer 1

You can do it dynamically using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.blacklist.task.maxTaskAttemptsPerNode=<number of retries>