How to disable automatic remove of application of failures

Question

Hi. One of my application keeps failing continuously and then gets removed after that. I have read somewhere that after a certain number of failures, the application gets removed and that's the default behavior. Now, I want to make changes to this behavior. I don't want the application to be removed. How can I do this?

Answer 1

Yes, you have read it right. The application will be removed after it fails for a particular number of times (by default this is 10). To prevent the application from being removed, you can to set this to -1. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.deploy.maxExecutorRetries=-1