How to set stage abort in Spark

Question

This is what I want. When a stage is running, I want the stage to be aborted after attempting to try for 2 times. After attempting for 2 times, it should automatically be aborted. How to set this up?

Answer 1

You can do it as follows. Use the property dynamically to set the number of attempts before aborting.

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.stage.maxConsecutiveAttempts=2