How to relaunch tasks that are running slowly

Question

I am running a Spark application and I want to make the following configuration. When a set of tasks are being executed, suppose few tasks are running slow in a stage, then I want Spark to relaunch that particular task. How can I do this?

Answer 1

The technical term for what you want to do is Speculation. Speculation is when you relaunch the task if it is running slow. To do this, you can use the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.speculation=true