How to automatically kill executors on blacklisting?

0 votes
I have certain executors that keep going into blacklist again and again and after a particular time, they come back to execution. What I want is that whenever an executor is blacklisted, I want that executor to be killed automatically. How can I do this?
Mar 12, 2019 in Apache Spark by Uma
125 views

1 answer to this question.

0 votes

You can set the property to directly kill the executor when it is blacklisted like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.blacklist.killBlacklistedExecutors=true

Now whenever an executor is blacklisted, it will be killed. 

answered Mar 12, 2019 by Veer

Related Questions In Apache Spark

0 votes
1 answer

How to stop messages from being displayed on spark console?

In your log4j.properties file you need to ...READ MORE

answered Apr 24, 2018 in Apache Spark by kurt_cobain
• 9,310 points
2,454 views
0 votes
1 answer

How to stop INFO messages displaying on Spark console?

Just do the following: Edit your conf/log4j.properties file ...READ MORE

answered Aug 21, 2018 in Apache Spark by nitinrawat895
• 10,920 points
785 views
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

answered Mar 6, 2019 in Apache Spark by Rohit
70 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
369 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,069 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
722 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
29,963 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

answered Mar 13, 2019 in Apache Spark by Venu
398 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
920 views