How to automatically kill executors on blacklisting

0 votes
I have certain executors that keep going into blacklist again and again and after a particular time, they come back to execution. What I want is that whenever an executor is blacklisted, I want that executor to be killed automatically. How can I do this?
Mar 12, 2019 in Apache Spark by Uma
372 views

1 answer to this question.

0 votes

You can set the property to directly kill the executor when it is blacklisted like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.blacklist.killBlacklistedExecutors=true

Now whenever an executor is blacklisted, it will be killed. 

answered Mar 12, 2019 by Veer

Related Questions In Apache Spark

0 votes
1 answer

How to stop messages from being displayed on spark console?

In your log4j.properties file you need to ...READ MORE

answered Apr 24, 2018 in Apache Spark by kurt_cobain
• 9,390 points
3,557 views
0 votes
1 answer

How to stop INFO messages displaying on Spark console?

Just do the following: Edit your conf/log4j.properties file ...READ MORE

answered Aug 21, 2018 in Apache Spark by nitinrawat895
• 11,380 points
1,197 views
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

answered Mar 6, 2019 in Apache Spark by Rohit
140 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
590 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,859 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,098 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
48,468 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

answered Mar 13, 2019 in Apache Spark by Venu
780 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
1,713 views