How many executors run when dynamic allocation is enabled?

0 votes
I have enabled dynamic allocation for my Spark application. I want to know how many executors are running. And also, suppose I want to change the number of executors, how can I do it?
Mar 13 in Apache Spark by Tina
29 views

1 answer to this question.

0 votes

The number of executors running by default is the number of executors set to the spark.dynamicAllocation.minExecutors. To change this, you need to change the property spark.dynamicAllocation.initialExecutors:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.dynamicAllocation.initialExecutors = <number of executors>
answered Mar 13 by Venu

Related Questions In Apache Spark

0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

answered Mar 13 in Apache Spark by Venu
81 views
0 votes
1 answer

How to enable dynamic resource allocation in Spark?

To dynamically enable dynamic resource allocation, you ...READ MORE

answered Mar 12 in Apache Spark by veer
89 views
0 votes
1 answer

In how many modes Apache spark can run?

Hey, You can launch spark application in four ...READ MORE

answered Aug 2 in Apache Spark by Gitika
• 25,340 points
20 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,680 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
279 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,310 views
0 votes
1 answer

When is speculation enabled in Spark?

Speculation is enabled when a fraction of ...READ MORE

answered Mar 12 in Apache Spark by veer
100 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28 in Apache Spark by Raj
145 views