How many executors run when dynamic allocation is enabled

Question

I have enabled dynamic allocation for my Spark application. I want to know how many executors are running. And also, suppose I want to change the number of executors, how can I do it?

Answer 1

The number of executors running by default is the number of executors set to the spark.dynamicAllocation.minExecutors. To change this, you need to change the property spark.dynamicAllocation.initialExecutors:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.dynamicAllocation.initialExecutors = <number of executors>