Thread to use Yarn application master is limited

Question

Hi. I am running Spark application with Yarn and I am trying to increase the number of threads to use in Yarn application master to launch the executor containers but the problem is that the number of threads is not going beyond a particular threshold. Please help

Answer 1

This is because the maximum number of threads that can be used to launch the executor containers is set to 25. So unless and until you change this value, you can increase the number of threads beyond this. To change the max limit, open Spark shell and run the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.containerLauncherMaxThreads= <max number of threads>