Increase number of cores in Spark

Question

Hi. I am working on Spark and have started a driver job. It was running slow so checked the configuration, it seems like it is using only 1 core. I want to increase the number of cores. How can I do it?

Answer 1

Now that the job is already running, changing the configuration by hard-code is not advisable. It is good that you have decided to dynamically change it. To do this, refer the below code:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --conf spark.driver.cores=1

Comments

Why is it not a good idea to change the config by hard-code?

---

Changing the configuration by hard-code is not advisable in this scenario. Kiran has jobs running. And if you change the configuration by hard-code, it might affect the running jobs.