How to set cpu cores for spark task

Question

I am running some tasks in my Spark application and it is running a little slow so I am thinking of increasing the number of cores assigned to each task. How can I do it?

Answer 1

By default, each task is allocated with 1 cpu core. To increase this, you can dynamically change the number of cores allocated;

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.task.cpus=<number of cores>