How to set cpu cores for spark task

0 votes
I am running some tasks in my Spark application and it is running a little slow so I am thinking of increasing the number of cores assigned to each task. How can I do it?
Mar 12, 2019 in Apache Spark by Prateek
2,413 views

1 answer to this question.

0 votes

By default, each task is allocated with 1 cpu core. To increase this, you can dynamically change the number of cores allocated;

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.task.cpus=<number of cores>
answered Mar 12, 2019 by Veer

Related Questions In Apache Spark

0 votes
1 answer
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
636 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
2,192 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
1,742 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,216 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,171 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
52,869 views
0 votes
1 answer

How to set time for task speculation?

By default, the check for task speculation ...READ MORE

answered Mar 12, 2019 in Apache Spark by Veer
86 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

answered Mar 11, 2019 in Apache Spark by Raj
618 views