How to choose number of Executors and Executor cores

0 votes
Could you please help in detail, How to choose number of Executors and Executor cores while submitting a spark job?
Jul 29, 2019 in Big Data Hadoop by Kim
3,033 views

1 answer to this question.

0 votes

Refer to the below when you are submitting a spark job in the cluster:

spark-submit --master yarn-cluster --class com.yourCompany.code --executor-memory 32G --num-executors 5 --driver-memory 4g --executor-cores 3 --queue parsons YourJARfile.jar
answered Jul 29, 2019 by Veer

Related Questions In Big Data Hadoop

0 votes
1 answer

How to set the number of Map & Reduce tasks?

The map tasks created for a job ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,490 points
855 views
0 votes
1 answer

What is the command to check the number of cores in Spark?

Go to your Spark Web UI & ...READ MORE

answered May 17, 2018 in Big Data Hadoop by Shubham
• 13,490 points
4,948 views
0 votes
1 answer

How to calculate number of slave nodes?

Total number of slave nodes in your ...READ MORE

answered Aug 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
609 views
0 votes
1 answer

How to choose between Cassandra, Membase, Hadoop, MongoDB and RDBMS?

Actually it comes in two ways: One ...READ MORE

answered Sep 18, 2018 in Big Data Hadoop by Frankie
• 9,830 points
405 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
9,322 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,706 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
86,165 views
0 votes
1 answer

Hadoop hdfs: How to count number of lines?

You can count the number of lines ...READ MORE

answered Nov 19, 2018 in Big Data Hadoop by Omkar
• 69,210 points
730 views
0 votes
1 answer

How to find the number of blocks a hdfs file is divided into?

Yes. you can use the hadoop fsck command to do ...READ MORE

answered Nov 30, 2018 in Big Data Hadoop by Omkar
• 69,210 points
4,507 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP