How to choose number of Executors and Executor cores

Question

Could you please help in detail, How to choose number of Executors and Executor cores while submitting a spark job?

Answer 1

Refer to the below when you are submitting a spark job in the cluster:

spark-submit --master yarn-cluster --class com.yourCompany.code --executor-memory 32G --num-executors 5 --driver-memory 4g --executor-cores 3 --queue parsons YourJARfile.jar