Unable to spawn executor on remote worker node using multinode spark cluster in spark-shell

Question

I've installed spark cluster standalone on two nodes, one of them running as master and the other as worker. Spark shell works fine on worker node with word count but this comes up when I try and run it on the master.

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Somehow executor is not triggered. The worker is available to spark master its giving such a problem.

Answer 1

I think your executor node cannot connect to the driver port on the local machine. It may be your firewall blocking it or some issue with IP.

Make Sure that your spark.driver.bindAddress, spark.driver.host, spark.driver.port use the required values. refer Spark documentation here.

Also make sure that you're not requesting more resources than provided by workers.