Unable to spawn executor on remote worker node using multinode spark cluster in spark-shell

0 votes

I've installed spark cluster standalone on two nodes, one of them running as master and the other as worker. Spark shell works fine on worker node with word count but this comes up when I try and run it on the master.

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Somehow executor is not triggered. The worker is available to spark master its giving such a problem.

Jun 4, 2018 in DevOps Tools by Atul
• 5,530 points
317 views

1 answer to this question.

0 votes

I think your executor node cannot connect to the driver port on the local machine. It may be your firewall blocking it or some issue with IP.

Make Sure that your spark.driver.bindAddress, spark.driver.host, spark.driver.port use the required values. refer Spark documentation here.

Also make sure that you're not requesting more resources than provided by workers.

answered Jun 4, 2018 by ajs3033
• 7,280 points

Related Questions In DevOps Tools

0 votes
1 answer
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,810 points
49 views
0 votes
1 answer
0 votes
3 answers
+13 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by DragonLord999
• 8,380 points
114 views
0 votes
1 answer
0 votes
1 answer

Logs in phoenix web app deployed using edeliver

The newer versions of edilever and distillery ...READ MORE

answered Jun 19, 2018 in DevOps Tools by ajs3033
• 7,280 points
37 views
0 votes
1 answer

How to download the latest minor version of nexus artifact using REST?

If you're using nexus 2.x, you can ...READ MORE

answered Jun 27, 2018 in DevOps Tools by ajs3033
• 7,280 points
211 views