Unable to spawn executor on remote worker node using multinode spark cluster in spark-shell

0 votes

I've installed spark cluster standalone on two nodes, one of them running as master and the other as worker. Spark shell works fine on worker node with word count but this comes up when I try and run it on the master.

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Somehow executor is not triggered. The worker is available to spark master its giving such a problem.

Jun 4, 2018 in DevOps Tools by Atul
• 5,470 points
225 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

I think your executor node cannot connect to the driver port on the local machine. It may be your firewall blocking it or some issue with IP.

Make Sure that your spark.driver.bindAddress, spark.driver.host, spark.driver.port use the required values. refer Spark documentation here.

Also make sure that you're not requesting more resources than provided by workers.

answered Jun 4, 2018 by ajs3033
• 7,000 points

Related Questions In DevOps Tools

0 votes
1 answer
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,520 points
40 views
0 votes
1 answer
0 votes
3 answers
+13 votes
2 answers
0 votes
1 answer
0 votes
1 answer

Logs in phoenix web app deployed using edeliver

The newer versions of edilever and distillery ...READ MORE

answered Jun 19, 2018 in DevOps Tools by ajs3033
• 7,000 points
24 views
0 votes
1 answer

How to download the latest minor version of nexus artifact using REST?

If you're using nexus 2.x, you can ...READ MORE

answered Jun 27, 2018 in DevOps Tools by ajs3033
• 7,000 points
86 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.