Unable to spawn executor on remote worker node using multinode spark cluster in spark-shell

0 votes

I've installed spark cluster standalone on two nodes, one of them running as master and the other as worker. Spark shell works fine on worker node with word count but this comes up when I try and run it on the master.

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Somehow executor is not triggered. The worker is available to spark master its giving such a problem.

Jun 4, 2018 in DevOps Tools by ffdfd
• 5,550 points
1,727 views

1 answer to this question.

0 votes

I think your executor node cannot connect to the driver port on the local machine. It may be your firewall blocking it or some issue with IP.

Make Sure that your spark.driver.bindAddress, spark.driver.host, spark.driver.port use the required values. refer Spark documentation here.

Also make sure that you're not requesting more resources than provided by workers.

answered Jun 4, 2018 by ajs3033
• 7,300 points

Related Questions In DevOps Tools

0 votes
1 answer
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,890 points
495 views
0 votes
0 answers

How to install java 8 using heira in puppet?

How do I install java8 using a ...READ MORE

Feb 3, 2020 in DevOps Tools by anonymous
• 19,610 points
436 views
0 votes
1 answer

How to create a secret from literals in Minishift Cluster?

Hi@akhtar, You can create a secret using generic ...READ MORE

answered Aug 18, 2020 in DevOps Tools by MD
• 95,440 points
861 views
+15 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
3,439 views
+2 votes
1 answer
0 votes
1 answer

Logs in phoenix web app deployed using edeliver

The newer versions of edilever and distillery ...READ MORE

answered Jun 19, 2018 in DevOps Tools by ajs3033
• 7,300 points
730 views
0 votes
1 answer

How to download the latest minor version of nexus artifact using REST?

If you're using nexus 2.x, you can ...READ MORE

answered Jun 27, 2018 in DevOps Tools by ajs3033
• 7,300 points
3,088 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP