AWS EMR script-runner access error

0 votes

I'm running emr-5.12.0, with Amazon 2.8.3, Hive 2.3.2, Hue 4.1.0, Livy 0.4.0, Spark 2.2.1 and Zeppelin 0.7.3 on 1 m4.large as my master node and 1 m4.large as core node.

I am trying to execute a bootstrap action that configures some parts of the cluster. One of these includes the line:

sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/alternatives/zeppelin-conf/interpreter.json

It makes sure that the Zeppelin uses python3.4 instead of python2.7. It works fine if I execute this in the terminal after SSH'ing to the master node, but it fails when I submit it as a Custom JAR step on the AWS Web interface. I get the following error:

ed: can't read /etc/alternatives/zeppelin-conf/interpreter.json : No such file or directory Command exiting with ret '2'

The same thing happens if I use

sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/zeppelin-conf/interpreter.json

Obviously I could just change it from the Zeppelin UI, but I would like to include it in the bootstrap action.


Aug 6, 2018 in AWS by bug_seeker
• 15,390 points

1 answer to this question.

0 votes

It turns out that a bootstrap action submitted through the AWS EMR web interface is submitted as a regular EMR step, so it's only run on the master node. This can be seen if you click the 'AWS CLI export' in the cluster web interface. The intended bootstrap action is listed as a regular step. Using the command line to launch a cluster with a bootstrap action bypasses this problem, so I've just used that.​

answered Aug 6, 2018 by Priyaj
• 57,450 points

Related Questions In AWS

0 votes
1 answer

A strange spark ERROR on AWS EMR

Those warning messages can be suppressed by ...READ MORE

answered Jul 13, 2018 in AWS by Flying geek
• 3,210 points
+1 vote
2 answers

How do I run python script on aws EC2 instance(ubuntu)?

I believe that you are using the ...READ MORE

answered Apr 17, 2018 in AWS by Cloud gunner
• 4,620 points
+1 vote
3 answers

Log in to AWS using Access Key ID and Secret Access Key ID

Access keys consist of an access key ...READ MORE

answered Aug 17, 2018 in AWS by Priyaj
• 57,450 points
+13 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by DragonLord999
• 8,400 points
0 votes
1 answer
0 votes
1 answer

Getting memory leak error on AWS EMR - Saving Dataframe into hive external table on S3 -

You can start by creating  a temp ...READ MORE

answered Aug 9, 2018 in AWS by Priyaj
• 57,450 points
0 votes
1 answer

AWS Access Key error when uploading to S3

You can try getting creating a new ...READ MORE

answered Feb 6, 2019 in AWS by Priyaj
• 57,450 points