AWS EMR script-runner access error

Question

I'm running emr-5.12.0, with Amazon 2.8.3, Hive 2.3.2, Hue 4.1.0, Livy 0.4.0, Spark 2.2.1 and Zeppelin 0.7.3 on 1 m4.large as my master node and 1 m4.large as core node.

I am trying to execute a bootstrap action that configures some parts of the cluster. One of these includes the line:

sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/alternatives/zeppelin-conf/interpreter.json

It makes sure that the Zeppelin uses python3.4 instead of python2.7. It works fine if I execute this in the terminal after SSH'ing to the master node, but it fails when I submit it as a Custom JAR step on the AWS Web interface. I get the following error:

ed: can't read /etc/alternatives/zeppelin-conf/interpreter.json : No such file or directory Command exiting with ret '2'

The same thing happens if I use

sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/zeppelin-conf/interpreter.json

Obviously I could just change it from the Zeppelin UI, but I would like to include it in the bootstrap action.

Thanks!

Answer 1

It turns out that a bootstrap action submitted through the AWS EMR web interface is submitted as a regular EMR step, so it's only run on the master node. This can be seen if you click the 'AWS CLI export' in the cluster web interface. The intended bootstrap action is listed as a regular step. Using the command line to launch a cluster with a bootstrap action bypasses this problem, so I've just used that.​