Need help bootstrapping Python module installation on Amazon EMR

0 votes
Hi all, pretty simple question.

My requirement is that I need to make use of a Spark cluster through an EMR console. I will be running a Spark script that has the local dependency on a certain Python package.

What is the easiest way to go about doing this?

All help appreciated.
Feb 11, 2019 in Python by Anirudh
• 2,080 points
694 views

1 answer to this question.

0 votes

Hello!

The easiest way to definitely do this is to create a bash script primarily. This script needs to contain your installation commands.

Later, you need to copy it to the S3 and set up a bootstrap action to point to the script (This is done from the console)

Consider the following example:

#!/bin/bash -xe

# Non-standard and non-Amazon Machine Image Python modules:
sudo pip install -U \
  awscli            \
  boto              \
  ciso8601          \
  ujson             \
  workalendar

sudo yum install -y python-psycopg2

Hope this helped!

answered Feb 11, 2019 by Nymeria
• 3,520 points

i tried but i got this error

sudo: easy_install-3.4: command not found
sudo: /usr/local/bin/pip3: command not found

Hello @ akash,

First of all: try pip3 instead of pip

Example:

pip3 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)

pip3 should be installed automatically together with Python3.x. The documentation hasn't been updated, so simply replace pip by pip3 in the instructions, when installing Flask for example.

Now, if this doesn't work, you might have to install pip separately.

Hope it helps!!
Thank you!

Related Questions In Python

0 votes
1 answer

Need help with Tkinter window formatting using Python

Tkininter comes with the columnspan argument to span the labels ...READ MORE

answered Sep 7, 2018 in Python by aryya
• 7,380 points
120 views
0 votes
1 answer

Need help extracting a schema to make use for an avro file in Python

Hi, nice question. So what I daily use ...READ MORE

answered Jan 10, 2019 in Python by Nymeria
• 3,520 points
1,786 views
–1 vote
1 answer
0 votes
1 answer

Need help with Python Text-to-Speech usage

Hi. Just before I begin my answer I ...READ MORE

answered Jan 16, 2019 in Python by Nymeria
• 3,520 points
131 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 1,481 views
0 votes
0 answers
+4 votes
6 answers

Lowercase in Python

You can simply the built-in function in ...READ MORE

answered Apr 11, 2018 in Python by hemant
• 5,800 points
1,113 views
0 votes
1 answer

Need help installing easy_install in Python 2.7.1 on Windows 7

That tool is part of the setuptools ...READ MORE

answered Dec 26, 2018 in Python by Nymeria
• 3,520 points
228 views
+1 vote
1 answer

Need some help with Python memory leaks

As far as best practices, keep an ...READ MORE

answered Nov 26, 2018 in Python by Nymeria
• 3,520 points
71 views