Need help bootstrapping Python module installation on Amazon EMR

0 votes
Hi all, pretty simple question.

My requirement is that I need to make use of a Spark cluster through an EMR console. I will be running a Spark script that has the local dependency on a certain Python package.

What is the easiest way to go about doing this?

All help appreciated.
Feb 11 in Python by Anirudh
• 2,070 points
72 views

1 answer to this question.

0 votes

Hello!

The easiest way to definitely do this is to create a bash script primarily. This script needs to contain your installation commands.

Later, you need to copy it to the S3 and set up a bootstrap action to point to the script (This is done from the console)

Consider the following example:

#!/bin/bash -xe

# Non-standard and non-Amazon Machine Image Python modules:
sudo pip install -U \
  awscli            \
  boto              \
  ciso8601          \
  ujson             \
  workalendar

sudo yum install -y python-psycopg2

Hope this helped!

answered Feb 11 by Nymeria
• 3,520 points

Related Questions In Python

0 votes
1 answer

Need help with Tkinter window formatting using Python

Tkininter comes with the columnspan argument to span the labels ...READ MORE

answered Sep 7, 2018 in Python by ariaholic
• 7,340 points
48 views
0 votes
1 answer

Need help extracting a schema to make use for an avro file in Python

Hi, nice question. So what I daily use ...READ MORE

answered Jan 10 in Python by Nymeria
• 3,520 points
614 views
0 votes
1 answer
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 337 views
+4 votes
6 answers
0 votes
1 answer

Need help installing easy_install in Python 2.7.1 on Windows 7

That tool is part of the setuptools ...READ MORE

answered Dec 26, 2018 in Python by Nymeria
• 3,520 points
78 views
+1 vote
1 answer

Need some help with Python memory leaks

As far as best practices, keep an ...READ MORE

answered Nov 26, 2018 in Python by Nymeria
• 3,520 points
32 views