Need help bootstrapping Python module installation on Amazon EMR

0 votes
Hi all, pretty simple question.

My requirement is that I need to make use of a Spark cluster through an EMR console. I will be running a Spark script that has the local dependency on a certain Python package.

What is the easiest way to go about doing this?

All help appreciated.
Feb 11 in Python by Anirudh
• 2,050 points
14 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Hello!

The easiest way to definitely do this is to create a bash script primarily. This script needs to contain your installation commands.

Later, you need to copy it to the S3 and set up a bootstrap action to point to the script (This is done from the console)

Consider the following example:

#!/bin/bash -xe

# Non-standard and non-Amazon Machine Image Python modules:
sudo pip install -U \
  awscli            \
  boto              \
  ciso8601          \
  ujson             \
  workalendar

sudo yum install -y python-psycopg2

Hope this helped!

answered Feb 11 by Nymeria
• 3,500 points

Related Questions In Python

0 votes
1 answer

Need help with Tkinter window formatting using Python

Tkininter comes with the columnspan argument to span the labels ...READ MORE

answered Sep 7, 2018 in Python by ariaholic
• 7,320 points
31 views
0 votes
1 answer

Need help extracting a schema to make use for an avro file in Python

Hi, nice question. So what I daily use ...READ MORE

answered Jan 10 in Python by Nymeria
• 3,500 points
194 views
0 votes
1 answer
0 votes
1 answer

how can i count the items in a list?

suppose you have a list a = [0,1,2,3,4,5,6,7,8,9,10] now ...READ MORE

answered May 2 in Python by Mohammad
• 920 points
19 views
+4 votes
6 answers
0 votes
1 answer

Need help installing easy_install in Python 2.7.1 on Windows 7

That tool is part of the setuptools ...READ MORE

answered Dec 26, 2018 in Python by Nymeria
• 3,500 points
45 views
0 votes
1 answer

Need some help with Python memory leaks

As far as best practices, keep an ...READ MORE

answered Nov 26, 2018 in Python by Nymeria
• 3,500 points
15 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.