Is there any way to include a python package with Hadoop streaming job

Question

I want to include a python package (NLTK) with a Hadoop streaming job, but am not sure how to do this without including every file manually via the CLI argument, "-file".Please note- I don't have the option to install this package on all the slaves.

Can someone please helps me, how to do this?

Thanks in advance!

Answer 1

See I don't know the answer to your question exactly but one thing you can surely try: 

Just zip up the package into a .tar.gz or a .zip and pass the entire tarball or archive in a -file option to your hadoop command. I've done this in the past with Perl but not in case of Python.

You can also use Python's zipimport at http://docs.python.org/library/zipimport.html, which allows you to import modules directly from a zip.

Hope this will answer your question to some extent.