How to include third party library in Python MapReduce

Question

I am writing MapReduce job in Python, and want to use some third libraries like chardet.

I konw that we can use option -libjars=... to include them for java MapReduce.

But how to include third party libraries in Python MapReduce Job ?

Answer 1

Problem has been solved by zipimport.

Then I zip chardet to file module.mod, and used like this:

importer = zipimport.zipimporter('module.mod')
chardet = importer.load_module('chardet')

Add -file module.mod in hadoop streaming command.

Now chardet can be used in script.